sum#

A sum adds up the elements of an array, over all of its dimensions or only some, such as the sum of each row. It is the simplest reduction, and the one where the order of additions shows most plainly: the same numbers added in a different order can give a different floating-point result, and parallel implementations add in many different orders.

In ExactRef, er.sum(x, axis=None, keepdims=False), also exactref.reduction.arithmetic.sum, is the sum over the dimensions in axis (all of them by default; an int or a tuple of ints), as in torch.sum.

Each output element is the exact sum, rounded once. The tolerance budgets the additions in the compute format and the final rounding; the terms are input values, so they are not rounded on their own. A sum is one of matmul's exact sums underneath (the elements times ones), so it shares matmul's tolerances and normal answer.

Arguments#

Argument

Meaning

Default

x

the array to sum

required

axis

the dimensions to sum over: an int or a tuple of ints

None, every dimension

keepdims

keep the summed dimensions, with size 1

False

dtype

the output format, of the same kind as the inputs

the input format; int64 for integer and bool inputs

rounding

the rounding of the reference and of the final step into the output format

er.Rounding.NEAREST_EVEN

overflow

what a result past the output format's range becomes: "ieee", "saturate" or "nan" for floats; "wrap", "error" or "saturate" for integers

each format's own rule; integers wrap

Examples#

Code
import exactref as er

# A 2x3 float32 matrix
M = er.asarray([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]], "float32")

# The sum of every element, rounded once to float32
print("all:", er.sum(M).rounded.tolist())
# One sum per column
print("axis=0:", er.sum(M, axis=0).rounded.tolist())
# One sum per row, keeping the summed dimension with size 1
print("axis=1, keepdims:", er.sum(M, axis=1, keepdims=True).rounded.tolist())
# Both dimensions, named in a tuple
print("axis=(0, 1):", er.sum(M, axis=(0, 1)).rounded.tolist())
Output
all: 21.0
axis=0: [5.0, 7.0, 9.0]
axis=1, keepdims: [[6.0], [15.0]]
axis=(0, 1): 21.0

The order of additions changes a floating-point sum, which is what the tolerance accounts for; How tolerance works shows it on sums.

Integer and bool inputs#

Integer and bool inputs give int64, as PyTorch sums them:

Code
# The format of an int8 sum
print("int8 sum dtype:", er.sum(er.asarray([1, 2, 3], "int8")).rounded.dtype.name)
# A bool sum counts the True elements
print("bool sum:", er.sum(er.asarray([True, True, False])).rounded.tolist())
Output
int8 sum dtype: int64
bool sum: 2

PyTorch's sum(x, dtype=d)#

torch.sum(x, dtype=d) converts x to d before summing. That is er.sum(er.multiply(x, 1, dtype=d)), whose sum accumulates in d's compute format:

Code
# 300 bfloat16 ones
xb = er.asarray([1.0] * 300, "bfloat16")
# xb converted to float32 (times 1, rounded once), then summed
print("dtype:", er.sum(er.multiply(xb, 1, dtype="float32")).rounded.dtype.name)
Output
dtype: float32

Complex sums, empty sums#

A complex sum adds its real and imaginary parts as two separate sums. An empty sum is +0:

Code
# Real parts 1 + 3, imaginary parts 2 + -1
print("complex sum:",
      er.sum(er.asarray([1 + 2j, 3 - 1j], "complex64")).rounded.tolist())
# Array.zeros with the shape (0,): a float32 array with no elements
print("empty sum:", er.sum(er.Array.zeros("float32", (0,))).rounded.tolist())
Output
complex sum: (4+1j)
empty sum: 0.0

Checking PyTorch#

Code
import torch

# A 2x3 float32 tensor
t = torch.tensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6]])
# Could a correct implementation produce PyTorch's row sums? covers() answers
# per element, and all() asks whether every element is covered.
print("covered:", er.sum(t, axis=1).covers(t.sum(dim=1)).all())
Output
covered: True

Backward#

er.sum.backward(x, grad, axis=, keepdims=) returns grad_x: at every element of x, the gradient of the output it was summed into.

Code
# The gradient of each row sum of M: 10 for row 0, 20 for row 1
grad = er.asarray([10.0, 20.0], "float32")
# Each element of M gets the gradient of its row's sum
print("grad_x:", er.sum.backward(M, grad, axis=1).rounded.tolist())
Output
grad_x: [[10.0, 10.0, 10.0], [20.0, 20.0, 20.0]]

The backward has one output, grad_x, so rounded is the array itself.

Full specification#

er.sum#

sum(x, axis=None, *, keepdims: bool = False, dtype=None, rounding: Rounding = Rounding.NEAREST_EVEN, overflow: str | None = None) -> Result#

Sum over axis (None: every dimension; an int or a tuple of ints), keeping the reduced dimensions as 1 with keepdims.

Each output element is the exact sum rounded once to dtype (default: the input dtype; int64 for integer and bool inputs, as PyTorch sums them). tolerance() budgets accumulation in the compute format and the final rounding; by default normal accumulation (index order of the reduced elements, row-major, or 2 or 4 SIMD-style lanes), with standard="any_order" any order. Each term is an input value, so it is not rounded on its own. A complex sum adds its real and imaginary parts as two separate sums. An exact zero sum is +0 (-0 when rounding toward -inf); an empty sum is +0. Integer results wrap on overflow unless overflow says otherwise. x may also be an earlier result. PyTorch's sum(x, dtype=d) converts x to d before summing: that is sum(multiply(x, 1, dtype=d)), whose sum accumulates in d's compute format.

er.sum.backward#

backward(x, grad, *, axis=None, keepdims: bool = False, dtype=None, rounding: Rounding = Rounding.NEAREST_EVEN, overflow: str | None = None) -> Result#

Gradient of sum(x, axis, keepdims=keepdims) for the output gradient grad.

grad_x holds, at every element of x, the gradient of the output it was summed into, in x's dtype (rounded once if grad's dtype is wider); an exact zero keeps grad's sign, as in PyTorch. grad must have the output's shape. x and grad may also be earlier results.

See also#