Generate golden reference values#

The problem#

You are testing an implementation that cannot call Python at test time: hardware, an FPGA design, a compiler's code generator, or a library in C, Rust or another language. You need fixed test vectors: inputs, expected outputs and how far each output may be from the expected one.

Why the usual approach falls short#

Expected outputs computed with a higher-precision library are not exact, and they come without a bound on what a correct implementation can output. A test against them needs a tolerance chosen by hand, in every language that runs the test.

The solution#

Generate the vectors once with ExactRef, and store the bits: the inputs' encodings, the reference's encodings, and the lowest and highest finite output a correct implementation can produce, plus the infinities and NaN it may produce. A checker in any language then compares numbers, with no tolerance to choose.

Code
import json
import exactref as er

# The inputs: a 1x3 and a 3x1 float32 matrix
A = er.asarray([[0.1, 0.2, 0.3]], "float32")
B = er.asarray([[1.0], [3.0], [7.0]], "float32")
# ExactRef's reference for their product
r = er.matmul(A, B)
# The lowest and highest finite output of a correct implementation
low, high = r.finite_range()

# Every element's raw encoding as 8 hex digits, in row-major order
def bits(array):
    return [f"{b:08x}" for b in array.encodings()]

# One test vector
case = {
    # The operation and the format
    "op": "matmul", "dtype": "float32",
    # The inputs' shapes and encodings
    "A": {"shape": list(A.shape), "bits": bits(A)},
    "B": {"shape": list(B.shape), "bits": bits(B)},
    # The reference, and the lowest and highest correct finite output
    "reference": bits(r.rounded), "low": bits(low), "high": bits(high),
    # Per element, whether a correct implementation may output +inf, -inf or NaN
    "may_be": {kind: r.may_be(kind).scalars() for kind in ("+inf", "-inf", "nan")},
}
print("test vector:")
print(json.dumps(case, indent=1))
Output
test vector:
{
 "op": "matmul",
 "dtype": "float32",
 "A": {
  "shape": [
   1,
   3
  ],
  "bits": [
   "3dcccccd",
   "3e4ccccd",
   "3e99999a"
  ]
 },
 "B": {
  "shape": [
   3,
   1
  ],
  "bits": [
   "3f800000",
   "40400000",
   "40e00000"
  ]
 },
 "reference": [
  "40333334"
 ],
 "low": [
  "40333332"
 ],
 "high": [
  "40333336"
 ],
 "may_be": {
  "+inf": [
   false
  ],
  "-inf": [
   false
  ],
  "nan": [
   false
  ]
 }
}

The checker decodes each output and tests low <= output <= high, or, for an inf or NaN output, the matching may_be flag. In Python, as a model for the checker in your language:

Code
import struct

# The float32 that 8 hex digits encode, most significant byte first
def decode(hex_bits):
    return struct.unpack(">f", bytes.fromhex(hex_bits))[0]

# Whether every output lies between its low and high value
def check(case, outputs):
    return all(decode(lo) <= out <= decode(hi)
               for out, lo, hi in zip(outputs, case["low"], case["high"]))

# The reference itself, and the float32 six values above it
print("reference passes:", check(case, [2.8000001907348633]))
print("6 floats above passes:", check(case, [2.8000016212463379]))
Output
reference passes: True
6 floats above passes: False

Store the format, the operation's options and the standard (normal or any order) with each vector, so the checker applies the same assumptions. For formats narrower than 32 bits, store narrower hex strings.