Breakbench
degraded-state testing

The measured object is violation probability as a function of fault severity, p̂ with a Wilson interval at n runs per cell. Two predicates are recorded separately for every run: a safety-invariant violation (operating region left, or obstacle clearance below the geometric threshold) and a mission failure (true pose did not enter the goal region before timeout). They are not folded into one score.

The robot's own success flag is stored as self-report and compared with the oracle in two confusion matrices. It is never an input to the verdict.

Phase 0 has two software axes, applied by fault stages between the sensors and the navigator. In the Gazebo reference they were ROS 2 nodes on the scan and odometry topics; in the Breakbench simulator they are in-process stages on the same clock. Each run records the requested value and the delivered value; the difference is data.

perception latency

scan stream held for the requested delay; delivered mean and message count logged

odometry scale bias

odometry position scaled about its origin; delivered mean out_x/in_x logged

latency grid

0, 0.15, 0.4, 0.8 s

bias grid

0, 0.08, 0.2, 0.4

repeats per cell

5

controller under test

simple_navigator

plants

Breakbench simulator (Rust, kinematic, rigid contact, ray-cast lidar) and Gazebo Harmonic 8.11 as reference

In simulation the oracle is the Gazebo ground-truth pose on /oracle/odom. Invariants are predicates over that stream only: operating_region_containment, obstacle_clearance, and mission_completion_true_pose. The evaluator does not read /odom, /tf, AMCL, or the navigator's result. A verdict isolation check replays measured poses with injected self-report success and failure and requires the same verdict.

A positive canary drives the robot open-loop into the obstacle to show the oracle can report a collision. It is labeled harness-only and excluded from corpus counts.

Breakbench simulator corpus

grid

80/80 exploration runs

manifests

151

nominal

n = 25, spread converged

safety matrix

success+safe 149, success+violation 0, n = 149

mission matrix

success+complete 74, success+mission_failure 75, n = 149

Gazebo Harmonic reference corpus

grid

80/80 exploration runs

manifests

143

nominal

n = 15, spread converged

safety matrix

success+safe 139, success+violation 0, n = 139

mission matrix

success+complete 64, success+mission_failure 75, n = 139

On both plants, safety-invariant violation was 0 in every sampled cell. Mission failure followed odometry bias: 0 at bias 0, 5/5 at each nonzero bias, at every sampled latency. In those runs the navigator reported success from biased odometry while the true pose stopped short of the goal. Latency alone did not produce mission failure in the sampled range. The Rust analysis code reproduces the Python analysis over the Gazebo corpus with zero value differences.

Individual trajectories and both confusion matrices are reproduced in section 6.

Simulation only, one corridor mission, one controller, two simulators that agree with each other and have not been checked against a physical robot. The Breakbench simulator is kinematic with rigid contact, not a dynamics engine, and its run-to-run spread is a declared noise setting rather than a measured property. The probability threshold q for a scalar margin is undeclared, so no margin m_i(q) is computed and no sub-margin joint violation is claimed. No statistical interaction model was declared or tested. Hardware transfer has not started and is an open question, not an assumption.

Nothing here is a certification or a claim of compliance with any standard.