A question path · Evidence

Tests prove the system is correct. Do they?

A passing test proves a particular expectation under particular conditions; the rest of the world has not taken the test.

The satisfying green suite

A suite passes after a change. That is good news. It says the assertions executed in that environment found no mismatch. It does not prove the assertions covered every important behavior, that the production configuration matches the test, or that the intended user can complete the task.

Tests can mirror the implementation's mistaken assumption. Two components may agree on the wrong field name. A mocked external service may return the answer the developer hoped to receive. An untested restart can leave a live process running old code.

Inspect its reach

The PHS enforcement standard says a passing suite is insufficient if it does not prove the applicable obligations. It calls for evidence manifests, contract tests, runtime probes, deployment gates, and recovery receipts. The Memory API test plan distinguishes service health from registry, gateway, and completeness checks. Each test group proves its declared scope, no more.

The claims discipline helps here: a tested behavior may be confirmed for its tested version and conditions; a broader expectation can remain inferred or not checked. That is not an insult to the test. It is a way to protect the value it actually earned.

Pause here: What did the tests actually exercise, and which critical route would still need an independent check?

Let a test invite the next check

A strong verification story chooses meaningful failure cases, checks output against an independent oracle, runs through the real route when possible, and reads back the changed state. It records the version, environment, and remaining gaps so another person can repeat it.

A lantern illuminates a useful circle of ground. More lanterns can widen the circle. None makes the unlit path disappear, and pretending otherwise is how an ordinary trip becomes a fall.

My answer: Tests support bounded claims about checked behavior. Correctness of the whole system needs coverage, independent verification, runtime readback, and honest residual limits.

Sources and limits

Examples and the red guide’s dialogue are imagined. Sources below are internal PHS documents. Proposals, draft math and historical notes do not establish live implementation or effectiveness. Interpretive conclusions are Virgil’s and remain open to critique.

  • PHS encyclopedia: PHS Enforcement, Evidence, and Runtime Assurance Standard (governing standard)
  • PHS encyclopedia: T01, Memory API and Memgraph Contract Validation (test plan)
  • PHS encyclopedia: G05, Claims Discipline (claims discipline draft)