Your Test Passed. But What Exactly Did It Pass?

The verification matrix says PASS.

Then, six months later, someone asks a perfectly reasonable question:

“Which software build was installed?”

The room goes quiet.

The report identifies the test date and the unit serial number. The procedure is attached. Screenshots show the expected response.

But the firmware was updated twice that week. The test script was edited on a shared drive. One requirement changed shortly afterward. Nobody can prove whether the passing result belongs to the product that is about to be delivered.

The test may have succeeded.

The evidence did not survive.

A pass is a relationship, not a sticker

A verification result is often treated like a permanent property of a requirement:

Requirement SYS-142: Passed.

That statement is incomplete.

A more defensible result says that a defined product configuration was evaluated against a defined requirement baseline using an approved method, procedure and test environment—and that the resulting evidence satisfied the acceptance criteria.

NASA describes product verification as confirmation that an end product conforms to its requirements or specifications. Its guidance identifies the product, verification plan, specified requirements baseline and enabling products such as test fixtures as inputs to the process [1].

Change any one of those inputs and the old result may no longer answer the question being asked.

The requirement might change. The design might change. The test equipment might be recalibrated. A simulation model might be updated. A defect could be corrected in one software branch but not another.

The word PASS does not reveal any of this.

Build an evidence chain

For each formal verification result, a reviewer should be able to follow a short, unbroken chain:

1. What was evaluated?
   Identify the hardware serial number, software or firmware build, model revision, configuration options and relevant supplier versions.
2. What was it evaluated against?
   Record the applicable requirement revision or baseline—not merely the current text displayed in the requirements tool.
3. How was it evaluated?
   Identify the approved procedure, analysis method, inspection criteria or demonstration steps and their revisions.
4. Under what conditions?
   Capture the test facility, equipment, calibration status, environmental conditions, input data and relevant operator-controlled settings.
5. What evidence was produced?
   Preserve raw measurements, logs, observations, analysis outputs and the calculation that turned them into a result.
6. Who accepted the conclusion?
   Record the reviewer, date, deviations, anomalies, limitations and final disposition.

This does not mean every test needs a mountain of paperwork. It means the evidence must retain enough context for another qualified person to understand what was demonstrated.

Control the configuration behind the evidence

Configuration management is sometimes treated as document administration performed after the “real engineering” is finished.

Verification exposes why that view is dangerous.

NASA’s configuration-management guidance says the practice should keep a product’s configuration known and consistent with the information describing it. It also calls for unique identifiers, historical configuration records and traceability of changes, deviations and waivers [2].

Those controls protect the meaning of a test result.

Consider an illustrative vibration test performed on Unit 03:

• The test report identifies the unit.
• The unit’s internal bracket was replaced before testing.
• The drawing was updated afterward.
• The production units use a different fastener from the tested assembly.

The vibration profile may have been executed perfectly. The real question is whether Unit 03 represents the configuration now being claimed as verified.

That question cannot be answered from the acceleration plot alone.

“Same test” does not always mean equivalent evidence

Teams are usually careful when the product changes. They can be less careful when the change is in the verification system.

Suppose a requirement is verified by analysis. The product design remains unchanged, but the team updates:

• The simulation model.
• Material properties.
• Boundary conditions.
• A data-processing script.
• The solver version.
• The acceptance calculation.

Is the earlier result still valid?

Possibly—but the conclusion needs an engineering rationale.

Software supply-chain guidance offers a useful parallel. The SLSA specification defines provenance as verifiable information about where, when and how a software artifact was produced. Build provenance connects an output to the source and process that produced it [3].

Physical-product teams need their own version of that question:

Can we trace this verification result back to the exact inputs and process that produced it?

A polished PDF is not provenance if its underlying files, revisions and assumptions cannot be reconstructed.

Separate three different decisions

A single green cell often hides three distinct judgments:

• Result: Did the observed evidence satisfy the acceptance criteria?
• Applicability: Does that evidence apply to the configuration now under review?
• Coverage: Is the evidence sufficient to close the requirement completely?

These judgments can produce different answers.

A thermal test might pass its stated criteria while applying only to one operating mode. An inspection might confirm the drawing dimensions but not the performance requirement. A qualification result might remain technically sound while no longer applying to a redesigned component.

Instead of forcing every case into PASS or FAIL, allow statuses such as:

• Passed and applicable.
• Passed with limitation.
• Passed for an earlier configuration.
• Superseded.
• Retest or re-analysis required.
• Blocked pending anomaly disposition.

The purpose is not to make the dashboard more complicated. It is to stop an attractive summary from overstating the evidence.

Decide what a change invalidates

Not every engineering change requires repeating every verification activity.

The team does, however, need a deliberate impact assessment.

For each approved change, ask:

1. Which requirements, interfaces and failure modes could be affected?
2. Which previous verification results depend on the changed item?
3. Were any test tools, models or assumptions also changed?
4. Can existing evidence remain applicable with a documented rationale?
5. What must be repeated, supplemented or reopened?
6. Who has authority to accept that determination?

Record the answer beside the evidence it affects.

“Retest not required” is an engineering conclusion. It deserves the same visibility as “retest required.”

Where Ngenaire helps

Ngenaire’s published platform includes requirements baselines, test plans and procedures, a coverage matrix and a full requirements verification traceability matrix. It also provides a pass/fail/blocked execution trail per build label, rolled up to coverage [4].

Those capabilities can help teams connect a result to the build and requirement it supports, retain execution history and expose verification gaps.

They do not determine whether two configurations are technically equivalent. That remains an engineering judgment supported by configuration analysis, domain expertise and appropriate approval.

The practical value is preserving the evidence chain so that judgment can be reviewed instead of reconstructed from memory.

Ask one more question before closing the requirement

A passing test is good news.

But before turning the cell green, ask:

If someone challenged this result a year from now, could we prove exactly what passed—and whether that result still applies?

If the answer depends on finding the right person, shared-drive folder or unlabeled test unit, the requirement may be marked complete while the verification argument remains unfinished.

What part of your verification evidence is hardest to reconstruct: the tested build, the requirement baseline, the test setup or the decision that accepted the result?

References

[1] National Aeronautics and Space Administration, “5.3 Product Verification,” NASA Systems Engineering Handbook, updated Jul. 26, 2023. Accessed: Sep. 17, 2026.

[2] National Aeronautics and Space Administration, “6.5 Configuration Management,” NASA Systems Engineering Handbook, updated Jul. 26, 2023. Accessed: Sep. 17, 2026.

[3] Supply-chain Levels for Software Artifacts, “Provenance,” SLSA Specification v1.2. Accessed: Sep. 17, 2026.

[4] Ngenaire, “We Solve Hard Engineering Problems,” platform features. Accessed: Sep. 17, 2026.