The test engineer looks at the screen again.

FAIL.

Not marginal.

Not “within measurement uncertainty.”

Not a logging glitch.

The system was required to complete the operation in less than 500 milliseconds.

It took 684 milliseconds.

Someone reruns the test.

691 milliseconds.

Again.

679 milliseconds.

The room gets quieter.

A software engineer opens the profiler. A systems engineer pulls up the specification. Someone asks whether the network configuration changed. The project manager wants to know whether this affects the delivery date.

Then the inevitable question arrives:

“So... what is wrong with the design?”

Maybe something is.

But we do not know that yet.

A failed verification test tells you something very specific:

The observed evidence did not satisfy the defined acceptance criteria under the conditions of that verification.

That is important.

It is not the same thing as proving the design is wrong.

The failure could be in the product.

It could also be in the test procedure.

Or the verification method.

Or the test configuration.

Or the interpretation of the requirement.

Or the requirement itself.

And if you keep tracing far enough backward, you may discover something even more uncomfortable:

The system may be doing exactly what the stakeholder actually needs—and failing a requirement that never should have existed in its current form.

That is why good systems engineers do not stop at:

Why did the test fail?

They ask:

What exactly failed—and where did the expectation being tested come from?

Imagine This Happening on a Real Program

Consider a surveillance system that receives a command from an operator, processes it, drives a sensor to a new position, and reports that the requested position has been reached.

There is a system requirement:

SYS-PERF-042: The system shall respond to an operator pointing command within 500 ms.

It looks reasonable.

It has an ID.

It contains a number.

It contains a shall.

Someone assigned Test as the verification method.

A test procedure was written.

The test team sends a pointing command and measures the time until the system reports completion.

Result:

684 ms.

Requirement:

≤ 500 ms.

Verification result:

FAIL.

At first glance, this seems simple.

The system is too slow.

Engineering needs to make it faster.

Except good verification failure analysis does not begin by modifying the design.

It begins by walking backward through the evidence.

Step 1: Did the System Actually Fail the Test?

This sounds almost insultingly obvious.

It isn't.

Before opening a change request against the product, establish that the test result itself is trustworthy.

Ask what was actually measured.

Where did the timer begin?

When the operator clicked the command?

When the workstation transmitted the packet?

When the system received it?

When the control loop accepted it?

And where did the timer stop?

When physical motion began?

When the requested line of sight was reached?

When the position sensor reported it?

When the software declared the motion complete?

When the acknowledgement travelled back across the network?

When the workstation finally displayed the updated status?

Those are not the same measurement.

Our apparently simple requirement contains a dangerous word:

respond.

What does respond mean?

The test procedure has silently answered that question by choosing two measurement points.

Perhaps it measures:

Command generated at workstation → Completion status displayed at workstation

But perhaps the design team interpreted the requirement as:

Command received by controller → Motion initiated

The same system could satisfy one interpretation easily and fail the other badly.

Nothing about the physical design changed.

The meaning of the requirement did.

This is precisely why requirement quality and verifiability matter. NASA's requirements guidance asks whether a requirement is clear, unambiguous, measurable, realistic, and stated precisely enough to establish verification success criteria. It also specifically warns against terms whose interpretation prevents objective verification [1].

A requirement does not become unambiguous simply because somebody added a number to it.

Step 2: Was the Test Procedure Actually Testing the Requirement?

Now suppose everyone agrees on what "response" means.

The next question is whether the procedure actually creates the conditions specified—or intended—by the requirement.

Look deeper into the test configuration.

The workstation was connected through a development Ethernet switch.

Debug logging was enabled.

A packet capture was running.

A diagnostic telemetry stream was transmitting at maximum rate.

The test was executed using engineering-development software rather than the release candidate.

The system had been operating for four hours.

The position command required nearly maximum travel.

The temperature was close to the lower operating limit.

None of those conditions appears in the requirement.

Some may be irrelevant.

Some may explain everything.

This is where a failed test can become surprisingly political.

The design team says:

“Nobody uses the system like that.”

The test team responds:

“The requirement doesn't say we can't.”

And technically, both may be right.

NASA's verification guidance explicitly identifies the verification environment, procedures, configuration, enabling equipment and applicable requirements baseline as part of the verification problem. Verification records are expected to capture not just a pass/fail conclusion, but the procedures, environments, assumptions, results, anomalies and configurations involved [2].

The correct question is therefore not:

Did it take 684 ms?

That part is already known.

The question is:

Does 684 ms under this exact test configuration demonstrate non-compliance with SYS-PERF-042?

Those are very different questions.

Step 3: Was Test Even the Right Verification Method?

Engineers frequently default to testing because testing feels like the strongest form of evidence.

Build it.

Run it.

Measure it.

Done.

But verification does not mean "perform a test."

Verification means obtaining objective evidence that the specified requirement has been satisfied.

Depending on the requirement, that evidence might come from:

• Test
• Analysis
• Inspection
• Demonstration
• Or a justified combination of methods

NASA uses these same core verification methods and emphasizes that method selection should match the characteristic being verified [2].

Imagine our 500 ms requirement applies across a huge operating envelope:

• different network loads,
• multiple software configurations,
• different target positions,
• temperature extremes,
• voltage tolerances,
• processor loads,
• command sequences,
• payload configurations.

One physical test at one operating point might not prove compliance across that envelope.

Running thousands of physical test combinations may also be impractical.

The more defensible verification strategy could be:

Test + Analysis

Use instrumented testing to characterize real system timing.

Use analysis to decompose the latency contributors and demonstrate worst-case performance across the required operating envelope.

Perhaps the original verification plan never considered that.

So the problem is no longer:

“The product failed verification.”

It becomes:

“Our chosen verification strategy does not adequately establish compliance.”

That is a very different corrective action.

Step 4: Now Read the Requirement Again

Return to SYS-PERF-042:

The system shall respond to an operator pointing command within 500 ms.

We already have one problem:

What is "respond"?

But there are more.

From what starting state?

Does the system have to be stationary?

Can another command already be executing?

What happens during initialization?

Does degraded operation count?

For what command?

A 0.1-degree adjustment?

A 180-degree reposition?

Under what conditions?

Nominal temperature?

Full environmental range?

Maximum network traffic?

Normal payload configuration?

Measured where?

Operator workstation?

Network interface?

Controller input?

Mechanical output?

500 ms to do what?

Acknowledge receipt?

Begin movement?

Reach the requested position?

Settle within pointing tolerance?

Report completion?

The requirement that looked perfectly testable has become much less impressive.

This is one of the most expensive illusions in systems engineering:

A requirement can be measurable without being well defined.

“Less than 500 ms” gives us a threshold.

It does not tell us whether we are measuring the right thing.

ISO/IEC/IEEE 29148 treats requirements engineering as a lifecycle process and defines requirements characteristics and associated information rather than treating a specification as merely a collection of shall-statements [3]. The current 2018 edition was confirmed by ISO in 2024, while a new edition is under development as of 2026.

The discipline exists because words eventually become architecture, interfaces, software, hardware—and pass/fail decisions.

Ambiguity does not disappear downstream.

It becomes more expensive.

Step 5: Where Did 500 ms Come From?

This is my favourite question to ask about suspicious requirements:

Why that number?

Not:

“What document is it in?”

Not:

“Who approved it?”

Why is the requirement 500 milliseconds?

The systems engineer searches the history.

Nothing.

There is no rationale attached.

The parent requirement says only:

The system shall provide responsive operator control.

A review presentation from two years ago contains a bullet:

Target operator response: ~0.5 s

Someone remembers a meeting where a customer said:

“We don't want the display to feel sluggish.”

At some point, approximately half a second became:

shall respond within 500 ms

The approximation disappeared.

The context disappeared.

The original meaning of response disappeared.

But the number survived.

Then something remarkable happened.

Once it entered the specification, the project began treating 500 ms as a physical truth.

Software architecture was evaluated against it.

Processing budgets were allocated around it.

Test equipment was purchased to measure it.

A formal verification procedure was written around it.

And now engineers may spend weeks redesigning the system to save 184 milliseconds.

Maybe that is necessary.

Maybe it isn't.

Without the rationale, nobody knows.

NASA explicitly recommends capturing the reason for a requirement, its assumptions, its relationship to operational use, relevant design constraints and links to sources such as the ConOps or parent requirement. Its guidance makes a simple point: the reason for a requirement is often not obvious and can be lost if it is not recorded [4].

That lost "why" becomes painfully important when verification fails.

Step 6: Find the Assumption Nobody Wrote Down

Eventually someone finds the engineer who originally worked on the requirement.

They remember the discussion.

The customer was not worried about how long it took the sensor to complete a large reposition.

They were worried about something much narrower:

When the operator entered a command, there should be immediate visual confirmation that the command had been accepted.

The original assumption was that command acknowledgement and motion completion were effectively the same thing because the early prototype moved quickly enough that nobody cared about the distinction.

The production design changed.

The sensor became heavier.

Its pointing accuracy improved.

The control system now deliberately uses a smoother motion profile to reduce overshoot.

The old assumption is no longer true.

And it was never documented.

The requirement survived.

Its assumption did not.

Now the 684 ms failure looks very different.

The production system may acknowledge the command in 72 ms.

It begins moving in 110 ms.

It reaches the requested position in 684 ms.

The stakeholder may be perfectly happy.

The test may still correctly declare the requirement failed.

Those two statements can coexist.

That is the difference between verification and validation.

Verification asks whether the realized system satisfies its specified requirements.

Validation asks whether the system fulfills stakeholder needs and intended use.

NIST's systems engineering guidance expresses the distinction clearly: verification provides objective evidence that specified requirements are fulfilled, while validation concerns whether the realized system fulfills its intended mission or stakeholder objectives in its operational environment [5].

A system can therefore fail verification against a bad requirement while still satisfying the underlying stakeholder need.

It can also pass verification against every written requirement and still fail validation.

Neither outcome should make a systems engineer comfortable.

Step 7: Go All the Way Back to the Stakeholder Need

The team finally interviews operators.

Not managers speaking for operators.

Not a two-year-old meeting minute.

The people actually performing the task.

They discover that operators care about three different behaviours.

1. Command acknowledgement

They need immediate feedback that the system received the command.

Anything above roughly 200 ms begins to feel uncertain.

2. Motion initiation

They want visible evidence that repositioning has started quickly.

3. Final positioning

The acceptable completion time depends heavily on the size of the requested move.

Nobody expects a 180-degree reposition to complete as quickly as a tiny correction.

Now compare those needs with the original requirement:

The system shall respond to an operator pointing command within 500 ms.

The requirement collapsed three different behaviours into one undefined word.

The test then selected one interpretation—final completion—and turned it into a binary engineering decision.

The test did exactly what it was written to do.

The test procedure may not be wrong.

The product may not be wrong.

The translation between stakeholder need and technical requirement is where the defect occurred.

And it took a failed verification event months or years later to expose it.

The Corrective Action Is Not “Change the Requirement Until the Test Passes”

There is an obvious danger here.

If engineers discover that a requirement is questionable immediately after the product fails it, changing the requirement can look suspiciously like moving the goalposts.

Sometimes that is exactly what is happening.

A difficult requirement should not be weakened simply because the design missed it.

The correct response is disciplined change control.

The team needs to establish:

1. What the stakeholder actually needs.
2. Why the current requirement exists.
3. Whether the current requirement correctly represents that need.
4. Whether the verification approach correctly evaluates the requirement.
5. What system elements depend on the requirement.
6. What safety, contractual, regulatory or interface consequences a change may have.
7. Whether changing the requirement creates risk elsewhere.
8. Who has authority to approve the change.

Only then should the baseline move.

You do not change the requirement because the test failed.

You change the requirement if the engineering evidence demonstrates that the requirement is wrong.

There is an enormous difference.

The Revised Requirement Might Look Very Different

After stakeholder validation and engineering analysis, SYS-PERF-042 might disappear entirely.

In its place:

SYS-PERF-042A: The operator workstation shall provide visual acknowledgement of a valid pointing command within 200 ms of command initiation under nominal operating conditions.

And:

SYS-PERF-042B: The pointing subsystem shall initiate commanded motion within 250 ms of receiving a valid pointing command.

And perhaps:

SYS-PERF-042C: The pointing subsystem shall achieve the commanded line-of-sight position within the time envelope defined in Table X as a function of commanded angular displacement, payload configuration and operating temperature.

Now we have very different engineering questions.

We can choose appropriate verification methods.

We can define measurement points.

We can identify initial conditions.

We can determine the operating envelope.

We can establish test tolerances.

We can trace each technical threshold back to the behaviour operators actually care about.

And if one of those verification activities fails, the team has a much better chance of determining why.

A Failed Test Should Trigger a Trace, Not a Guess

When verification fails, teams often jump directly from:

Test Failure

to:

Design Fix

But the real diagnostic path should look more like this:

Observed failure

↓
Is the result valid?

↓
Was the correct configuration tested?

↓
Did the procedure implement the intended verification?

↓
Was the verification method appropriate?

↓
Are the acceptance criteria correct?

↓
Is the requirement clear and technically valid?

↓
What assumptions support the requirement?

↓
What parent requirement or operational objective produced it?

↓
What stakeholder need are we actually trying to satisfy?

And then—and only then—you work forward again.

Stakeholder need

↓
Validated requirement

↓
Verification strategy

↓
Procedure / analysis / inspection / demonstration

↓
Design implementation

↓
Objective evidence

That backward-and-forward movement is what traceability is supposed to make possible.

NASA describes bidirectional traceability as maintaining links forward to subsequent lifecycle artifacts and backward to preceding ones. It notes that traceability helps identify missing implementation, unjustified functionality and the impacts of changes [6].

Traceability is therefore not primarily about producing a colourful matrix for a design review.

It is how you answer questions when reality stops matching the spreadsheet.

This Is Where Engineering Teams Lose Days

The process above sounds sensible when written sequentially.

Real projects rarely look like that.

The requirement is in one tool.

The stakeholder discussion is in meeting minutes.

The assumption exists in an engineer's notebook.

The verification method is in an RVTM spreadsheet.

The procedure is a Word document.

The test result is in another directory.

The anomaly is in an issue tracker.

The design decision is buried in a review presentation.

The requirement changed three times, and nobody is certain which revision the test procedure references.

Now somebody has to reconstruct the engineering story manually.

That is often the real bottleneck after a verification failure.

Not generating another test result.

Reconstructing why everything exists.

Where Ngenaire Can Help

This is exactly the kind of engineering problem Ngenaire is being built to reduce.

The useful part is not simply having an AI generate another test procedure.

The bigger opportunity is keeping the engineering chain connected.

Ngenaire brings requirements management, baselines, engineering change control, verification artifacts and an RVTM into a connected engineering workspace. Its current platform also supports test plans and procedures, execution history, requirement quality analysis and forward/backward trace relationships across engineering artifacts [7].

So when a verification fails, the investigation can begin from the failed evidence and move outward.

Which requirement did this test verify?

What version of the requirement?

What was the acceptance criterion?

What assumptions or project decisions are relevant?

What parent need caused the requirement to exist?

Which other tests verify it?

What requirements depend on it?

What design elements could be affected if it changes?

Are there open risks connected to the same behaviour?

That context is where AI becomes genuinely useful in engineering.

Instead of asking an engineer to manually search through hundreds of artifacts, an AI-assisted engineering environment can help surface the connected information, identify missing links, flag ambiguous or unverifiable requirements, summarize relevant changes, and support impact analysis.

The engineer still decides whether the design is wrong.

The engineer still decides whether the requirement is wrong.

The engineer still owns the technical judgement.

AI reduces the archaeology required to get to the evidence.

That distinction matters.

The goal should never be:

“Let AI decide why the test failed.”

The goal should be:

“Give the engineer the complete technical context needed to make that decision.”

The Most Dangerous Verification Result Is Sometimes PASS

There is another reason this matters.

Imagine the original system had completed the motion in 480 ms.

The test would have passed.

Nobody would have investigated the meaning of "respond."

Nobody would have discovered the undocumented assumption.

Nobody would have spoken to the operators.

The requirement would be marked:

VERIFIED

The RVTM cell would turn green.

And everyone would move on.

That is why passing verification should never be confused with proving that the engineering intent was correct.

A verification process can establish that you built what the specification asks for.

It cannot rescue a specification that asks for the wrong thing.

NASA's requirements-validation guidance explicitly warns that merely managing requirements does not ensure those requirements are correct and notes the need to revalidate them as they change through the lifecycle [8].

The failed test at least forced the team to ask the question.

The passing test might never have done so.

When the Test Fails, Resist the Urge to Fix Something Immediately

Engineers are problem solvers.

We see red and want to make it green.

That instinct is useful.

But verification failures require restraint.

Before modifying code, machining another part, changing a controller gain or asking a supplier for a redesign, ask:

What exactly have we proven?

Sometimes the answer will be:

The design is wrong.

Good. Fix it.

Sometimes it will be:

The test procedure is wrong.

Fix that.

Sometimes:

We tested the wrong configuration.

Correct it and rerun.

Sometimes:

The verification strategy cannot establish compliance.

Rework the strategy.

Sometimes:

The requirement is ambiguous.

Clarify it through controlled requirements change.

Sometimes:

The requirement is precise, but it does not represent the stakeholder's actual need.

That is the uncomfortable one.

Because by the time you discover it, a considerable amount of technically excellent engineering may already have been built around the wrong expectation.

But that is still better than continuing.

Verification Is Where the Engineering Story Has to Survive Contact With Evidence

A requirement is written months or years before somebody finally asks the physical system to prove it.

Between those two events, an enormous amount happens.

People leave.

Designs change.

Suppliers change.

Interfaces evolve.

Assumptions become forgotten.

Trade studies disappear into archives.

Software gets rewritten.

Operational concepts mature.

And eventually someone presses RUN.

The test does not care how many reviews the requirement passed.

It does not care how polished the specification looks.

It produces evidence.

When that evidence contradicts the expected result, the engineering organisation gets a choice.

It can treat the failure as a local defect and immediately start changing the product.

Or it can use the failure for what verification is exceptionally good at doing:

forcing the engineering logic to defend itself.

Why this procedure?

Why this method?

Why this threshold?

Why this requirement?

Why this assumption?

Why this design decision?

Why does the stakeholder need it?

If your engineering information is traceable enough to answer those questions quickly, a failed test is manageable.

If it isn't, the failure becomes an archaeological expedition.

And sometimes, after tracing everything back to the beginning, you discover that the system was never the thing that failed.

The requirement did.

References

[1] National Aeronautics and Space Administration, “Appendix C: How to Write a Good Requirement,” NASA Systems Engineering Handbook. Accessed: Aug. 24, 2026. NASA — How to Write a Good Requirement

[2] National Aeronautics and Space Administration, “5.3 Product Verification,” NASA Systems Engineering Handbook. Accessed: Aug. 24, 2026. NASA — Product Verification

[3] ISO, ISO/IEC/IEEE 29148:2018—Systems and Software Engineering—Life Cycle Processes—Requirements Engineering, 2nd ed., Nov. 2018, confirmed 2024. Accessed: Aug. 24, 2026. ISO/IEC/IEEE 29148:2018

[4] National Aeronautics and Space Administration, “4.2 Technical Requirements Definition,” NASA Systems Engineering Handbook. Accessed: Aug. 24, 2026. NASA — Technical Requirements Definition

[5] R. Ross, M. Winstead, and M. McEvilley, Engineering Trustworthy Secure Systems, NIST Special Publication 800-160 Vol. 1 Rev. 1, National Institute of Standards and Technology, Nov. 2022. NIST SP 800-160 Vol. 1 Rev. 1

[6] National Aeronautics and Space Administration, “SWE-047 — Traceability Data,” NASA Software Engineering Handbook. Accessed: Aug. 24, 2026. NASA — Traceability Data

[7] Ngenaire, “Ngenaire — AI-Augmented Engineering,” Accessed: Aug. 24, 2026. Ngenaire

[8] National Aeronautics and Space Administration, “SWE-055 — Requirements Validation,” NASA Software Engineering Handbook. Accessed: Aug. 24, 2026. NASA — Requirements Validation