The Fault Cleared. Why Hasn’t the System Recovered?

The cable is reconnected. The controller is responding. The alarm has cleared.

But the system still will not run.

One subsystem is waiting for initialization. Another is holding yesterday’s command. The operator presses reset, then tries a power cycle.

Consider this illustrative commissioning scene. The fault response worked: the equipment detected a problem and stopped. What remains uncertain is how it gets back to useful operation.

A restart is an action. Recovery is an outcome.

That distinction deserves a place in the architecture—not just a troubleshooting note written after delivery.

Surviving the fault is only half the story

Google’s published SRE lessons distinguish resilience testing from recovery testing: enduring a disruption is different from returning to normal operation after a complete shutdown. Both need attention. [1]

The same question is useful beyond online services.

A controller may correctly inhibit its outputs after losing feedback. A measurement system may correctly flag its data as invalid. Neither result establishes what happens when the missing input returns.

Does operation resume automatically? Is acknowledgement required? Must the system reacquire a reference? Which old commands remain valid?

These are design decisions.

“Fault cleared” describes the condition that triggered the response. It does not necessarily describe the readiness of everything affected by it.

Design the return journey

For an illustrative automated inspection station, imagine that communication with the positioning controller disappears halfway through a scan.

The fault response inhibits further motion. Communication then returns.

Before resuming, the design needs answers to several different questions:

• Position: Is the current position known and trustworthy?
• Commands: Could a buffered command execute after reconnection?
• Data: Which measurements belong to the interrupted scan?
• Authority: Can the controller resume, or must an operator authorize it?
• Readiness: What evidence confirms that the station can perform its next operation?

The appropriate answers depend on the equipment and its risk assessment. Automatic restart is not a universal goal.

The useful habit is to review the entire journey: normal operation, fault response, degraded or inhibited operation, recovery checks, and permitted return.

For each transition, identify its trigger, prerequisites, responsible component and observable result.

A mode name alone—“safe,” “standby,” “recovering”—does not tell another engineer what the system may actually do.

Define what “recovered” means

A green connection indicator is weak evidence if the service the user needs remains unavailable.

For the inspection station, a proposed recovery endpoint might be:

The station has re-established valid position feedback, resolved the interrupted scan, rejected obsolete commands and demonstrated readiness for a newly authorized scan.

That is an illustrative acceptance concept, not a universal requirement. Timing limits, tolerances and permitted operator actions still need to be derived for the application.

The important improvement is that the endpoint describes useful capability.

Measure the interval that matters to the user. If communication returns quickly but initialization and operator intervention take much longer, reporting only reconnection time hides most of the interruption.

Also distinguish:

• Recovery to limited functionality.
• Recovery to full functionality.
• Continued inhibition pending maintenance.

All three can be legitimate outcomes. They should not share an ambiguous “OK” indicator.

Check whether recovery depends on what failed

A recovery mechanism has dependencies of its own.

The remote reset may use the unavailable network. Diagnostic records may disappear during the reset. The operator may need credentials from a service affected by the outage.

Google documents an incident in which teams’ expected communication tools depended on services disrupted by that same incident. Its lesson was to provide and test independent backup communication channels. [1]

For your design, walk through recovery with the failed dependency still unavailable.

What remains accessible? What state survives? Can the operator distinguish “not ready” from “not responding”?

NASA’s Small Satellite Reliability Initiative knowledge base recommends preparing telemetry-trending tools before faults occur and retaining a ground test setup to investigate problems and try solutions. [2]

The practical implication is straightforward: decide what evidence recovery will need while you can still design that evidence into the system.

Make sure the cure does not prolong the outage

“Keep trying until it works” sounds resilient.

Sometimes it is the reason nothing recovers.

Google’s discussion of cascading failures explains how retries can add load to an already overloaded service. It recommends bounded retries, randomized exponential backoff and attention to retry amplification across layers. [3]

Those are distributed-service techniques, not a recipe to copy blindly into every control system. The broader review question is useful:

Can the recovery action recreate or worsen the condition it is trying to remove?

For an embedded system, examine repeated resets, simultaneous device initialization and the handling of commands during reconnection.

Specify when another attempt is permitted, when attempts stop and what happens next. A system that continually retries without making progress needs a defined response—not merely a persistent animation.

Add one recovery test to the next build

Here is a proposed starting checklist, to adapt to the application:

1. Establish the initial condition. Record configuration, operating mode, load and relevant retained state.
2. Introduce a controlled fault. Use an approved test environment, simulation or fault-injection arrangement with appropriate protections.
3. Verify the immediate response. Check outputs, indications, containment and preserved evidence.
4. Remove the initiating fault. Do not quietly reset unrelated components to help the test pass.
5. Follow the intended recovery path. Include only the operator actions the delivered procedure permits.
6. Demonstrate useful operation. Check data validity, command handling and the defined recovery endpoint.
7. Challenge the boundary. Repeat with an interrupted recovery, an unavailable dependency or a fault that returns.

Record every extra action needed to succeed. An undocumented power cycle or engineering-console command is evidence of a gap between the intended recovery process and the demonstrated one.

For hazardous equipment, recovery testing and restart behavior require application-specific safety review. Clearing an alarm must not be treated as permission to bypass an interlock.

Where Ngenaire helps

Ngenaire’s published platform features include UML and SysML diagrams, test plans and procedures, verification traceability, and execution history by build label. [4]

Those capabilities can support a recovery review: describe the intended behavior, connect it to verification activities and retain the results for the tested build.

They do not establish that recovery is safe or effective. That still requires engineering analysis and representative evidence.

The value is keeping “how it should recover” close to “what we actually demonstrated.”

Ask what happens after the alarm clears

At the next design review, choose one credible fault.

Follow it beyond detection. Beyond shutdown. Beyond the moment the missing signal returns.

Then ask:

What proves that the system is ready to be trusted again?

Which recovery step on your projects still depends on someone saying, “Try restarting it one more time”?

References

[1] A. Walcer, K. Guliani, M. Ward, S. Hsiao, and V. Stacey, “Lessons Learned from Twenty Years of Site Reliability Engineering,” Google SRE. Accessed: Sep. 16, 2026.

[2] NASA, “On-Orbit Fault Recovery,” Small Satellite Reliability Initiative Knowledge Base. Accessed: Sep. 16, 2026.

[3] M. Ulrich, “Addressing Cascading Failures,” in Site Reliability Engineering, ch. 22, Google, 2016. Accessed: Sep. 16, 2026.

[4] Ngenaire, “Bring us your hardest engineering problem,” platform features. Accessed: Sep. 16, 2026.