OTA Firmware Updates That Cannot Brick the Fleet

A failed server deployment is an inconvenience because someone can roll it back. A failed firmware update on a device installed behind a wall, on a pole, or inside a machine is something else entirely: if the device does not come back up, recovering it costs a site visit, and across a fleet those visits can exceed the value of whatever the update was delivering. Designing for that asymmetry is what separates a firmware update mechanism from a firmware download mechanism.

The foundation is that a device must never overwrite the only copy of software it has. The standard approach is two firmware slots, A and B, where the device runs from one while writing the update into the other, then switches on the next boot. If the new image fails, the previous one is still intact and the bootloader can fall back to it. This doubles the flash budget for application code, which on constrained hardware is a genuine cost, but the alternative is a device whose update path has a window during which any interruption is permanent.

The switch itself must be decided by something the new firmware cannot lie about. The usual mechanism is a watchdog and a confirmation flag: the bootloader marks the new image as provisional, boots it, and unless the new firmware explicitly confirms itself healthy within a defined period, the next reset returns to the previous slot. What counts as healthy deserves thought. Reaching the main loop proves very little. Successfully contacting the management server proves considerably more, because it demonstrates that the part of the system needed to deliver a fix still works. A device that boots happily but has lost its network stack is bricked in every sense that matters.

Images must be signed, and signature verification must happen before anything is written to the boot path. An update channel is a remote code execution path into every device in the fleet by design, so it is the single most valuable thing an attacker can compromise. Verification belongs in the bootloader, where the running application cannot be persuaded to skip it.

Transfer needs to assume the network is bad, because in the field it usually is. Downloads should be resumable in chunks rather than restarting from zero, each chunk should be integrity-checked on arrival rather than only at the end, and the whole image should be verified before the device commits to it. On metered cellular links, delta updates that ship only the changed blocks can reduce transfer volume enough to change the economics of updating at all.

Staging is the control that catches what testing did not. Releasing to a small percentage of the fleet first, waiting long enough to observe real behaviour, and only then widening the rollout turns a fleet-wide incident into a contained one. The waiting period has to be long enough to cover a full duty cycle: a bug that only appears after a device has been running for six hours will not show up in a thirty-minute canary. Automatic halting on a rise in failed check-ins is worth more than any dashboard, because it acts at the speed the rollout does.

Power loss deserves explicit design rather than optimism. Devices lose power mid-write, and the question is only whether the firmware layout survives it. Writes should be ordered so that the metadata declaring an image valid is written last, after the image itself is fully present and verified. Then an interruption at any point leaves a partial image that is simply never marked bootable, and the device continues running what it was running before. A device that survives being unplugged at the worst possible moment is one that has had this ordering thought through, and it is the cheapest insurance in the entire update path.