On September 7 we found that a bucket of finished video files had gone ten days with no backup while the nightly report said everything was fine. The same day, we built an off-site copy of everything the platform holds, 11,357 files and 22 gigabytes, mirrored in 197 minutes, and then pulled six of them back from the copy and checked every one against its original. All six matched. One of them was a 1.25 gigabyte video, restored in full.
This is the piece about backups, and about the difference between having them and being able to prove it.
How a green light lies
The nightly backup copied our storage from a list of buckets. Eight of them. When we added a ninth bucket in late August for finished video files, nobody added it to the list. That is an ordinary mistake. What made it dangerous was that the check that watched the backup worked from the same list. The job skipped the new bucket; the watchdog agreed nothing was missing; the nightly checks board, the subject of the previous piece, showed a clean line for ten nights.
The comment at the top of the job even warned that a new bucket only needed to be added to the list. The entry it asked for was exactly the one nobody made.
A check that shares the assumptions of the thing it checks cannot disagree with it. That is the whole finding, and it recurs wherever a check is built from the same list as the job it watches.
What we changed
No list. The backup now asks the system what exists and copies all of it, so a bucket created tomorrow is copied tomorrow night. The watchdog was rebuilt to ask a different question from the job: not "did the copies succeed" but "did the job report at all," so it cannot inherit the job's blind spot.
And the copy now leaves the building. Until September, our backups were copies inside the same account as the originals. Data isolation, not disaster isolation. A whole-account loss would have taken both. The off-site mirror runs every night to a second provider on the other side of the internet, at a cost that rounds to a couple of dollars a month, with no charge to copy data back out, so a restore costs nothing.
The drill
A backup nobody has restored from is a hope. So the first thing we did after the first mirror was restore from it: six files across five buckets, pulled back from the off-site copy and compared byte for byte against the originals. Six of six identical, including the largest video file we had. That drill is now the standard. A backup system is not done when it copies. It is done when it has given something back and the hash matched.
What I decided not to buy
The database provider offers point-in-time recovery, the ability to roll the database back to any moment in the last week. I looked at the price and declined it. That is a real trade-off and I want it on the record rather than hidden: we take daily database backups and nightly storage mirrors, and we accept that a catastrophic database error could cost us up to a day. For a newsroom of our size that is the right call today. It may not be next year, and the board will tell us when the numbers change.
Owning the stack means owning that decision, and being the one who explains it if it goes wrong.
This is No. 11 in We Built Our Own. Previous: The site checks itself every night. Next: The database says no.