W25 · A Bad Rollback.
The rollback took forty minutes. Thirty-five of them went to working out which version to roll back to.
Ask AI
Forty frantic minutes.
Five minutes after release, the error rate climbed. Deciding to roll back took one minute. Performing the rollback took forty.
Where did the time go? Scrolling commits to find which build had actually shipped and held. Checking the artifact still existed. Working out whether the database migration had to come back too.
The tooling was not the problem.
We had deploy scripts. We had version tags. What we lacked was one thing: a record that could immediately answer "where should we return to right now".
We added it afterwards. Every successful release writes its version, timestamp, and commit hash to a file. A rollback reads that first.
Next week.
Rehearse a rollback on staging, with a stopwatch. Target: under five minutes, with nobody scrolling chat history.
— Discussion
GitHubComments.
Comments are powered by GitHub Discussions. Keep the conversation focused and respectful.
Comments did not load this time. Refresh to retry, or join the discussion directly on GitHub.