ReliabilityAdmin tooling
Engineers out of the loop for routine booking failures
Hotel booking platform, 1M+ users a month
Otello (2GIS)
2025 to 2026 · On-call rotation and my own tickets
~10
manual fixes a day removed
The problem
A booking could fail with a fatal error at any provider step: a timeout on the availability check, a receipt that did not confirm, a flaky payment step.
Support escalated to an engineer, who looked at the case and fixed it by editing the database. No audit, no role checks. And a high share of these errors were transient: a second attempt would have succeeded.
What I did
- Added automatic retries for the fatal errors that turned out to be transient, covering the bulk of escalations.
- Built a retry endpoint in the admin panel that checks whether the booking is in a retryable state and logs who retried what.
- Replaced direct database edits with status changes in the admin panel: a role model, a confirmation dialog and a full audit trail.
- Replaced one heavy all-purpose page for support with focused exports, after asking support what they actually look at.
Decisions
- Retry automatically only where the failure is provably transient. Everything else goes through a human, with an audit trail.
- Treat the admin panel as a product: if an engineer does something by hand twice, it becomes a safe action there.
What changed
- About 10 manual fixes a day are gone. Most fatal errors resolve automatically, and support handles the rest without engineers.
- Every manual change is attributable: who, when, from what state to what state.
- Lower database load from the support pages.
Stack
GoPostgreSQLAdmin panel