A Quarterly on Infrastructure
On the quiet labor
of making systems
legible.
Feature · Essay
On infrastructure.
The argument for an observability tool that treats its operators as adults, told in three movements.
The conventional observability platform was designed for a manager who would read its dashboards over a Tuesday-morning coffee. The charts are smooth, the colors are calm, the alerts are summarized. Everything is averaged into the shape of a working week. Observatory was not built for that reader. Observatory was built for the engineer at three in the morning who has been promised, by the dashboard the manager bought, that nothing is on fire.
We have spent a decade as the first audience of these dashboards. We have watched as the systems we operate became more complex, the traffic more bursty, the cardinality more important, and the dashboards designed for the manager became less and less equipped to tell us what was happening. The vendors did not respond by building a better dashboard. They responded by charging more for the cardinality the dashboard refused to show.
Observatory is the response we have been writing for ourselves for a long time. It is open source under Apache 2.0. It is one binary. It treats every metric as taggable, every span as queryable, every percentile as sacred. It does not flatten the long tail into a soothing chart for a Tuesday morning. It shows you what your system is actually doing, including the parts you would rather not look at, especially those.
Field reports
From the field.
Three short dispatches from teams running production systems they did not have time to build twice.
I · Berlin
On the dignity of a quiet on-call shift.
It is 03:14 in a converted warehouse near Friedrichshain. The maintainer is awake because the system is awake, and she has come to think of this hour as the one in which the system tells her the truth. The dashboards she pays for show her averages. The dashboards she built show her the long tail. The difference is the work.
II · Toronto
A platform team rediscovers what it owns.
For three years, the metrics belonged to the vendor. The team owned the alerts but not the data. When the contract came up for renewal, the question was not whether to renew but what they had given away. They built Observatory in a room with two windows and a whiteboard, and the whiteboard is still there.
III · Mexico City
On reading a stack the way one reads a city.
Infrastructure has neighborhoods. Some are sleepy at three in the afternoon and busy at midnight. Some have been gentrified by a deploy that nobody remembers approving. The work of an observability tool, properly understood, is to give the engineer a map that tells the truth about the city, not the brochure version that tells the truth about the developer.
On the cover
An object lesson.
The objects on this issue's cover were photographed in the editor's office in Berlin during the second week of October, in the kind of low northern light that the season permits for roughly forty minutes at a time. The book is a gift from a former colleague. The pen has been resharpened twice. The vessel was thrown by a friend who left engineering for ceramics and has been happier since. None of these objects are products. All of them are kept on a desk, every day, by a person who runs production systems and reads things slowly.