No. 002 · July 29, 2025
First, Make It Measurable
I arrived at a startup where things ran on “more or less.” The backend had no error rate, no latency, none of it — it wasn’t observable. I don’t know why it hadn’t happened before, and by the time I got there it didn’t matter: it wasn’t there, so it went first. I’ve written elsewhere about the strangler-fig migration that came later, and about the four things — instrumentation, templates, infrastructure as code, a fast pipeline — that had to exist before the first service could be extracted. What I didn’t say there is how the order of everything before that got decided, or what it cost to hold that order while people wanted to start elsewhere. That’s this post.
The genuinely hard part wasn’t spotting what was missing. Walk into a system like that and the list of things that need fixing writes itself in an afternoon — I could have named twenty. The hard part was deciding what not to do yet, and doing that repeatedly, out loud, to people who also had a list and wanted to start on theirs.
The order that held, in the end:
- Make the system measurable.
- Make shipping cheap and repeatable.
- Change the architecture.
Not because the architecture mattered less, but because the first two made architectural change far less risky.
Why anyone listened
None of what follows was imposed. I proposed it, and it landed because the managers and the team already trusted my judgment — from the system design interview, from the ideas I kept bringing in day to day, and from seeing how I approached problems. Trust like that is not something you can plan for on a project timeline, but it’s worth naming, because it’s the reason “not yet” worked as an answer instead of reading as stalling.
Where the urgency came from
Some of this was professional judgment. Some of it was closer to fear, and it’s worth being honest about the difference.
I came from a product with real throughput — the kind where you don’t choose whether to watch rates and percentiles, you watch them because the alternative is flying blind at a scale that punishes it fast. The first deploy here made that literal: I pushed a change with no way to tell how many requests per minute the system was even taking, let alone whether anything had broken. I’d never shipped without that number before. Arriving here and finding nothing measured felt, for a while, like exactly that: working blind, on a system I hadn’t yet spent enough time with to know its failure modes by feel, if that was ever really possible without the numbers. The same instinct went off around queues with no backpressure, though what became of that belongs in the structural section below, not here.
But fear isn’t a plan, and I’d been through enough systems to know the other half of it too: you can’t foresee everything, and trying to is its own kind of waste — especially at a startup that hasn’t proven it survives to have the scale problem yet. So it was never a clean choice between building the thing that holds up under load we had no evidence we’d ever see, and building the thing that gets us through this month. It was both, argued out again for every decision: enough headroom that we wouldn’t be the reason something broke, not so much that we spent the runway on a future that might not show up. That tension didn’t resolve once. It’s underneath everything that follows.
Observability first
Sentry went in first, and the reason was almost entirely practical: it was the fastest thing to try with the budget that already existed. No procurement conversation, no infrastructure to stand up, no migration for anyone else’s code — just start seeing errors that had previously gone nowhere. I already knew Sentry wouldn’t cover profiling or the fuller tracing story once things got more serious. That wasn’t a lesson I learned afterward — it was priced in before we adopted it. I picked it anyway, because “the tool that will eventually fall short” and “the tool that gets something measured this week” weren’t competing for the same slot; we needed the second one first, and could afford to outgrow it later.
Once error tracking existed and people got used to looking at it, and things did get more serious, Datadog followed, for the broader metrics and tracing story Sentry was never going to tell. And once the Datadog bill caught up with the growth it had helped make visible, that got replaced with OpenTelemetry, Prometheus, and Grafana — same category of insight, a cost profile that scaled the way the team’s budget actually needed it to.
Knowing where Sentry’s ceiling was going in is what made it fine to hit it — that sequence wasn’t improvised as it went. The part worth stating plainly is that observability was the priority ahead of other problems that everyone could already see. Nothing else on the list got fixed faster because it was measured first; it got fixed correctly, because by the time we touched it, we could tell whether the fix worked.
Then, how fast we could ship
Next was Jenkins to GitHub Actions. The Jenkins setup was a service running on its own in ECS — a continuous cost for something that worked poorly, was awkward to maintain, and needed its own seats and accounts for anyone on the team who touched it. Moving CI into GitHub, where the code already lived, meant one fewer tool to run, one fewer thing to grant access to, and a straight line from a merged PR to a deploy. The point wasn’t the tool migration itself; it was that shipping got simpler and faster, and a team that can ship quickly can also correct quickly — which matters more than getting everything right on the first attempt.
The part where people wanted to start already
By this point, the metrics from step one told me which cracks were real, not just which ones scared me the most. That’s what let me make the case for event-driven over patching the existing synchronous calls — still my call, still a judgment, but one backed by evidence instead of a hunch. And once that backbone existed, it stopped being a single decision and started being the thing everything else plugged into.
Everything past observability and CI was structural: queues with backpressure, retries and circuit breakers, replacing Ribbon and Eureka with a proper application load balancer, a new gateway, and an event-driven architecture that didn’t exist when I arrived — services talked to each other directly, synchronously. All of it mattered. None of it went first.
Holding that order — observability, then shipping, then architecture — meant repeatedly telling people — including myself, some days — that something real and needed was going to wait. That’s a harder conversation than it sounds like from the outside, because nobody on the other end of it is wrong. The queue that needs backpressure really will fall over eventually. The service discovery setup really is fragile. The tension isn’t technical; it’s that everyone is holding a legitimate “why not now” and only one thing can go first.
I don’t have a clean anecdote for this one the way I do for the others above — what I have is planning, sprint after sprint: the same fight for time on work whose value wasn’t visible yet, or that people had simply stopped thinking about since the last time it came up. Propose the order, explain what it buys you to have observability and a fast pipeline before you touch anything structural, and accept that the explanation has to be given again at the next planning session, because being right the last time doesn’t bank patience for the next one.
The event-driven piece went in gradually, service by service, not as a rewrite. The notification service is the clean example: it went from being called synchronously to sitting on queues attached to whatever topics were relevant to it — a user gets created, that lands on the users topic, and the notification service’s queue, already listening there, just picks it up. Nobody has to know the notification service exists to trigger it. Part of the case for doing this then, rather than later, was that larger clients were starting to show interest, and backpressure stops being optional once the traffic behind it is real. The same shift is also what eventually let data flow into the datalake and, from there, into the datawarehouse — more on that below — instead of every downstream consumer querying the operational databases directly.
The structural work is also where the pieces from the strangler-fig post — the gateway that made an endpoint’s migration a routing change, the event backbone the extractions grew — actually came from. That post picks up once those existed; this is where they got built, and why they weren’t built first.
Still change the architecture: what the datawarehouse was for
The same decision showed up outside the backend. The event backbone existed because of the extractions — every service publishing to topics instead of calling each other directly. That backbone is what made the datawarehouse cheap to add later: the pipeline that fed it didn’t need new integration work, it just subscribed to the same topics the notification service already listened to, batching instead of reacting in real time. The architecture didn’t ask anything of this upfront; it paid it back here.
Clients were asking for daily, weekly, and monthly reports: portfolio analysis, full historical views, the kind of query that’s heavy precisely because it looks backward over everything. The instinct, every time, is to point that query at the operational database, because that’s where the data already is and it’s already correct.
It wasn’t one fight, it was a recurring one: every so often a new reporting need came up and they wanted it fast — a backoffice with the freshest numbers available. Explaining that the data didn’t propagate near real time, that there was a whole mechanism between something happening and it showing up in a query, never really landed; they weren’t technical people, and had no reason to think about propagation at all. Explanation after explanation, we still kept the boundary in place.
I pushed back on that by default, for two reasons. First, cost and safety — a heavy analytical query against the same database serving live traffic is a straightforward way to degrade the thing you actually need to stay up. Second, and more durable: doing it right was nearly free, since the topics already existed for other reasons. The pipeline wrote in batch into a datalake, then an ELT step loaded it into the datawarehouse — raw batches first, transformed after, rather than shaping the data on the way in. The datawarehouse became the source of truth for analytics and reporting. We’d stood up Metabase on top of it so sales and business could pull their own numbers without filing a ticket every time they needed one. Every report that leaned on the warehouse instead of the live database was one more case where someone got their answer without engineering in the loop.
So the questions I asked back, every time a “we need this data” request came in, were the same three: who actually needs it, how fresh does it really have to be, and how do they want to get it — a pushed report, or something they can go pull themselves. Almost nobody asking for data has thought about the second question; they default to wanting it as fresh as possible because fresh sounds free. It isn’t. The person asking is not the one who has to answer for a degraded database at 2pm, so the asking-back has to happen on the other side.
What the order actually argues
None of this was a grand plan laid out in week one. It reads that way in retrospect because the order held: observability before velocity, velocity before architecture, and inside each of those, more questions asked back than requests taken at face value. Every step meant telling someone — sales, engineering, occasionally myself — that something they needed would have to wait for something else first.
The system I arrived to wasn’t badly built. It was built for a size the company had already outgrown, which is a different thing, and worth saying plainly rather than turning into a story about everyone before me having done it wrong. What changed was scale, and what scale asks for first is not more architecture. It’s a way to tell whether anything you do next is actually working.