Sally Roth

SRE Lab

A reliability-first lab: a few working SRE tools and step-through explainers, written for running these systems across many teams. The tools cover error budgets and burn-rate alerts, telling real outages from blips, searching real postmortems by failure mode, and judging an agent's incident work by its evidence. The explainers cover incident management, observability, Kubernetes, Linux, infrastructure as code, delivery and security. Engineering with AI agents has its own section further down.

Tools

SLO Burn SimulatorSee how fast an error budget drains, which burn-rate alerts fire and when, matching Prometheus rules, and how to run SLOs across thousands of services.updated Down or MeUptime checks with an AI classifier that decides whether a failure is real, and alerts only when both of its confidence thresholds are met.updated The Incident Files318 real postmortems, searchable by failure mode, plus the nine failure patterns that keep coming back and the controls that retire them.updated Did the Agent Do It Right?Three simulated agent runs on the same incident: one checks the evidence, one confidently blames the wrong service, one gets stuck retrying. Pydantic Evals judges each by its tool-call trace, not its explanation.updated

Reliability explainers

Incident ManagementIncident management as a system for hundreds of teams: severity as a company-wide contract, routing as data, many incidents at once, legal clocks, fleet on-call health, learning across incidents and metrics that mean something.updated KubernetesHow Kubernetes works in 18 step-through diagrams, across hundreds of teams and clusters: admission policy, capacity governance, the fleet upgrade program, platform health signals, and DNS, storage and network limits.updated LinuxWhat Linux does with memory and packets across thousands of hosts: base-image settings, PSI, fleet OOM policy, conntrack and TCP limits, network failure signals and finding the few bad hosts.updated ObservabilityRunning telemetry for hundreds of teams: budgets per team, the Collector pipeline as a platform, Prometheus topology, shared PromQL and SLO rules, trace propagation, sampling policy, log retention tiers, and paging as an org control loop.updated OwnershipResearch on how very large companies decide who owns code and services: OWNERS files and CODEOWNERS, service registries synced from HR, routing pages, incidents, vulnerabilities and costs to owners, keeping it accurate through reorgs, and fleet-wide changes that route around owners.updated Taming CardinalityHow one label multiplies series and cost, where Prometheus pays for them, how to find the culprit, and how to drop, aggregate and govern before the bill arrives.updated SecuritySecurity run as a system across hundreds of teams and accounts: org guardrails and policy as code, federated identity, secrets and supply chain as programs, vulnerability flow, fleet-wide detection, agents, and the numbers that show progress.updated DeliveryFrom commit to production, for one team and for hundreds: CI, build once and promote, drift control across clusters, rollout policy as a platform default, flag governance, schema-change guardrails, supply-chain signing, and merge queues.updated Infrastructure as CodeRunning IaC across hundreds of teams: state layout as blast-radius policy, shared modules, environments, one change-control gate with policy as code, drift routed to owners, safety rails, and testing.updated Terraform, Five Years OnWhat changed since 1.0, across a fleet: version governance, moved, import and removed blocks, checks and tests, Terraform or OpenTofu, ephemeral values, and Stacks.updated

AI engineering

How coding agents change the work: running them safely across an org, building for the next model, and planning when capabilities move every few weeks.

Agentic EngineeringHow Stripe, Spotify, Shopify and others run hundreds of coding agents and ship safely: harnesses the model can't skip, verifiers it can't edit, no route to prod, risk-tiered review, and Lauren Tan's pstack.updated One Agent to HundredsA five-stage ladder from one coding agent you trust to hundreds in parallel, built on Lauren Tan's pstack, then an org layer on top: one platform for every team, budgets and quotas per team, pooled limits, audit, and merge rate next to change failure rate.updated Engineering for the Next AgentBuild for the model that is coming, not the one you have: assets that compound across model generations, scaffolding to delete, capability trends with their caveats, evals, safe model upgrades, and boundaries and interfaces that outlive any one model.updated Graphs and LoopsRunning agent loops and graphs across hundreds of teams: loop sprawl and what it costs, loops with a check and a stop, graphs of steps, branches and joins, workflows reviewed as code, and what every unattended loop must declare before it runs.updated

Changelog

What each daily research pass changed, newest first. Every change is checked against a primary source.

  • Observability: the "What breaks at scale" box is gone from all eight chapters, and four more are rewritten for the fleet: Prometheus topology (shards, HA pairs, remote write, target coverage), trace propagation as a fleet standard with a completeness number, an org sampling policy with cost and fidelity per team, and log schema, label and retention-tier standards with a bill per team. Every chapter is shorter. The Incident Files: each pattern's wards are now org-wide controls with a "retired when" fleet metric.
  • One Agent to Hundreds keeps its five-stage ladder for one engineer and adds an org layer on top: one platform for every team, managed settings and an identity per agent, then team budgets and quota splits, pooled API and GitHub limits, an audit trail, and merge rate next to change failure rate.
  • Did the Agent Do It Right? adds how an org gates incident agents (a nightly suite on real incidents, autonomy levels, tool budgets outside the model), and Down or Me's AI box becomes alert triage as a fleet policy.
  • Explainer titles lose the “Drawn” suffix, Ownership drops “at Scale”, and One Pattern, Three Languages is retired.
  • Engineering on an Exponential and Taming Cardinality: the separate scale boxes are folded into the chapters, and every chapter is shorter. New in the steps: fast and slow lanes when the model builds most of the system, fleet load from bigger agent tasks, sharding before one Prometheus runs out, and label standards, ingestion guardrails and series quotas per team. Cardinality chapter 4 is now Org-wide ingestion guardrails.
  • Incident Management at Scale and Security at Scale are now Incident Management and Security. Those two and Delivery drop the "What breaks at scale" box. Its points moved into each chapter's steps and summary, and all 27 chapters got shorter.
  • Agentic Engineering, Engineering for the Next Agent and Graphs and Loops: the "What breaks at scale" boxes are folded into each chapter's steps and 30-second answer, so scale is the main story, and all 22 chapters are shorter. Chapter 5 of Agentic Engineering is now Reviewing agent output.
  • Infrastructure as Code and Terraform, Five Years On: scale is now the main thread of every chapter, and the separate "What breaks at scale" box is gone. Four chapters are reframed: state layout as blast-radius policy, change control across hundreds of teams with a change fail rate per root module, provider and version governance, and standardizing on Terraform or OpenTofu. Shorter than before.
  • Security for SREs is now Security at Scale: nine new chapters on running security across the whole org, from org-level guardrails and policy as code to federated identity, secret and key programs, supply chain, vulnerability flow with CISA BOD 26-04, fleet detection, agents and program metrics.
  • Graphs and Loops, Engineering on an Exponential and the SLO Burn Simulator move up to org scale: loop sprawl, workflows as reviewed artifacts, policy for unattended loops, the operating-model shift, what compounds in an engineering org, and running SLOs across thousands of services.
  • Sane Incident Management is now Incident Management at Scale: ten chapters on running incidents as a system across many teams, from severity and routing to legal clocks, fleet on-call health and program metrics. Shorter than before.
  • Delivery and Observability: eight chapters rewritten for running them across hundreds of teams. Rollout policy as a platform default, flag governance, schema-change safety, drift and break-glass across clusters, telemetry budgets per team, shared query and SLO rule libraries, the telemetry pipeline with its own SLOs and pager, and paging as an org control loop. Each ends with the org-wide control and the metric that proves it works.
  • Kubernetes and Linux: every chapter now leads with running it across many clusters or thousands of hosts, and the "What breaks at scale" box is gone. Five chapters are new: the fleet upgrade program, capacity governance, platform health signals, finding the few bad hosts, and network failure signals across the fleet. Also org-wide admission policy, kernel and sysctl baselines in the base image, kernels rolled out in waves, and systemd-oomd as fleet OOM policy. Every chapter is shorter than before.
  • Engineering on an Exponential: chapter 1 rewritten. Plan for the curve over months, not the next release, and when the model builds most of the system, separate its three roles (author, runtime, operator) with four tests: swap, regenerate, enforce, reverse.
  • The Incident Files now opens with the recurring patterns, with search below. Plan B is retired.
  • One Agent to Hundreds: treat a handoff as a claim to recheck, run a sentinel agent with a kill switch, let gates that need a person fail closed, and remember that more parallel agents is not the goal. New lines on Meta's thin orchestrator, Stripe's agent-run ML experiments and Steve Yegge's 20+ agents.
  • One Agent to Hundreds: worktree cleanup that could lose commits and turn caps a plugin could outlive (both fixed in Claude Code 2.1.304), and GitHub refusing anonymous clones under unusual traffic.
  • Agentic Engineering: sandbox defaults that loosen on upgrade, step-up scopes for MCP, installs that don't verify packages, credentials in shared agent memory, and model-only vulnerability reports that bounced.
  • Did the Agent Do It Right?: read transcripts, not just scores; score the artifact, not the steps; measure recall; and benchmarks are losing their human baseline.
  • Delivery: ubuntu-latest moves to Ubuntu 26.04 by 19 November, and a "read-only" Argo CD diff that could run commands.
  • Security: attackers run agents too, and no single log shows the whole attack.
  • Plan B: your AI guardrail is a dependency too, and a shared gateway must bind MCP sessions to callers.
  • Smaller: Python 3.15's sampling profiler (Patterns), ITBench-AA and Anthropic's harness follow-up (Agentic), a Codex daemon update, Devin's reviewed skill edits, Harrison Chase on re-tuning harnesses and pstack 0.15.29 (fleet), two moved links (Incident Management).
  • Did the Agent Do It Right?: three new limits. A guard model can be talked out of its verdict by fake policy text, a grader that reads only the final reply fails good runs, and judges drift unless people keep checking them.
  • One Agent to Hundreds: give each agent its own identity and contain a misbehaving one in steps, keep formal checks in sync with the code they cover, and define done as verified, not compiled.
  • Plan B: a retired DeepSeek model name now quietly gets a newer model, and why unnamed stealth models don't belong in a failover path.
  • Kubernetes: the new PVC Unused condition is a cleanup review queue, not permission to delete an agent's workspace.
  • Smaller: Devin Oncall on deploys as leads, not causes (Incident Management), Agent Sandbox warm pools and scheduling gates (Kubernetes), the OpenTelemetry tool call id and a Claude Code telemetry fix (Agentic), a moved PagerDuty link, an archived Ona link.
  • Agentic Engineering: the Vercel microVM escape now has a CVE and fixed kernels, and a new item on an Agent SDK flaw where a session ID became a command-line flag.
  • One Agent to Hundreds: checks you have seen fail (pstack's new formal verification skill), Addy Osmani on layered verification instead of line-by-line review, and pstack's updated catch-up rules.
  • Did the Agent Do It Right?: graders can be reward-hacked too, and why each eval task gets its own microVM.
  • Smaller: DeepSeek's search and fetch billing for failover (Plan B), a Docker Hub token gotcha (Security), Microsoft on prompts as no enforcement boundary (Agentic), Boris Cherny on Claude Code Review (fleet).

Full changelog →