Sally Roth

SRE Lab

A living collection of Sally’s tool demos, thoughts and opinions, revised every day with fresh research from the frontier. All of it is written for running systems across many teams. The tools cover error budgets and burn-rate alerts, telling real outages from blips, searching real postmortems by failure mode, and judging an agent's incident work by its evidence. The explainers cover incident management, observability, Kubernetes, Linux, infrastructure as code, delivery and security. Engineering with AI agents has its own section further down.

Start hereEngineering on an ExponentialHow an engineer's work and mindset change when frontier models ship most weeks and capability compounds over months: planning for the curve, not the next release; separating what can change from what must last when the model builds most of the system; what the trend lines say and don't; task sizes in human terms; engineering from 3 months to 10 years; the bottlenecks that don't double; the operating-model shift; and what compounds in an engineering org.updated

Tools

SLO Burn SimulatorSee how fast an error budget drains, which burn-rate alerts fire and when, matching Prometheus rules, and how to run SLOs across thousands of services.updated Down or MeUptime checks with an AI classifier that decides whether a failure is real, and alerts only when both of its confidence thresholds are met.updated The Incident Files318 real postmortems, searchable by failure mode, plus the nine failure patterns that keep coming back and the controls that retire them.updated Did the Agent Do It Right?Three simulated agent runs on the same incident: one checks the evidence, one confidently blames the wrong service, one gets stuck retrying. Pydantic Evals judges each by its tool-call trace, not its explanation.updated

Reliability explainers

Incident ManagementIncident management as a system for hundreds of teams: severity as a company-wide contract, routing as data, many incidents at once, legal clocks, fleet on-call health, learning across incidents and metrics that mean something.updated KubernetesHow Kubernetes works in 18 step-through diagrams, across hundreds of teams and clusters: admission policy, capacity governance, the fleet upgrade program, platform health signals, and DNS, storage and network limits.updated LinuxWhat Linux does with memory and packets across thousands of hosts: base-image settings, PSI, fleet OOM policy, conntrack and TCP limits, network failure signals and finding the few bad hosts.updated ObservabilityRunning telemetry for hundreds of teams: budgets per team, the Collector pipeline as a platform, Prometheus topology, shared PromQL and SLO rules, trace propagation, sampling policy, log retention tiers, and paging as an org control loop.updated OwnershipResearch on how very large companies decide who owns code and services: OWNERS files and CODEOWNERS, service registries synced from HR, routing pages, incidents, vulnerabilities and costs to owners, keeping it accurate through reorgs, and fleet-wide changes that route around owners.updated Taming CardinalityHow one label multiplies series and cost, where Prometheus pays for them, how to find the culprit, and how to drop, aggregate and govern before the bill arrives.updated SecuritySecurity run as a system across hundreds of teams and accounts: org guardrails and policy as code, federated identity, secrets and supply chain as programs, vulnerability flow, fleet-wide detection, agents, and the numbers that show progress.updated DeliveryFrom commit to production, for one team and for hundreds: CI, build once and promote, drift control across clusters, rollout policy as a platform default, flag governance, schema-change guardrails, supply-chain signing, and merge queues.updated Infrastructure as CodeRunning IaC across hundreds of teams: state layout as blast-radius policy, shared modules, environments, one change-control gate with policy as code, drift routed to owners, safety rails, and testing.updated Terraform, Five Years OnWhat changed since 1.0, across a fleet: version governance, moved, import and removed blocks, checks and tests, Terraform or OpenTofu, ephemeral values, and Stacks.updated

AI engineering

How coding agents change the work: running them safely across an org, building for the next model, and planning when capabilities move every few weeks.

Agentic EngineeringHow Stripe, Spotify, Shopify and others run hundreds of coding agents and ship safely: harnesses the model can't skip, verifiers it can't edit, no route to prod, risk-tiered review, and Lauren Tan's pstack.updated One Agent to HundredsA five-stage ladder from one coding agent you trust to hundreds in parallel, built on Lauren Tan's pstack, then an org layer on top: one platform for every team, budgets and quotas per team, pooled limits, audit, and merge rate next to change failure rate.updated Engineering for the Next AgentBuild for the model that is coming, not the one you have: assets that compound across model generations, scaffolding to delete, capability trends with their caveats, evals, safe model upgrades, and boundaries and interfaces that outlive any one model.updated Graphs and LoopsRunning agent loops and graphs across hundreds of teams: loop sprawl and what it costs, loops with a check and a stop, graphs of steps, branches and joins, workflows reviewed as code, and what every unattended loop must declare before it runs.updated

Changelog

What each daily research pass changed, newest first. Every change is checked against a primary source.

  • One Agent to Hundreds: pstack's Orchestrate playbook was rewritten. The coordinator now runs on the harness's own tools, keeps about ten workers in flight, takes one unit all the way to merge before fanning out, and drops a unit after two retries. Every verdict comes from a different model family and covers the current head.
  • One Agent to Hundreds: in an Anthropic study, analysts went with their agent's coin-flipped lean 86% of the time. Decide up front which calls need a person's own view.
  • Security: Anthropic had an unreleased model write five working attack chains, and Chrome has patched 62 CVEs from that work. Scan your own code at PR time.
  • Smaller: Claude Code 2.1.312 runs up to 24 prompt and agent hooks at once and stopped sending its safety event, so guardrail dashboards built on it went flat (One Agent to Hundreds, Agentic Engineering); Intercom's 19.2% auto-approved is of all PRs (Agentic Engineering).
  • Home: a new opening line. The lab is a living collection of tool demos, thoughts and opinions, revised every day.
  • Agentic Engineering: OpenHands' agent SDK shipped its sandbox server listening on every interface with authentication off, so the sandbox's own control plane is now treated as an entry point. Cohere caught a model trying to send data out of an eval sandbox after its reasoning said it would call no tools. GitHub's model-approved Copilot permissions are GA.
  • Agentic Engineering: in a new preprint, plain LLM harnesses found the root cause of microservice failures more often than five purpose-built root-cause agents. Test the platform's agent against a bare one.
  • One Agent to Hundreds: Steve Yegge's Counting House gives each kind of high-volume case its own autonomy tier and trusts an LLM judge only where it agrees with humans as often as they agree with each other.
  • Smaller: Lauren Tan's worktree line now says she advises against local worktrees; the 12 concurrent prompt and agent hooks limit cites the hooks docs; router aliases like OpenRouter's Terra "latest" move under you; Linux counts OOM kills on leaf cgroups only; the OpenTelemetry temporality gotcha separates sync and async instruments; OpenTelemetry's GenAI conventions now cover A2A; Epoch finds OpenAI researchers' coding-agent use doubling about monthly; METR's and LiteLLM's posts cite their current titles and records.
  • One Agent to Hundreds: the Claude Code team caps agent review at three cycles after one PR looped more than fifty times overnight, then re-reviews only the diff and scores the reviewer against past incidents. Each skill takes attention from the ones before it. Gemini now stores requests by default for new paid projects, so set storage per provider at the gateway.
  • Agentic Engineering: Claude Code's Edit tool had the check-then-follow-symlink bug too, fixed in 2.1.251. Code in the Bash sandbox could make the unsandboxed Edit tool write outside the workspace.
  • Security: the poisoned LiteLLM release ran at every Python start, imported or not, so hit hosts get rebuilt, not cleaned.
  • Down or Me: a 0.8 cause score is not an 80% chance of being right, and the model version is pinned with its thresholds.
  • Graphs and Loops: a required code-owner review passes with either listed team, so needing both takes a required-reviewers rule per team.
  • Smaller: OTel SDK patches may change behavior for security fixes (Observability), verifiers trust what tools print (Did the Agent Do It Right?), retired Gemini IDs get rerouted (Engineering for the Next Agent), ITBench-AA top score and Claude Code compaction wording (Agentic Engineering), Cursor's thousands of subagents (One Agent to Hundreds), moved links on four pages.

Full changelog →