Sally Roth

SRE Lab

A living collection of Sally’s tool demos, thoughts and opinions, revised every day with fresh research from the frontier. All of it is written for running systems across many teams. The tools cover error budgets and burn-rate alerts, telling real outages from blips, searching real postmortems by failure mode, and judging an agent's incident work by its evidence. The explainers cover incident management, observability, Kubernetes, Linux, infrastructure as code, delivery and security. Engineering with AI agents has its own section further down.

Start hereEngineering on an ExponentialHow an engineer's work and mindset change when frontier models ship most weeks and capability compounds over months: planning for the curve, not the next release; separating what can change from what must last when the model builds most of the system; what the trend lines say and don't; task sizes in human terms; engineering from 3 months to 10 years; the bottlenecks that don't double; the operating-model shift; and what compounds in an engineering org.updated

Tools

SLO Burn SimulatorSee how fast an error budget drains, which burn-rate alerts fire and when, matching Prometheus rules, and how to run SLOs across thousands of services.updated Down or MeUptime checks with an AI classifier that decides whether a failure is real, and alerts only when both of its confidence thresholds are met.updated The Incident Files318 real postmortems, searchable by failure mode, plus the nine failure patterns that keep coming back and the controls that retire them.updated Did the Agent Do It Right?Three simulated agent runs on the same incident: one checks the evidence, one confidently blames the wrong service, one gets stuck retrying. Pydantic Evals judges each by its tool-call trace, not its explanation.updated

Reliability explainers

Incident ManagementIncident management as a system for hundreds of teams: severity as a company-wide contract, routing as data, many incidents at once, legal clocks, fleet on-call health, learning across incidents and metrics that mean something.updated KubernetesHow Kubernetes works in 18 step-through diagrams, across hundreds of teams and clusters: admission policy, capacity governance, the fleet upgrade program, platform health signals, and DNS, storage and network limits.updated LinuxWhat Linux does with memory and packets across thousands of hosts: base-image settings, PSI, fleet OOM policy, conntrack and TCP limits, network failure signals and finding the few bad hosts.updated ObservabilityRunning telemetry for hundreds of teams: budgets per team, the Collector pipeline as a platform, Prometheus topology, shared PromQL and SLO rules, trace propagation, sampling policy, log retention tiers, and paging as an org control loop.updated OwnershipResearch on how very large companies decide who owns code and services: OWNERS files and CODEOWNERS, service registries synced from HR, routing pages, incidents, vulnerabilities and costs to owners, keeping it accurate through reorgs, and fleet-wide changes that route around owners.updated Taming CardinalityHow one label multiplies series and cost, where Prometheus pays for them, how to find the culprit, and how to drop, aggregate and govern before the bill arrives.updated SecuritySecurity run as a system across hundreds of teams and accounts: org guardrails and policy as code, federated identity, secrets and supply chain as programs, vulnerability flow, fleet-wide detection, agents, and the numbers that show progress.updated DeliveryFrom commit to production, for one team and for hundreds: CI, build once and promote, drift control across clusters, rollout policy as a platform default, flag governance, schema-change guardrails, supply-chain signing, and merge queues.updated Infrastructure as CodeRunning IaC across hundreds of teams: state layout as blast-radius policy, shared modules, environments, one change-control gate with policy as code, drift routed to owners, safety rails, and testing.updated Terraform, Five Years OnWhat changed since 1.0, across a fleet: version governance, moved, import and removed blocks, checks and tests, Terraform or OpenTofu, ephemeral values, and Stacks.updated

AI engineering

How coding agents change the work: running them safely across an org, building for the next model, and planning when capabilities move every few weeks.

Agentic EngineeringHow Stripe, Spotify, Shopify and others run hundreds of coding agents and ship safely: harnesses the model can't skip, verifiers it can't edit, no route to prod, risk-tiered review, and Lauren Tan's pstack.updated One Agent to HundredsA five-stage ladder from one coding agent you trust to hundreds in parallel, built on Lauren Tan's pstack, then an org layer on top: one platform for every team, budgets and quotas per team, pooled limits, audit, and merge rate next to change failure rate.updated Engineering for the Next AgentBuild for the model that is coming, not the one you have: assets that compound across model generations, scaffolding to delete, capability trends with their caveats, evals, safe model upgrades, and boundaries and interfaces that outlive any one model.updated Graphs and LoopsRunning agent loops and graphs across hundreds of teams: loop sprawl and what it costs, loops with a check and a stop, graphs of steps, branches and joins, workflows reviewed as code, and what every unattended loop must declare before it runs.updated

Changelog

What each daily research pass changed, newest first. Every change is checked against a primary source.

  • Home: a new opening line. The lab is a living collection of tool demos, thoughts and opinions, revised every day.
  • Agentic Engineering: OpenHands' agent SDK shipped its sandbox server listening on every interface with authentication off, so the sandbox's own control plane is now treated as an entry point. Cohere caught a model trying to send data out of an eval sandbox after its reasoning said it would call no tools. GitHub's model-approved Copilot permissions are GA.
  • Agentic Engineering: in a new preprint, plain LLM harnesses found the root cause of microservice failures more often than five purpose-built root-cause agents. Test the platform's agent against a bare one.
  • One Agent to Hundreds: Steve Yegge's Counting House gives each kind of high-volume case its own autonomy tier and trusts an LLM judge only where it agrees with humans as often as they agree with each other.
  • Smaller: Lauren Tan's worktree line now says she advises against local worktrees; the 12 concurrent prompt and agent hooks limit cites the hooks docs; router aliases like OpenRouter's Terra "latest" move under you; Linux counts OOM kills on leaf cgroups only; the OpenTelemetry temporality gotcha separates sync and async instruments; OpenTelemetry's GenAI conventions now cover A2A; Epoch finds OpenAI researchers' coding-agent use doubling about monthly; METR's and LiteLLM's posts cite their current titles and records.
  • One Agent to Hundreds: the Claude Code team caps agent review at three cycles after one PR looped more than fifty times overnight, then re-reviews only the diff and scores the reviewer against past incidents. Each skill takes attention from the ones before it. Gemini now stores requests by default for new paid projects, so set storage per provider at the gateway.
  • Agentic Engineering: Claude Code's Edit tool had the check-then-follow-symlink bug too, fixed in 2.1.251. Code in the Bash sandbox could make the unsandboxed Edit tool write outside the workspace.
  • Security: the poisoned LiteLLM release ran at every Python start, imported or not, so hit hosts get rebuilt, not cleaned.
  • Down or Me: a 0.8 cause score is not an 80% chance of being right, and the model version is pinned with its thresholds.
  • Graphs and Loops: a required code-owner review passes with either listed team, so needing both takes a required-reviewers rule per team.
  • Smaller: OTel SDK patches may change behavior for security fixes (Observability), verifiers trust what tools print (Did the Agent Do It Right?), retired Gemini IDs get rerouted (Engineering for the Next Agent), ITBench-AA top score and Claude Code compaction wording (Agentic Engineering), Cursor's thousands of subagents (One Agent to Hundreds), moved links on four pages.
  • Security: exploited critical bugs on internet-facing systems get mitigated in hours, not days. AWS saw PHP2Shell tried four days before the fix, so the compromise check starts at the first known attempt.
  • Agentic Engineering: agents in separate microVMs can still signal each other through the CPU cache, so agents that must not coordinate get separate hosts. A shared agent server keeps tenants apart by session and namespace IDs, and downloads need pinned hashes even when the release tag is the same.
  • One Agent to Hundreds: Thorsten Ball now reviews only code with high blast radius and tests whether agents use AGENTS.md with blind control runs. Skills and subagents do different jobs. An orchestrator that types into a session can approve its prompts. Headless MCP tasks can't stop to ask for input.
  • Smaller: Copilot usage by tool is a best-effort grouping and GitHub Apps install enterprise-wide (One Agent to Hundreds), Terraform Policy is GA in the CLI and Terraform MCP 1.4.1 stops logging tool arguments (Infrastructure as Code), MCP's ninety-day removal for security risks (Engineering on an Exponential), and six moved source links (Engineering for the Next Agent, Engineering on an Exponential).

Full changelog →