Sally Roth

SRE Lab

A reliability-first lab: a few working SRE tools and step-through explainers, written for running these systems across many teams. The tools cover error budgets and burn-rate alerts, telling real outages from blips, searching real postmortems by failure mode, and judging an agent's incident work by its evidence. The explainers cover incident management, observability, Kubernetes, Linux, infrastructure as code, delivery and security. Engineering with AI agents has its own section further down.

Start hereEngineering on an ExponentialHow an engineer's work and mindset change when frontier models ship most weeks and capability compounds over months: planning for the curve, not the next release; separating what can change from what must last when the model builds most of the system; what the trend lines say and don't; task sizes in human terms; engineering from 3 months to 10 years; the bottlenecks that don't double; the operating-model shift; and what compounds in an engineering org.updated

Tools

SLO Burn SimulatorSee how fast an error budget drains, which burn-rate alerts fire and when, matching Prometheus rules, and how to run SLOs across thousands of services.updated Down or MeUptime checks with an AI classifier that decides whether a failure is real, and alerts only when both of its confidence thresholds are met.updated The Incident Files318 real postmortems, searchable by failure mode, plus the nine failure patterns that keep coming back and the controls that retire them.updated Did the Agent Do It Right?Three simulated agent runs on the same incident: one checks the evidence, one confidently blames the wrong service, one gets stuck retrying. Pydantic Evals judges each by its tool-call trace, not its explanation.updated

Reliability explainers

Incident ManagementIncident management as a system for hundreds of teams: severity as a company-wide contract, routing as data, many incidents at once, legal clocks, fleet on-call health, learning across incidents and metrics that mean something.updated KubernetesHow Kubernetes works in 18 step-through diagrams, across hundreds of teams and clusters: admission policy, capacity governance, the fleet upgrade program, platform health signals, and DNS, storage and network limits.updated LinuxWhat Linux does with memory and packets across thousands of hosts: base-image settings, PSI, fleet OOM policy, conntrack and TCP limits, network failure signals and finding the few bad hosts.updated ObservabilityRunning telemetry for hundreds of teams: budgets per team, the Collector pipeline as a platform, Prometheus topology, shared PromQL and SLO rules, trace propagation, sampling policy, log retention tiers, and paging as an org control loop.updated OwnershipResearch on how very large companies decide who owns code and services: OWNERS files and CODEOWNERS, service registries synced from HR, routing pages, incidents, vulnerabilities and costs to owners, keeping it accurate through reorgs, and fleet-wide changes that route around owners.updated Taming CardinalityHow one label multiplies series and cost, where Prometheus pays for them, how to find the culprit, and how to drop, aggregate and govern before the bill arrives.updated SecuritySecurity run as a system across hundreds of teams and accounts: org guardrails and policy as code, federated identity, secrets and supply chain as programs, vulnerability flow, fleet-wide detection, agents, and the numbers that show progress.updated DeliveryFrom commit to production, for one team and for hundreds: CI, build once and promote, drift control across clusters, rollout policy as a platform default, flag governance, schema-change guardrails, supply-chain signing, and merge queues.updated Infrastructure as CodeRunning IaC across hundreds of teams: state layout as blast-radius policy, shared modules, environments, one change-control gate with policy as code, drift routed to owners, safety rails, and testing.updated Terraform, Five Years OnWhat changed since 1.0, across a fleet: version governance, moved, import and removed blocks, checks and tests, Terraform or OpenTofu, ephemeral values, and Stacks.updated

AI engineering

How coding agents change the work: running them safely across an org, building for the next model, and planning when capabilities move every few weeks.

Agentic EngineeringHow Stripe, Spotify, Shopify and others run hundreds of coding agents and ship safely: harnesses the model can't skip, verifiers it can't edit, no route to prod, risk-tiered review, and Lauren Tan's pstack.updated One Agent to HundredsA five-stage ladder from one coding agent you trust to hundreds in parallel, built on Lauren Tan's pstack, then an org layer on top: one platform for every team, budgets and quotas per team, pooled limits, audit, and merge rate next to change failure rate.updated Engineering for the Next AgentBuild for the model that is coming, not the one you have: assets that compound across model generations, scaffolding to delete, capability trends with their caveats, evals, safe model upgrades, and boundaries and interfaces that outlive any one model.updated Graphs and LoopsRunning agent loops and graphs across hundreds of teams: loop sprawl and what it costs, loops with a check and a stop, graphs of steps, branches and joins, workflows reviewed as code, and what every unattended loop must declare before it runs.updated

Changelog

What each daily research pass changed, newest first. Every change is checked against a primary source.

  • One Agent to Hundreds: the Claude Code team caps agent review at three cycles after one PR looped more than fifty times overnight, then re-reviews only the diff and scores the reviewer against past incidents. Each skill takes attention from the ones before it. Gemini now stores requests by default for new paid projects, so set storage per provider at the gateway.
  • Agentic Engineering: Claude Code's Edit tool had the check-then-follow-symlink bug too, fixed in 2.1.251. Code in the Bash sandbox could make the unsandboxed Edit tool write outside the workspace.
  • Security: the poisoned LiteLLM release ran at every Python start, imported or not, so hit hosts get rebuilt, not cleaned.
  • Down or Me: a 0.8 cause score is not an 80% chance of being right, and the model version is pinned with its thresholds.
  • Graphs and Loops: a required code-owner review passes with either listed team, so needing both takes a required-reviewers rule per team.
  • Smaller: OTel SDK patches may change behavior for security fixes (Observability), verifiers trust what tools print (Did the Agent Do It Right?), retired Gemini IDs get rerouted (Engineering for the Next Agent), ITBench-AA top score and Claude Code compaction wording (Agentic Engineering), Cursor's thousands of subagents (One Agent to Hundreds), moved links on four pages.
  • Security: exploited critical bugs on internet-facing systems get mitigated in hours, not days. AWS saw PHP2Shell tried four days before the fix, so the compromise check starts at the first known attempt.
  • Agentic Engineering: agents in separate microVMs can still signal each other through the CPU cache, so agents that must not coordinate get separate hosts. A shared agent server keeps tenants apart by session and namespace IDs, and downloads need pinned hashes even when the release tag is the same.
  • One Agent to Hundreds: Thorsten Ball now reviews only code with high blast radius and tests whether agents use AGENTS.md with blind control runs. Skills and subagents do different jobs. An orchestrator that types into a session can approve its prompts. Headless MCP tasks can't stop to ask for input.
  • Smaller: Copilot usage by tool is a best-effort grouping and GitHub Apps install enterprise-wide (One Agent to Hundreds), Terraform Policy is GA in the CLI and Terraform MCP 1.4.1 stops logging tool arguments (Infrastructure as Code), MCP's ninety-day removal for security risks (Engineering on an Exponential), and six moved source links (Engineering for the Next Agent, Engineering on an Exponential).
  • One Agent to Hundreds: LiteLLM's root-cause report corrects the cause. The way in was a leaked, never-revoked key to its agent platform, whose agent configs held an org owner's unscoped GitHub token, not gateway logs. Also new: production agent monitors that miss most attacks from an attacker who knows them and don't watch compaction summaries, MCP jobs that keep running after the agent exits, Claude Code cloud sessions and Remote Control gated by role, and LangChain's one recipe for all its internal agents.
  • Agentic Engineering: a permission classifier sees only what it is shown, policy hooks can fail open, Vercel approves a whole plan with short-lived scoped tokens, and OpenAI now trains frontier models with no internet access after a model reached it through a DNS misconfiguration.
  • Did the Agent Do It Right?: test the monitor on replayed real transcripts, score it by what ships after retries, and pin benchmark task versions.
  • Security: agent config is a credential store, from LiteLLM's root-cause report.
  • Terraform, Five Years On: -minimal-refresh in 1.17 saves API calls but hides drift on anything the plan doesn't change, so provider upgrades get a full refresh.
  • Smaller: Postgres and MySQL lanes kept apart, role timeouts load only at login, and lock_timeout bounds lock waits, not runtime (Delivery), provider requirements accept variables in Terraform 1.17 (Terraform), Agent Sandbox v1.0.8 caps command output (Kubernetes), shared Terraform MCP servers answer without a token unless MCP_REQUIRE_TOKEN is set and a moved Pulumi link (IaC), a moved Google Cloud link (Security), pstack 0.15.32 statecharts, reflect edits kept out of the plugin cache, Thariq Shihipar's new label, Cindy Sridharan on reviewing every line, Addy Osmani's video evidence and HumanLayer's open-source release (One Agent to Hundreds).

Full changelog →