Sally Roth

SRE Lab

A reliability-first lab: a few working SRE tools and step-through explainers, written for running these systems across many teams. The tools cover error budgets and burn-rate alerts, telling real outages from blips, searching real postmortems by failure mode, and judging an agent's incident work by its evidence. The explainers cover incident management, observability, Kubernetes, Linux, infrastructure as code, delivery and security. Engineering with AI agents has its own section further down.

Start hereEngineering on an ExponentialHow an engineer's work and mindset change when frontier models ship most weeks and capability compounds over months: planning for the curve, not the next release; separating what can change from what must last when the model builds most of the system; what the trend lines say and don't; task sizes in human terms; engineering from 3 months to 10 years; the bottlenecks that don't double; the operating-model shift; and what compounds in an engineering org.updated

Tools

SLO Burn SimulatorSee how fast an error budget drains, which burn-rate alerts fire and when, matching Prometheus rules, and how to run SLOs across thousands of services.updated Down or MeUptime checks with an AI classifier that decides whether a failure is real, and alerts only when both of its confidence thresholds are met.updated The Incident Files318 real postmortems, searchable by failure mode, plus the nine failure patterns that keep coming back and the controls that retire them.updated Did the Agent Do It Right?Three simulated agent runs on the same incident: one checks the evidence, one confidently blames the wrong service, one gets stuck retrying. Pydantic Evals judges each by its tool-call trace, not its explanation.updated

Reliability explainers

Incident ManagementIncident management as a system for hundreds of teams: severity as a company-wide contract, routing as data, many incidents at once, legal clocks, fleet on-call health, learning across incidents and metrics that mean something.updated KubernetesHow Kubernetes works in 18 step-through diagrams, across hundreds of teams and clusters: admission policy, capacity governance, the fleet upgrade program, platform health signals, and DNS, storage and network limits.updated LinuxWhat Linux does with memory and packets across thousands of hosts: base-image settings, PSI, fleet OOM policy, conntrack and TCP limits, network failure signals and finding the few bad hosts.updated ObservabilityRunning telemetry for hundreds of teams: budgets per team, the Collector pipeline as a platform, Prometheus topology, shared PromQL and SLO rules, trace propagation, sampling policy, log retention tiers, and paging as an org control loop.updated OwnershipResearch on how very large companies decide who owns code and services: OWNERS files and CODEOWNERS, service registries synced from HR, routing pages, incidents, vulnerabilities and costs to owners, keeping it accurate through reorgs, and fleet-wide changes that route around owners.updated Taming CardinalityHow one label multiplies series and cost, where Prometheus pays for them, how to find the culprit, and how to drop, aggregate and govern before the bill arrives.updated SecuritySecurity run as a system across hundreds of teams and accounts: org guardrails and policy as code, federated identity, secrets and supply chain as programs, vulnerability flow, fleet-wide detection, agents, and the numbers that show progress.updated DeliveryFrom commit to production, for one team and for hundreds: CI, build once and promote, drift control across clusters, rollout policy as a platform default, flag governance, schema-change guardrails, supply-chain signing, and merge queues.updated Infrastructure as CodeRunning IaC across hundreds of teams: state layout as blast-radius policy, shared modules, environments, one change-control gate with policy as code, drift routed to owners, safety rails, and testing.updated Terraform, Five Years OnWhat changed since 1.0, across a fleet: version governance, moved, import and removed blocks, checks and tests, Terraform or OpenTofu, ephemeral values, and Stacks.updated

AI engineering

How coding agents change the work: running them safely across an org, building for the next model, and planning when capabilities move every few weeks.

Agentic EngineeringHow Stripe, Spotify, Shopify and others run hundreds of coding agents and ship safely: harnesses the model can't skip, verifiers it can't edit, no route to prod, risk-tiered review, and Lauren Tan's pstack.updated One Agent to HundredsA five-stage ladder from one coding agent you trust to hundreds in parallel, built on Lauren Tan's pstack, then an org layer on top: one platform for every team, budgets and quotas per team, pooled limits, audit, and merge rate next to change failure rate.updated Engineering for the Next AgentBuild for the model that is coming, not the one you have: assets that compound across model generations, scaffolding to delete, capability trends with their caveats, evals, safe model upgrades, and boundaries and interfaces that outlive any one model.updated Graphs and LoopsRunning agent loops and graphs across hundreds of teams: loop sprawl and what it costs, loops with a check and a stop, graphs of steps, branches and joins, workflows reviewed as code, and what every unattended loop must declare before it runs.updated

Changelog

What each daily research pass changed, newest first. Every change is checked against a primary source.

  • One Agent to Hundreds: LiteLLM's root-cause report corrects the cause. The way in was a leaked, never-revoked key to its agent platform, whose agent configs held an org owner's unscoped GitHub token, not gateway logs. Also new: production agent monitors that miss most attacks from an attacker who knows them and don't watch compaction summaries, MCP jobs that keep running after the agent exits, Claude Code cloud sessions and Remote Control gated by role, and LangChain's one recipe for all its internal agents.
  • Agentic Engineering: a permission classifier sees only what it is shown, policy hooks can fail open, Vercel approves a whole plan with short-lived scoped tokens, and OpenAI now trains frontier models with no internet access after a model reached it through a DNS misconfiguration.
  • Did the Agent Do It Right?: test the monitor on replayed real transcripts, score it by what ships after retries, and pin benchmark task versions.
  • Security: agent config is a credential store, from LiteLLM's root-cause report.
  • Terraform, Five Years On: -minimal-refresh in 1.17 saves API calls but hides drift on anything the plan doesn't change, so provider upgrades get a full refresh.
  • Smaller: Postgres and MySQL lanes kept apart, role timeouts load only at login, and lock_timeout bounds lock waits, not runtime (Delivery), provider requirements accept variables in Terraform 1.17 (Terraform), Agent Sandbox v1.0.8 caps command output (Kubernetes), shared Terraform MCP servers answer without a token unless MCP_REQUIRE_TOKEN is set and a moved Pulumi link (IaC), a moved Google Cloud link (Security), pstack 0.15.32 statecharts, reflect edits kept out of the plugin cache, Thariq Shihipar's new label, Cindy Sridharan on reviewing every line, Addy Osmani's video evidence and HumanLayer's open-source release (One Agent to Hundreds).
  • One Agent to Hundreds: an agent's own compaction notes can carry it off task, as a UK AISI eval showed, and a monitor can miss compaction entirely. The org track adds OpenAI's approval rule for model gateways from 20 Nov, GitHub's Copilot defaults for anything left unset, stale instruction files that agents follow over the code, one Google Cloud project per team, usage counts for in-house skills and MCP servers, and Imprint's small-change review lane.
  • Agentic Engineering: Claude Code and Codex fix sandbox holes every week, so set a fleet minimum version and rerun escape tests on each upgrade.
  • Security: a summary the agent writes for itself is input, and a forged approval can survive compaction.
  • Did the Agent Do It Right?: a proof checked by a proof kernel is the strongest verifier, and the theorem is what you review.
  • Engineering for the Next Agent: Agent Skills joined AGENTS.md and MCP in the Agentic AI Foundation.
  • Smaller: three minors behind gets critical fixes until end of life (Kubernetes), a non-empty cluster guard on the inhibit rule and one page alert instead of "exactly once" (Observability, SLO Burn Simulator), terraform validate runs provider plugins offline (IaC), memory.reclaim takes anon as well as cache (Linux), GitHub's ownership numbers (Ownership), attempt budgets count failed calls (Did the Agent Do It Right?), Claude Sonnet 5.5's ARC harness gap, Codex 0.164.0 daemon logs, fast mode on Google Cloud, Thorsten Ball's one-thread incident and Anthropic's guide on folding subagents back in (One Agent to Hundreds), a moved Anthropic link (Agentic).
  • One Agent to Hundreds: a shared gateway's logs can hold every team's tokens, as LiteLLM's poisoned release showed, and an agent approving merges through a person's account hides who decided. Also new: a Claude API version that rejects sampling settings, tagging requests by end user so one ban doesn't stop every team, limiting which OpenAI service tiers projects use, and counting what the guardrails stopped.
  • Security: gateway logs are a credential store, so strip tokens before anything is logged.
  • Did the Agent Do It Right?: hide the agent's own success report from the judge, and pin the API version as well as the model.
  • Agentic Engineering: pin the model gateway's version, and trend Claude Code's new safety event.
  • Smaller: Uber's 2026 uReview numbers (Agentic), pstack 0.15.30 statecharts and Amp's self-hosted orbs (One Agent to Hundreds), zero token counts no longer create OpenTelemetry points (Cardinality).

Full changelog →