Sally Roth

SRE Lab

A reliability-first lab: small, working SRE tools and illustrated, step-through explainers. The tools cover error budgets and burn-rate alerts, telling real outages from blips, failing over when a model provider breaks, searching real postmortems by failure mode, and judging an agent's incident work by its evidence. The explainers cover incident management, observability, Kubernetes, Linux, infrastructure as code, delivery and security, and what breaks once you run them at scale. Engineering with AI agents has its own section further down.

Start hereEngineering on an Exponential, DrawnHow an engineer's work and mindset change when frontier models leapfrog each other every few weeks and measured capability doubles every few months: why planning horizons must stay long, what the trend lines say and don't, task sizes in human terms, engineering from 3 months to 10 years, the bottlenecks that don't double, and the skills that compound.updated

Tools

SLO Burn SimulatorSee how fast an error budget drains, which burn-rate alerts fire and when, and get matching Prometheus rules.updated Down or MeUptime checks with an AI classifier that decides whether a failure is real, and alerts only when both of its confidence thresholds are met.updated The Incident Files318 real postmortems, searchable by failure mode, plus the nine failure patterns that keep coming back and how companies warded them off.updated Plan BBreak an AI model on purpose and watch a backup from a different provider take over through Cloudflare AI Gateway, with which model answered and how long each attempt took.updated Did the Agent Do It Right?Three simulated agent runs on the same incident: one checks the evidence, one confidently blames the wrong service, one gets stuck retrying. Pydantic Evals judges each by its tool-call trace, not its explanation.updated

Reliability explainers

Sane Incident ManagementIncident response people can follow at 3am: severity, roles, the lifecycle, communication, sustainable on-call, blameless reviews, action items that land, and readiness.updated Kubernetes, DrawnHow Kubernetes works in 18 step-through diagrams, from kubectl apply to DNS, storage and upgrades, each with a short summary, gotchas, questions to think through and what breaks at scale.updated Linux, DrawnWhat a Linux node really does with memory and packets: page cache, the OOM killer, cgroups, a packet's path, conntrack, TCP queues and the first 60 seconds of performance analysis.updated Observability, DrawnHow telemetry flows: OpenTelemetry and the Collector, Prometheus scraping and PromQL, tracing and sampling, Loki logs, and alerting that wakes the right person.updated Taming CardinalityHow one label multiplies series and cost, where Prometheus pays for them, how to find the culprit, and how to drop, aggregate and govern before the bill arrives.updated Delivery, DrawnFrom commit to production: CI, build once and promote, GitOps with Argo CD, canaries, feature flags, safe schema migrations, supply-chain signing, and merge queues at scale.updated Infrastructure as Code, DrawnRunning IaC well on a team and across an org: remote state and locking, modules, environments, plan and apply in CI with policy checks, drift, safety rails, testing, and scale.updated Terraform, Five Years OnWhat changed since 1.0: moved, import and removed blocks, checks and tests, the license change and OpenTofu, ephemeral values, and splitting state at scale.updated Security for SREs, DrawnHow AWS decides allow or deny, federation instead of keys, TLS 1.3 and mTLS, secrets that rotate, network segmentation, least privilege and audit trails.updated Ownership at ScaleResearch on how very large companies decide who owns code and services: OWNERS files and CODEOWNERS, service registries synced from HR, routing pages, incidents, vulnerabilities and costs to owners, keeping it accurate through reorgs, and fleet-wide changes that route around owners.updated One Pattern, Three LanguagesEveryday SRE tasks (logs, APIs, AWS cleanup, Kubernetes, Prometheus, fleet checks, exporters) in Python, Go and Ruby, to compare how each language structures the same job, and what breaks at scale.updated

AI engineering

How coding agents change the work: running them safely at scale, building for the next model, and planning when capabilities move every few weeks.

Agentic Engineering, DrawnHow Stripe, Spotify, Shopify and others run hundreds of coding agents and ship safely: harnesses the model can't skip, verifiers it can't edit, no route to prod, risk-tiered review, and Lauren Tan's pstack.updated One Agent to HundredsA five-stage ladder from one coding agent you trust to hundreds in parallel: what each stage adds, what breaks without it, and the checks that say you're ready for the next. Built on Lauren Tan's pstack, with Anthropic, Cursor and Claude Code sources.updated Engineering for the Next Agent, DrawnBuild for the model that is coming, not the one you have: assets that compound across model generations, scaffolding to delete, capability trends with their caveats, evals, safe model upgrades, and boundaries and interfaces that outlive any one model.updated Graphs and Loops, DrawnStop prompting agents one message at a time and write the loops and graphs that prompt them: where the viral “building graphs and loops” quote really comes from, loops with a check and a stop, graphs of steps, branches and joins, whether to draw or describe them, and how to get your agent to build them.updated

Changelog

What each daily research pass changed, newest first. Every change is checked against a primary source.

  • Agentic Engineering: the Vercel microVM escape now has a CVE and fixed kernels, and a new item on an Agent SDK flaw where a session ID became a command-line flag.
  • One Agent to Hundreds: checks you have seen fail (pstack's new formal verification skill), Addy Osmani on layered verification instead of line-by-line review, and pstack's updated catch-up rules.
  • Did the Agent Do It Right?: graders can be reward-hacked too, and why each eval task gets its own microVM.
  • Smaller: DeepSeek's search and fetch billing for failover (Plan B), a Docker Hub token gotcha (Security), Microsoft on prompts as no enforcement boundary (Agentic), Boris Cherny on Claude Code Review (fleet).
  • New: Graphs and Loops, Drawn. What it means to stop prompting and build the loops and graphs that prompt your agents, who really said it, whether to draw or describe them, and how to get your agent to build them.
  • One Agent to Hundreds: Lauren Tan's “inverse pyramid” for trusting agent work, Birgitta Böckeler on dropping test-first prompts, Dex Horthy's SlopCodeBench results, the Claude Code team's risk-tiered review, and how Amp gates deploys without pull requests.
  • Plan B: DeepSeek-V4.1-Flash-1017 adds MIT-licensed open weights, and a new note on why failing over costs you the prompt cache.
  • Plan B: OpenCode's breach, tied to a distillation attack on users with linked provider keys, and what it means for keys you hand to a harness.
  • Engineering on an Exponential and Engineering for the Next Agent: Epoch's new price trend (about 13x cheaper a year for the same performance) replaces its March 2025 range.
  • Verification research on One Agent to Hundreds and Did the Agent Do It Right?: what agents' reports leave out, where judges place the first mistake, monitors fooled by edited transcripts, and self-grading loops.
  • Smaller: Claude Code 2.1.302 fork-subagent default (fleet), Copilot approvals preview (Ownership), an OpenTelemetry versioned-metrics gotcha (Observability), Anthropic's agent-hours data (Exponential), pstack wording (Agentic), a harness-dependence example and METR's cheating caveat, and this changelog.
  • Engineering on an Exponential: frontier models now leapfrog each other every few weeks, not every few months; a full source check of it and of Engineering for the Next Agent.
  • Fleet and agentic pages: Opus 5.5 fast mode out of research preview, Claude Code Review GA, context resets, review backpressure, XBOW's parallel campaigns, Amp's SOC 2 without pull requests.
  • Smaller: about 60 edits across agent evals, Linux, Kubernetes, Terraform and patterns; moved GitHub links on four pages; a site icon.

Full changelog →