Sally Roth

SRE Lab

Small, working tools and step-through explainers for reliability engineering. The tools cover reliability targets, telling real outages from blips, and the failures that keep coming back. The explainers cover how the systems actually work, from Kubernetes and Linux to infrastructure as code, delivery, observability, security, incident response and agentic engineering, and what breaks once you run them at scale.

Start hereEngineering on an Exponential, DrawnHow an engineer's work and mindset change when frontier models leapfrog each other every few weeks and measured capability doubles every few months: why planning horizons must stay long, what the trend lines say and don't, task sizes in human terms, engineering from 3 months to 10 years, the bottlenecks that don't double, and the skills that compound.updated
SLO Burn SimulatorSee how fast an error budget drains, which burn-rate alerts fire and when, and get matching Prometheus rules.updated Down or MeUptime checks with an AI classifier that decides whether a failure is real, and alerts only when both of its confidence thresholds are met.updated Plan BBreak an AI model on purpose and watch a backup from a different provider take over through Cloudflare AI Gateway, with which model answered and how long each attempt took.updated Did the Agent Do It Right?Three simulated agent runs on the same incident: one checks the evidence, one confidently blames the wrong service, one gets stuck retrying. Pydantic Evals judges each by its tool-call trace, not its explanation.updated The Incident Files318 real postmortems, searchable by failure mode, plus the nine failure patterns that keep coming back and how companies warded them off.updated Kubernetes, DrawnHow Kubernetes works in 18 step-through diagrams, from kubectl apply to DNS, storage and upgrades, each with a 30-second answer, gotchas, a quiz and what breaks at scale.updated Linux, DrawnWhat a Linux node really does with memory and packets: page cache, the OOM killer, cgroups, a packet's path, conntrack, TCP queues and the first 60 seconds of performance analysis.updated Sane Incident ManagementIncident response people can follow at 3am: severity, roles, the lifecycle, communication, sustainable on-call, blameless reviews, action items that land, and readiness.updated Taming CardinalityHow one label multiplies series and cost, where Prometheus pays for them, how to find the culprit, and how to drop, aggregate and govern before the bill arrives.updated Terraform, Five Years OnWhat changed since 1.0: moved, import and removed blocks, checks and tests, the license change and OpenTofu, ephemeral values, and splitting state at scale.updated Delivery, DrawnFrom commit to production: CI, build once and promote, GitOps with Argo CD, canaries, feature flags, safe schema migrations, supply-chain signing, and merge queues at scale.updated Observability, DrawnHow telemetry flows: OpenTelemetry and the Collector, Prometheus scraping and PromQL, tracing and sampling, Loki logs, and alerting that wakes the right person.updated Security for SREs, DrawnHow AWS decides allow or deny, federation instead of keys, TLS 1.3 and mTLS, secrets that rotate, network segmentation, least privilege and audit trails.updated One Pattern, Three LanguagesEveryday SRE tasks (logs, APIs, AWS cleanup, Kubernetes, Prometheus, fleet checks, exporters) in Python, Go and Ruby, to compare how each language structures the same job, and what breaks at scale.updated Infrastructure as Code, DrawnRunning IaC well on a team and across an org: remote state and locking, modules, environments, plan and apply in CI with policy checks, drift, safety rails, testing, and scale.updated Agentic Engineering, DrawnHow Stripe, Spotify, Shopify and others run hundreds of coding agents and ship safely: harnesses the model can't skip, verifiers it can't edit, no route to prod, risk-tiered review, and Lauren Tan's pstack.updated Engineering for the Next Agent, DrawnBuild for the model that is coming, not the one you have: assets that compound across model generations, scaffolding to delete, capability trends with their caveats, evals, safe model upgrades, and boundaries and interfaces that outlive any one model.updated One Agent to HundredsA five-stage ladder from one coding agent you trust to hundreds in parallel: what each stage adds, what breaks without it, and the checks that say you're ready for the next. Built on Lauren Tan's pstack, with Anthropic, Cursor and Claude Code sources.updated Ownership at ScaleResearch on how very large companies decide who owns code and services: OWNERS files and CODEOWNERS, service registries synced from HR, routing pages, incidents, vulnerabilities and costs to owners, keeping it accurate through reorgs, and fleet-wide changes that route around owners.updated

Changelog

What each daily research pass changed, newest first. Every change is checked against a primary source.

  • One Agent to Hundreds: Lauren Tan's “inverse pyramid” for trusting agent work, Birgitta Böckeler on dropping test-first prompts, Dex Horthy's SlopCodeBench results, the Claude Code team's risk-tiered review, and how Amp gates deploys without pull requests.
  • Plan B: DeepSeek-V4.1-Flash-1017 adds MIT-licensed open weights, and a new note on why failing over costs you the prompt cache.
  • Plan B: OpenCode's breach, tied to a distillation attack on users with linked provider keys, and what it means for keys you hand to a harness.
  • Engineering on an Exponential and Engineering for the Next Agent: Epoch's new price trend (about 13x cheaper a year for the same performance) replaces its March 2025 range.
  • Verification research on One Agent to Hundreds and Did the Agent Do It Right?: what agents' reports leave out, where judges place the first mistake, monitors fooled by edited transcripts, and self-grading loops.
  • Smaller: Claude Code 2.1.302 fork-subagent default (fleet), Copilot approvals preview (Ownership), an OpenTelemetry versioned-metrics gotcha (Observability), Anthropic's agent-hours data (Exponential), pstack wording (Agentic), a harness-dependence example and METR's cheating caveat, and this changelog.
  • Engineering on an Exponential: frontier models now leapfrog each other every few weeks, not every few months; a full source check of it and of Engineering for the Next Agent.
  • Fleet and agentic pages: Opus 5.5 fast mode out of research preview, Claude Code Review GA, context resets, review backpressure, XBOW's parallel campaigns, Amp's SOC 2 without pull requests.
  • Smaller: about 60 edits across agent evals, Linux, Kubernetes, Terraform and patterns; moved GitHub links on four pages; a site icon.
  • Plan B: Meta's Muse Spark Open weights make open models a failover tier again.
  • Agent security: the SneakPeek preview-pane flaw, a Vercel microVM escape, Letta subagent permissions, a Deep Agents symlink fix.
  • Smaller: 27 edits across nine pages, including Kubernetes readiness, reviewer simulations, Claude's OpenTelemetry GenAI traces, Prometheus MCP and GitHub autofix.
  • Gemini 4 Argon GA, added as a dated example of an API change; pstack 0.15.24 updates on Agentic Engineering.
  • Smaller: 21 edits across agentic, fleet, Terraform, IaC, Kubernetes, security and delivery (Kiro sandbox GA, a Copilot for Visual Studio CVE, Terraform MCP, kubectl-ai, GitHub Actions workload identity).
  • 15 fixes on seven pages: IaC drift and lock timeouts, AWS SourceIdentity provenance, Linux effective memory.min, Transparent Huge Pages stalls, Prometheus 3.15 native histograms.
  • Smaller: a dead link replaced and per-version Terraform changelog links.
  • The Incident Files: two new records, Docker Hub PAT transfer and Fly.io readiness checks (318 in all).
  • Smaller: TLS 1.3 now cites RFC 9846, an agent-team permissions note, a restored Cloudflare incident link and Kubernetes doc links.
  • The Incident Files: six new records (310 to 316).
  • Accuracy updates: Collector component renames, agent-sandbox status, Terraform 1.16 lifecycle destroy, ECR replication, GitHub Actions cache limits.
  • Smaller: 43 moved links updated across 10 pages.
  • 39 verified fixes in 18 files: stale versions, Datadog renames, the MCP 2026-07-28 spec, SLSA v1.2, OpenTelemetry Profiles alpha, and three Incident Files records.
  • Smaller: moved links and accuracy fixes in the ownership essay.