Durable SRE Investigations: Putting HolmesGPT into production
Martez Killens
Solutions Engineer
Alice Gibbons
Customer Success Lead
The recording is available now. This was the second webinar in our series on building practical agents on Diagrid Catalyst and it was the SRE version: an agent your on-call team can actually run. Watch the full session below.
We started the session with a Durable SRE Investigator. It was a Chainlit chat UI in front of HolmesGPT that triaged live incidents. We asked it "auth-service is in CrashLoopBackOff, what's going on?" and HolmesGPT reasoned across Kubernetes, ArgoCD, Prometheus, Grafana, GitHub, and MongoDB to pick the right tools and assemble an answer. Because every tool call was recorded with its inputs and outputs, on-call could scroll back through any past investigation, inspect what each tool returned, and re-run any individual step against the current state of your systems. That is what turned the agent from a one-shot answer engine into something on-call could interrogate and trust during a real incident.
We also shared what we learned building it: why we picked HolmesGPT and Chainlit so we were not writing the SRE brain or the chat UI from scratch, how feeding conversation history back into each query turned the agent from a question-answering box into something on-call could iterate with, and where small choices, like how skills match a question and which toolsets we exposed, changed how usable the agent felt in practice.
Durability is what made any of this credible. Most SRE agent demos do not survive a pod restart. Wrapping the agent in a Catalyst workflow persisted every LLM call and every tool invocation as a step, so the investigation picked up where it left off if anything died mid-run.
We closed with an agent you can run yourself: an Incident Docs Agent that reads runbooks and postmortems from Azure Blob or S3 and surfaces the right precedent when a new incident comes in. The repo is yours to clone, deploy on Catalyst, and take further.
Who this is for
- Platform and SRE teams running Diagrid Catalyst (or evaluating it) for agentic AI
- Engineers building production AI agents who need durability past the happy path
- AIOps and observability teams evaluating agentic AI for incident response
- HolmesGPT users looking to put it into production beyond ad-hoc CLI use
- Architects deciding between bespoke agent infrastructure and a managed runtime
What the demo covered
Live, end-to-end. We opened the Chainlit UI, asked "auth-service is in CrashLoopBackOff, what's going on?", and watched HolmesGPT call kubectl, ArgoCD, Prometheus, and GitHub MCP in sequence, with each tool invocation streamed in as a durable step. The headline moment: /replay <instance_id> <seq> re-ran any individual tool call from a past investigation directly against current state: no LLM, no workflow restart, no token spend. The same primitives that make the workflow crash-safe make every step independently inspectable and re-runnable.
