Work / 01 · In production on AWS EKS
An autonomous incident-investigation (RCA) agent.
- faster triage
- 480×faster triage
- tools
- 44tools
- data sources
- 12+data sources
- automated tests
- 1,553automated tests
01
Investigation replay
↵ replaysP1Lambda success rate below 95%· recompute-jobs-lambda · PagerDuty, fired by a Datadog monitor
t+15.0s
Done in 15.0 seconds. Confidence 0.93.
- 01parse_incident0.4sservice, window, severity extracted
- 02pagerduty.follow_alert0.9sresolved the monitor that fired
- 03lambda.inspect_runtimee11.8sinvocations clustered near timeout
- 04elk.search_logse32.6sanomaly · downstream HTTP 500s in the alert window
- 05github.read_deployed_codee21.4shandler calls that endpoint
- 06elk.trace_request_ids3.1sanomaly · failing ids followed across three services
- 07guardrail.review0.2sguardrail · rejected "Lambda runtime fault": no OOM, no cold-start spike
- 08rank_hypotheses4.6sranked 3 hypotheses, wrote Jira and Slack
Root cause · ranked #1 of 3
The success-rate drop came from the downstream recompute API returning errors. The Lambda itself was healthy.
0.93confidence
Jira draft · P1Investigate downstream recompute API failures behind Lambda success-rate alarm
02
2 hours vs 15 seconds
By hand~2 h
TraceLens~15 s
480×faster. The bar for 15 seconds is drawn to scale; it is meant to be hard to see.
03
Problem, approach, result
Problem
- Triage by hand took about two hours: guess the owner, stop at the first exception, assume the wrapper exception is the cause, treat Lambda as a black box, then post a vague Slack message.
- Four production clusters live in separate AWS accounts. Looking in the wrong one is easy and silent.
Approach
- A native tool-calling ReAct loop on Azure OpenAI (GPT-4o). No agent frameworks.
- 44 tools across 12+ sources: Elasticsearch/ELK, Prometheus, AWS (ASG, Spot, ELB health, Lambda runtime, CloudWatch), Datadog (monitors, DB dashboards, DBM queries, APM), Grafana, Airflow (MWAA), GitHub, Jira, Confluence, PagerDuty and an internal service catalog.
- It starts from a live PagerDuty feed and follows an alert back to the Datadog monitor that fired it (query, thresholds, runbook) before the first LLM call.
- Guardrails reject premature conclusions. It never falls back silently to the wrong cluster across 4 production clusters in separate AWS accounts.
- A self-learning knowledge graph on Apache AGE (PostgreSQL) with bi-temporal, provenance-carrying edges. Beliefs are versioned, not overwritten. A CodeGraph pipeline maps service topology across about 215 services.
- FastAPI with an SSE backend and a React UI. JWT/SSO auth. History in PostgreSQL and S3. Metrics pushed to VictoriaMetrics. A CLI client. 1,553 automated tests. WCAG 2.2 AA enforced in CI.
Result
- Ranked, confidence-scored hypotheses with evidence, a Jira-ready ticket and a Slack summary.
- About 15 seconds instead of about 2 hours of manual triage: 480× faster.
04
By hand vs TraceLens
| By hand | TraceLens |
|---|---|
| Guess the owner | Parse service, window, severity |
| Stop at the first exception | Search broad, narrow, trace request ids |
| Assume wrapper exceptions are the cause | Unwrap them |
| Treat Lambda as a black box | Read runtime data and deployed code |
| Vague Slack message | Ranked hypotheses and a Jira-ready summary |
05
Architecture
no agent frameworkHover or focus a node to see what it does and what it talks to. Pulses show data moving.
06