← Back to inbox

Prod API error spike

Incident ID: org_devdocs:prod-api-5xx-spike

criticalinvestigatingbetter_stack
Updated 6/30/2026, 10:02:18 AM

Prod API error spike

Production 5xx rate above threshold

Org
org_devdocs
Service
api
Repository
devdocsorg/devdocsai-sre
Environment
production
Approval
pending
Duplicates
3

Operator Controls

Approval is explicit. Remediation only advances after approval, and production resolution only happens after live verification on sre.devdocs.ai.

Evidence

Alert receivedinfo
Accepted better_stack signal for api.
better_stack
{
  "orgId": "org_devdocs",
  "service": "api",
  "repository": "devdocsorg/devdocsai-sre",
  "environment": "production",
  "severity": "critical",
  "title": "Prod API error spike",
  "summary": "Production 5xx rate above threshold",
  "fingerprint": "prod-api-5xx-spike"
}
Blast radius scopedinfo
Org org_devdocs / repo devdocsorg/devdocsai-sre / env production.
sre-control-planedevdocsorg/devdocsai-sre
Initial triage hypothesisinfo
Likely api degradation surfaced through better_stack.
triage-agentsearch_service_tools

Trace

transition6/30/2026, 10:02:18 AM
Duplicate signal received from better_stack.
investigating
{
  "duplicateCount": 3
}
transition6/30/2026, 10:02:16 AM
Duplicate signal received from better_stack.
investigating
{
  "duplicateCount": 2
}
transition6/30/2026, 10:00:47 AM
Duplicate signal received from better_stack.
investigating
{
  "duplicateCount": 1
}
evidence6/30/2026, 10:00:43 AM
Seeded initial evidence and hypotheses for the operator console.
investigating
transition6/30/2026, 10:00:43 AM
Incident created from better_stack signal.
detected

Approval Ledger

No approval decisions recorded yet.

Hypotheses

Primary hypothesis: degraded upstream dependency or failed deploymenthigh
Current walking skeleton uses source/severity heuristics until live MCP evidence is attached.

Remediation Plan

Collect live evidence from connected providers before changing productionproposed
devdocs-mcp · search_service_tools
Risk: low
Preconditions: MCP API key available · provider account connected
Rollback: No-op; read-only.
Prepare origin/main-only fix and verify on production after Vercel buildproposed
github/vercel · github + vercel_token_auth
Risk: medium
Preconditions: Root cause confirmed · human approval captured · fix branch tested locally
Rollback: Revert on origin/main if production verification fails.

Verification

Production health endpoint returns okpending
Expected after deploy to sre.devdocs.ai.
Operator confirms impacted flow on productionpending
Required to call a production incident resolved.

RCA Draft

# RCA for Prod API error spike

- Severity: critical
- Source: better_stack
- Initial summary: Production 5xx rate above threshold
- Current state: live evidence collection pending in standalone SRE console.