03Detect04Understand

Know what broke, where, and why.

Detectors watch every span as it arrives. When something goes wrong, one incident opens with the failing span, the evidence and a diagnosis you can check, plus a recommended way back.

Start free trial Read the incidents docs

INC‑5, reconstructed

run
triage‑agent starts a support‑ticket‑resolution run in production and hands the research step to resolver‑agent.
tool
lookup_order calls orders-db: 503 Service Unavailable (connection pool exhausted).
×3
Third failure within 10 minutes: the tool_failure detector’s threshold.
INC‑5
Incident opened. Critical. Later failures are counted as occurrences (44 so far), not opened again.
evidence
Failure location and evidence: failing span 3eb91d0f65c815a4, the error message and an 18% tool error rate.
diagnosis
78%, high confidence. Root cause: orders-db returns 503. It cites ev‑1, ev‑2, ev‑3 and ev‑6, and it is marked LLM‑assisted.
next step
Suggested: pause the support‑ticket‑resolution workflow until orders-db recovers. The pause is enforced at the gateway: 423 workflow_paused.

Eleven detectors for how AI fails.

Stream detectors react to spans within about a second; sweep detectors check aggregates on a schedule. Enable, disable and tune each one per organization.

Production environments and critical workflows raise the severity by one level. Repeated detections of the same problem merge into the open incident.

Incident detectors
Detector	Fires when	Default severity
Agent loopagent_loop	An agent repeats an identical prompt, or exceeds its step limit, in one run	High
Repeated tool callrepeated_tool_call	The same tool is called with the same arguments again and again in one run	Medium
Timeouttimeout	Model or tool calls time out or run past their thresholds	Medium
Provider failureprovider_failure	A provider or model’s error rate spikes, or calls fail in a row	High
Tool failuretool_failure	A tool keeps failing, or a run ends in error at a tool call	High
Token anomalytoken_anomaly	A call uses far more tokens than the agent normally does	Medium
Cost anomalycost_anomaly	Cost per request jumps versus the agent’s 7-day baseline, or a run exceeds its cost limit	High
Stuck workflowstuck_workflow	A run started but never finished within the workflow’s threshold	High
Policy violationpolicy_violation	A request or tool call was blocked by, or broke, a policy	Medium
Error rateerror_rate	An agent’s error rate crosses the threshold over a window	High
Budget thresholdbudget_threshold	Spend crosses a budget’s alert thresholds; an incident at 100%	High

Every incident comes with its evidence.

The evidence pack is built from the spans around the failure the moment the incident opens: what happened, the failure location, numbered evidence items and the run’s span sequence.

1. The affected run, step by step.
2. Highlighted steps are part of the detected pattern.
3. Latency per step.

Run span sequence for the affected run, with the failing steps highlighted — Incident · Run span sequence · sample data from the built‑in demo organization

04Understand

Deterministic first. AI second. Always cited.

Diagnosis runs rule-based checks for each incident type first. When a model credential is available, a model then reasons over the evidence, and it has to cite it.

Deterministic findings: Rule-based checks for the incident type run first, and they complete even when no model is available.
LLM‑assisted: With the platform’s OpenAI key or your own OpenAI‑compatible credential, a model reasons over the evidence for a root cause, a category and alternative hypotheses.
Citations: Every claim cites evidence items by number. Citations that match no evidence are dropped.
Confidence: A score and a label you can check against the cited evidence, plus similar past incidents.

Suggested next step: follow the tool dependency outage runbook, check orders-db and its connection pool, then retry the affected runs — Incident · Diagnosis tab · sample data from the built‑in demo organization

1. A suggested next step, here the runbook for a tool dependency outage.
2. Recovery options, one click from the diagnosis.

From diagnosis to recovery, with you in control.

Every incident type comes with recommended actions, each with its risk, and versioned runbooks link themselves to the incidents they match.

Pause and stop: Pause an agent or workflow, or stop a run, from the incident. The gateway answers 423 or 409 within seconds.
Retry: Retry the affected runs through a signed webhook to your system once the dependency is healthy.
Escalate: Reach every alert channel: email, Slack and signed webhooks, by event type and minimum severity.
Approvals: On Business and Enterprise, high‑risk actions wait for a second person’s approval, and nobody approves their own request. Every action is permission‑checked and audited.
Response targets: Each severity has a response-time target, 15 minutes for critical, and Worksi tells you when one is missed.

Benchmark · 24 September 2026

How reliably failures are caught.

We run a failure benchmark against the product: scripted failure scenarios and healthy runs through the real gateway, with time to detect measured from the failing call to an open incident.

8of 8

Scripted failure scenarios detected

0 false alarms in 6 healthy runs in the same round. Median time to detect: 1.1 s.

Provider outage, provider_failure: 0.13 s
Policy violation, policy_violation: 0.54 s
Repeated tool call, repeated_tool_call: 0.72 s
Cost spike, token_anomaly: 0.97 s
Tool failure, tool_failure: 1.3 s
Agent loop, agent_loop: 1.9 s
Provider timeout, timeout: 5.0 s
Stuck workflow, stuck_workflow, 60 s threshold: 63.2 s

Time from the failing call to an open incident. 8 scripted failures, log scale.

Internal benchmark, 24 September 2026, one round. Single-replica Docker Compose stack on a 12 vCPU / 16 GB Linux host (gateway limited to 1.5 CPU / 512 MiB), load generated with k6. The built‑in simulator provider answered model calls with a pinned 1 ms latency so the gateway’s own overhead could be isolated. Every detected incident also had its failure location, a diagnosis and a linked runbook. The stuck‑workflow scenario is detected only after its 60‑second inactivity threshold. Your detection times depend on your thresholds and traffic, and every threshold is adjustable. Not covered: real provider latency and errors (simulator only), multiple replicas, cloud networking, TLS termination and managed data stores, multi-hour soak tests.

Break something on purpose.

The built‑in simulator fails on request. Send a burst of 503s and watch an incident open, with its evidence.

Start free trial Read the incidents docs

Terminal

# while the simulator is your default provider,# it fails on requestURL=https://gateway.worksi.ai/v1/chat/completionsfor i in $(seq 8); do  curl -s -o /dev/null "$URL" \    -H "Authorization: Bearer $AIW_API_KEY" \    -H "X-AIW-Simulate: error=503" \    -H "Content-Type: application/json" \    -d '{"model": "gpt-5-mini", "messages":         [{"role": "user", "content": "Hi"}]}'done# then open Operations → Incidents