Incidents & alerts

Worksi watches your AI traffic and opens an incident when something goes wrong: a loop, a failing tool, a provider outage, a cost spike, a stuck workflow or a policy violation. Each incident comes with evidence, a diagnosis and suggested next steps, and can reach your team by email, Slack or webhook.

Detectors#

Eleven detectors run continuously. Stream detectors react to spans as they arrive; sweep detectors check aggregates on a schedule. Tune them in Operations → Detectors: switch them on or off, override the severity and adjust thresholds.

Production environments and critical workflows raise the severity by one level. Repeated detections of the same problem are merged into one open incident instead of opening new ones.

Lifecycle and response times#

Incidents are numbered INC-1, INC-2, … per organization and move through open, acknowledged, investigating, mitigated, resolved and closed. Each severity has a target response time — critical 15 minutes, high 1 hour, medium 4 hours, low 24 hours — and Worksi notifies you when an incident passes it. Acknowledge, assign, escalate, comment and resolve from Operations → Incidents; everything is audited and shown on the incident’s timeline.

Evidence and diagnosis#

Each incident carries an evidence pack built from up to 200 spans: a headline, the exact failure location (span, agent, tool, step and error), numbered evidence items, the run’s span sequence, cost and token metrics, and the affected agents, workflows, models and tools. Previews follow your capture mode.

The diagnosis runs deterministic checks first, then, when a model credential is available, asks a model to reason over the evidence: through the platform’s OpenAI key, your organization’s OpenAI credential, or the OpenAI‑compatible credential you set to Use for platform diagnosis and embeddings in Settings → Providers. The model must cite evidence items; citations that don’t exist are discarded, and without a valid citation the diagnosis stays deterministic. You see the root cause and its category, a confidence score, the cited evidence, alternative hypotheses and a suggested next step. Similar past incidents and the best-matching runbook are linked automatically.

LLM-assisted diagnoses with the platform key are counted per month by plan (50 on Starter, 500 on Pro, 5,000 on Business). With your own credential they are unlimited.

Recovery#

Recommendations on each incident propose concrete actions with their risk: stop a run, pause an agent or workflow, retry through a signed webhook, switch model, escalate, or follow a runbook. Pausing takes effect at the gateway within seconds (423 agent_paused). High-risk actions can require a second person’s approval. Every action is permission-checked, audited and recorded on the incident timeline.

Alerts#

Create channels in Operations → Alerts: email on every plan, Slack and webhooks on Pro and above. Pick the events (incident.created, incident.escalated, incident.resolved, approval.requested, budget.threshold, action.executed) and a minimum severity, then send a test. Escalations reach every channel regardless of severity. Each delivery gets up to three attempts and is logged per channel.

Webhook deliveries are JSON:

{"event_type": "incident.created", "organization": "…", "incident": {"id": "…", "number": 42, "title": "…", "severity": "high", "status": "open",              "type": "tool_failure", "agent": "resolver-agent", "workflow": "support-ticket-resolution",              "url": "https://worksi.ai/operations/incidents/…"}, "occurred_at": "…"}

They are signed: X-AIW-Timestamp carries the Unix time and, when the channel has a secret, X-AIW-Signature carries sha256= plus the HMAC-SHA256 of "<timestamp>." + body. Reject deliveries with a missing timestamp or one more than five minutes off:

import hashlib, hmac, time def verify(secret: str, raw_body: bytes,           timestamp: str, signature: str) -> bool:    # X-AIW-Timestamp: reject anything older than five minutes    if not timestamp.isdigit():        return False    if abs(time.time() - int(timestamp)) > 300:        return False    signed = f"{timestamp}.".encode() + raw_body    digest = hmac.new(secret.encode(), signed, hashlib.sha256)    # compare with X-AIW-Signature in constant time    expected = "sha256=" + digest.hexdigest()    return hmac.compare_digest(expected, signature)