| published by | Boris Chabeda |
|---|---|
| in blog | The New Stack |
| published date | 2026-11-18 |
| original entry | 200 OK is dead as an AI agent health signal. Here’s what comes next. |
A customer asks whether they can swap unworn boots for a size 10. The agent replies, in under a second, that it cannot help with that topic. The web service returns 200 OK. Latency is nominal, error counters are flat, and the Datadog dashboard stays solid green. A customer has just left, and nothing in the APM stack knows.
That is the failure this article is about: the kind that looks like success.
In lean manufacturing, this exact category of silent defect is solved using two twin principles: Poka-Yoke (mistake-proofing stations so bad parts cannot pass) and Jidoka (autonomation—building machines that detect anomalies themselves and automatically halt the line via an Andon emergency cord rather than waiting for downstream customer complaints).
“A customer has just left, and nothing in the APM stack knows. That is the failure this article is about: the kind that looks like success.”
Modern AI agent engineering needs to adopt this exact control-loop architecture, replacing passive health checks with active station barriers.
When we retrospectively ran an offline evaluation benchmark script against three weeks of archived Datadog logs, we found our primary intent classifier had silently dropped 4.2% in schema-valid classification accuracy—masked entirely by default fallback handlers returning 200 OK.
We started by conducting a manual, single-day audit of 100 randomly sampled 200 OK production traces pulled directly from our archived Datadog logs. Two engineers read the raw customer text and manually categorized every routing decision against a ground-truth spreadsheet rubric.
When we bucketed those 100 manual traces, the breakdown was stark:
OUT_OF_SCOPE.This manual audit revealed our critical blind spot. A false negative wastes a few cents in tokens; a schema crash raises a 500 error and pages on-call; but a false-positive block turns away a customer while returning a clean 200 OK. That audit gave us both our core metric to guard—the False Positive Block Rate (FPBR)—and the foundational insight needed to design our observability layer.
In our production customer support agent architecture, requests are rarely handled by a monolithic, generalist LLM. Instead, our deployments rely on a multi-agent orchestration pattern (router-specialist topology). An inbound user prompt first hits a triage router agent, which parses the query and dispatches it to downstream domain specialists (Billing, Account Access, Returns), or short-circuits it if it falls outside support bounds (OUT_OF_SCOPE).
Of all the components in this multi-agent stack, we evaluate the Intent Classification Agent first because it sits at the front door.
“An intent classifier false positive has the highest cascading blast radius in the multi-agent topology because it causes a silent, permanent customer drop-off that bypasses all downstream observability.”
If the intent classifier makes a false-positive error—misinterpreting a legitimate exchange or warranty claim as OUT_OF_SCOPE—it triggers an immediate short-circuit. Downstream specialists are never invoked, tokens are saved, and the system cheerfully returns a 200 OK refusal. An intent classifier false positive has the highest cascading blast radius in the multi-agent topology because it causes a silent, permanent customer drop-off that bypasses all downstream observability.
While the industry frequently defaults to “LLM-as-a-judge” frameworks (using models like GPT-4o or Llama 3) for agent monitoring, we deliberately deployed a fine-tuned encoder model (RoBERTa-base NLI cross-encoder).
To judge an intent classification output, you aren’t asking a model to generate free-form text; you are asking it to evaluate context, verify constraints, and output a strict classification label. Encoders are native classification architectures that look at the entire prompt context multidirectionally rather than predicting the next token autoregressively. This makes them faster, entirely deterministic, free of token-generation overhead, and natively suited for semantic entailment tasks.
We chose RoBERTa-base specifically for its infrastructure footprint: at ~1.5 GB RAM, it dropped onto our existing c7i.xlarge nodes with zero new cloud cost overhead.
If your customer queries frequently exceed 256 tokens or you are hunting for marginal accuracy gains across enterprise-wide clusters, DeBERTa-v3-base is a better option. DeBERTa’s disentangled attention mechanism yields a 1–2% bump in semantic entailment accuracy. However, that comes at the cost of a heavier CPU compute profile requiring careful thread pinning if co-hosted on legacy nodes alongside core routing services.
Before loading our evaluator in production, we lock PyTorch execution to a single dedicated core and run a container warmup pass to eliminate cold-start latency penalties:
Python
import os
from typing import Any, Dict
from enum import Enum
import torch
from transformers import pipeline
# Lock PyTorch math engine to 1 CPU core before model initialization
torch.set_num_threads(1)
torch.set_num_interop_threads(1)
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
class AllowedIntent(str, Enum):
RETURN_POLICY = 'RETURN_POLICY' # Exchanges classified here
ORDER_STATUS = 'ORDER_STATUS'
ACCOUNT_ACCESS = 'ACCOUNT_ACCESS'
OUT_OF_SCOPE = 'OUT_OF_SCOPE'
class RoBERTaEvaluator:
"""Local, thread-bounded NLI Cross-Encoder Evaluator."""
def __init__(self, model_path: str):
self.pipe = pipeline("text-classification", model=model_path, device="cpu")
self.id2label = self.pipe.model.config.id2label
# Engineering scar: Run dummy inference to force graph compilation during container startup
self.pipe("warmup prompt")
def check(self, query: str) -> Dict[str, Any]:
res = self.pipe(query[:2000])[0]
raw_label = res["label"]
if raw_label.startswith("LABEL_"):
idx = int(raw_label.replace("LABEL_", ""))
label_name = self.id2label.get(idx, raw_label)
else:
label_name = raw_label
try:
matched_intent = AllowedIntent(label_name)
except ValueError:
matched_intent = AllowedIntent.OUT_OF_SCOPE
return {"top_intent": matched_intent, "score": float(res["score"])}
To construct our training and validation datasets, we combined our 100 manually audited production traces with 20 synthetic edge cases, yielding a 120-query evaluation dataset partitioned into train, dev, and held-out test splits.
An evaluator is a second opinion, not ground truth. To validate our RoBERTa judge, we evaluated it against our held-out test split of 40 hand-labeled queries (20 positive, 20 negative):
Adding inline evaluation on the hot path introduces an immediate engineering anxiety: compute cost and latency.
We ran our classifier on a co-hosted c7i.xlarge instance (4 vCPUs, 8 GB RAM) capable of handling an average baseline of 100 requests per minute (~1.6 req/sec). To ensure our quality gate did not degrade our p95 SLA or starve adjacent services, we constrained the engine using PyTorch CPU optimization guardrails:
torch.set_num_threads(1).To fulfill our first Lean requirement, we implement a Control Poka-Yoke as the software equivalent of a manufacturing limit switch that intercepts malformed outputs before downstream tokens are wasted.
Before invoking any classifier, we establish our schema contract:
Python
from typing import Any, Optional, Tuple
from pydantic import BaseModel, Field, ValidationError
class IntentResult(BaseModel):
selected_intent: AllowedIntent
confidence: float = Field(ge=0.0, le=1.0)
def validate_schema(raw: Any) -> Tuple[Optional[IntentResult], Optional[Exception]]:
if not isinstance(raw, dict):
return None, TypeError('Raw output is not a dictionary')
try:
return IntentResult(
selected_intent=raw.get('intent'),
confidence=raw.get('confidence', 0.0)
), None
except ValidationError as exc:
return None, exc
validate_schema() acts as the station sensor: nothing crosses this line unvalidated. If an agent emits a malformed structure, execution immediately branches to an operator handoff without executing downstream steps.
Knowing our judge’s error profile allows us to mathematically derive our trip conditions. Below a 0.9 confidence score, the evaluator’s false-alarm rate spikes above 18%, causing operator fatigue. Therefore, we set our Andon page floor at 0.9.
A single agent turn ties together our previously defined schema validator and RoBERTa evaluator inside a nested OpenTelemetry trace tree. The parent span (intent_agent_turn) encompasses the primary interaction (primary_agent_interaction) and its downstream execution child spans (context_retrieval, prompt_render, llm_inference), followed synchronously by the local judge (nli_evaluation). When the judge detects a semantic mismatch, it triggers an andon_pulled event acting as our digital emergency stop cord:
Python
import hashlib
from typing import Any, Dict
from opentelemetry import trace, metrics
tracer = trace.get_tracer('intent_gate', '1.0.0')
meter = metrics.get_meter('intent_gate', '1.0.0')
disagreements_counter = meter.create_counter('intent_gate.disagreements')
def run_turn(conversation_id: str, query: str, prompt_version: str, agent, evaluator: RoBERTaEvaluator) -> Dict[str, Any]:
turn = {'prompt_version': prompt_version, 'disagreement_type': None}
with tracer.start_as_current_span('intent_agent_turn') as root:
root.set_attribute('gen_ai.conversation.id', conversation_id)
root.set_attribute('app.prompt.version', prompt_version)
root.set_attribute('app.query.sha256', hashlib.sha256(query.encode()).hexdigest()[:16])
with tracer.start_as_current_span('primary_agent_interaction') as s1:
with tracer.start_as_current_span('context_retrieval'): pass
with tracer.start_as_current_span('prompt_render'): pass
with tracer.start_as_current_span('llm_inference'):
raw = agent.classify(query=query, prompt_version=prompt_version)
result, error = validate_schema(raw)
s1.set_attribute('app.intent.schema_valid', result is not None)
if result is None:
s1.set_status(trace.Status(trace.StatusCode.ERROR, 'schema_violation'))
turn['action'] = 'HANDOFF_SCHEMA_FAILURE'
return turn
with tracer.start_as_current_span('nli_evaluation') as s2:
verdict = evaluator.check(query)
agent_intent, eval_intent = result.selected_intent, verdict['top_intent']
dtype = None
if agent_intent != eval_intent:
if agent_intent == AllowedIntent.OUT_OF_SCOPE:
dtype = 'SUSPECTED_FALSE_POSITIVE'
elif eval_intent == AllowedIntent.OUT_OF_SCOPE:
dtype = 'SUSPECTED_FALSE_NEGATIVE'
else:
dtype = 'INTENT_MISMATCH'
turn.update({
'agent_intent': agent_intent.value,
'evaluator_intent': eval_intent.value,
'evaluator_score': verdict['score'],
'disagreement_type': dtype
})
if dtype:
disagreements_counter.add(1, {'prompt_version': prompt_version, 'type': dtype})
s2.add_event('andon_pulled', {'type': dtype})
turn['action'] = 'HANDOFF_EVALUATOR_DISAGREEMENT'
return turn
turn['action'] = 'PROCEED_TO_GENERATION'
return turn
To maintain complete visibility across the agent’s lifecycle, our telemetry watches four distinct layers across the trace hierarchy:
The first time this observability gate ran in production was during the deployment of prompt version v1.2.0, shipped to tighten return abuse policies. The failure stemmed from a single un-audited prompt modifier that interpreted sizing exchanges as abuse vectors:
Diff
[System Prompt Diff: v1.1.0 -> v1.2.0]
- Handle customer return requests in accordance with standard 30-day policy windows.
+ Zero tolerance for policy circumvention. Intercept all ambiguous customer inquiries
+ regarding item modification, sizing swaps, or exchanges as potential return abuse vectors.
When a quality regression fires, the alert payload received on the #andon-agent-alerts Slack channel contains the exact trace ID, prompt version tag, candidate classification, and the evaluator’s confidence verdict:
JSON
{
"event": "QUALITY_REGRESSION_SUSPECTED",
"trace_id": "37a886864e32af95d4012a30acbda366",
"prompt_version": "v1.2.0",
"reason": "EVALUATOR_DISAGREEMENT",
"disagreement_type": "SUSPECTED_FALSE_POSITIVE",
"agent_intent": "OUT_OF_SCOPE",
"evaluator_intent": "RETURN_POLICY",
"evaluator_score": 0.9412
}
The incident timeline unfolded in under three minutes:
00:00:00: v1.2.0 deployed globally.00:12:02 (t = 0): The boots query fails 12 minutes after deployment. Agent returns OUT_OF_SCOPE; evaluator flags RETURN_POLICY (score 0.9412).00:12:04 (t + 2s): #andon-agent-alerts Slack channel receives trace webhook with debug link.00:14:38 (t + 156s): Traffic auto-drains and reverts to stable v1.1.0 following operator intervention. Total detection-to-reset time: 2m 36s.The offline evaluation suite missed this failure because its 250 static test cases contained zero colloquial variants for apparel or shoe sizing swaps.
200 OK refusals that pass passive APM health checks while turning away legitimate customers.torch.set_num_threads(1)) on co-hosted c7i.xlarge nodes, we achieved p50 = 42ms / p95 = 88ms latency with $0.00 in added cloud infrastructure cost.v1.2.0 accidentally flagged product sizing swaps as out-of-scope, the inline judge caught the regression at t=0, alerted #andon-agent-alerts via Slack webhook at t+2s, and enabled a full rollback to v1.1.0 within 2 minutes and 36 seconds.YOUTUBE.COM/THENEWSTACK
Tech moves fast, don't miss an episode. Subscribe to our YouTube channel to stream all our podcasts, interviews, demos, and more.