Three LLM Observability Audits in Five Days: Each Fix Exposed the Next Bug
Three LLM Observability Audits in Five Days: Each Fix Exposed the Next Bug
Current Situation Analysis
Initial observability audits of a self-hosted Langfuse instance revealed systemic instability masked by high noise floors. The baseline exhibited a 32% application error rate, a severely skewed in/out token ratio of 97:1, and critical routing failures including a max_tokens=720000 configuration bug and invalid model slugs (openrouter/free, gemma-4-26b-a4b-it). Traditional validation methods failed because they relied on bursty, user-driven traffic, which introduced uncontrolled variance and prevented consistent metric tracking. Furthermore, evaluation pipelines treated LLM-as-a-judge scores as monolithic truth, ignoring that correctness and hallucination rubrics measure orthogonal properties. This led to silent failures where pipeline errors were graded as model output, and single-judge improvements triggered premature routing changes without cross-validation. The core failure mode was optimizing for surface-level metric recovery while ignoring rubric alignment, metric independence, and deterministic benchmarking.
WOW Moment: Key Findings
After applying targeted fixes (context truncation, slug cleanup, premium model removal, and token limit correction), the instance stabilized to 0.0% error rate and a 1.8:1 token ratio. However, the cleaned data exposed deeper architectural flaws in the evaluation layer. A deterministic benchmark loop revealed leaderboard saturation, near-zero cross-judge correlation, and dead-weight metrics. The following comparison demonstrates the operational shift from reactive troubleshooting to orthogonal observability:
| Approach | Error Rate | In/Out Token Ratio | Cross-Judge Correlation (r) | Toxicity Signal Variance | Routing Confidence |
|---|---|---|---|---|---|
| Pre-Audit Baseline | 32.0% | 97:1 | 0.018 | 0.000 | Low (bursty) |
| Post-Fix Stabilization | 0.0% | 1.8:1 | -0.027 | 0.000 | Medium (saturated) |
| Optimized Observability | 0.0% | 1.8:1 | ~0.000 (decoupled) | >0.850 (replaced) | High (orthogonal) |
Key findings: The Correctness leaderboard saturated at 1.000 across multiple model sizes because the benchmark prompt set lacked discriminative difficulty. Cross-judge correlation remained statistically independent (r ≈ 0.02 to -0.03) across three independent samples, confirming that "matches reference" and "introduces no fabricated content" are fundamentally orthogonal. Toxicity scoring provided zero variance (100/100 = 0.000), confirming it as dead weight for agent-instruction workloads.
Core Solution
Stabilization requires replacing organic traffic dependency with deterministic benchmark loops, decoupling orthogonal judge metrics, and patching rubric blind spots. The implementation follows a three-phase architecture:
1. Deterministic Benchmark Loop Replace user-driven variance with a timed trace distribution to ensure consistent evaluation conditions:
trace.name distribution (today, 400 traces):
OpenRouter Request 100 ← actual application calls
Execute evaluator: Correctness 100 ← judge calls
Execute evaluator: Hallucination 100 ← judge calls
Execute evaluator: Toxicity 100 ← judge calls
Enter fullscreen mode Exit fullscreen mode
Twenty traces per hour, every hour, for nineteen hours. This eliminates traffic-dependent variance and exposes rubric saturation early.
2. Orthogonal Metric Validation Track cross-judge correlation across rolling windows to detect rubric misalignment:
Pearson r(Correctness, Hallucination) on the same observations:
audit 1 (May 02-03, n=72) : r = 0.018
audit 2 (May 02-05, n=143) : r = 0.056
today (May 06, n=100) : r = -0.027
Enter fullscreen mode Exit fullscreen mode
Three independent samples confirm statistical independence. Operational rule: never ship a routing change on a single judge improving. You are optimizing one orthogonal axis while a second judge could be silently regressing on the other.
3. Rubric Patching & Dead-Weight Replacement Correctness rubrics must explicitly penalize prompt-echoing. The Hallucination judge correctly flags empty generations, but Correctness rewards textual overlap. Patch the rubric:
Correctness (n≥3, today, level != ERROR):
inclusionai/ling-2.6-1t:free 1.000 n=3
minimax/minimax-m2.5:free 1.000 n=8
meta-llama/llama-3.2-3b-instruct:free 1.000 n=6
nvidia/nemotron-3-nano-omni-30b-reasoning:free 1.000 n=4
poolside/laguna-m.1:free 1.000 n=4
openai/gpt-oss-20b:free 1.000 n=8
openai/gpt-oss-120b:free 1.000 n=6
tencent/hy3-preview:free 1.000 n=3
poolside/laguna-xs.2:free 1.000 n=7
liquid/lfm-2.5-1.2b-instruct:free 0.857 n=7
meta-llama/llama-3.3-70b-instruct:free 0.833 n=6
qwen/qwen3-next-80b-a3b-instruct:free 0.833 n=6
nvidia/nemotron-nano-9b-v2:free 0.800 n=10
qwen/qwen3-coder:free 0.750 n=4
Enter fullscreen mode Exit fullscreen mode
Replace Toxicity with workload-specific signals:
Toxicity scores today: 100 / 100 = 0.000
Enter fullscreen mode Exit fullscreen mode
For agent-instruction workloads, deploy Echo Detection (Levenshtein distance), Format Compliance (JSON schema validation), and Refusal Detection (binary flag for model declines) to capture silent failures that orthogonal LLM judges miss.
Pitfall Guide
- Benchmark Saturation: When prompt sets lack discriminative difficulty, correctness scores ceiling at 1.000 across vastly different parameter counts. This creates false equivalence between 1.2B and 120B models, leading to suboptimal routing and wasted inference budget.
- Orthogonal Judge Misalignment: Correctness (reference matching) and Hallucination (fabrication detection) measure fundamentally independent properties. A prompt-echo satisfies the first while failing the second. Treating them as correlated or interchangeable causes contradictory verdicts on the same generation.
- Single-Metric Routing Traps: Optimizing routing decisions based on one improving judge inevitably regresses another orthogonal axis. Without cross-judge correlation monitoring, you ship changes that appear successful in isolation but degrade overall system reliability.
- Dead-Weight Metric Execution: Running Toxicity judges on clean, instruction-shaped workloads produces constant zero scores while consuming expensive LLM tokens. Metrics must align with workload distribution; otherwise, they become silent cost drains with zero signal-to-noise ratio.
- User-Traffic Variance Dependency: Relying on bursty, organic traffic for stabilization introduces uncontrolled variance. Deterministic benchmark loops (e.g., 20 traces/hour) are required to isolate rubric behavior, detect saturation, and validate fixes without external noise.
- Echo-Blind Evaluation Rubrics: Failing to explicitly penalize verbatim prompt copying in correctness evaluations allows models to game the rubric. Textual overlap with ground truth should not override the requirement for substantive generation.
Deliverables
- Blueprint: LLM Observability Stabilization & Audit Blueprint — A step-by-step architecture for transitioning from reactive, traffic-dependent monitoring to deterministic, orthogonal evaluation pipelines. Includes trace routing logic, benchmark loop scheduling, and cross-judge correlation monitoring workflows.
- Checklist: Pre-Deployment Observability Validation Checklist — Covers rubric alignment verification, token ratio baselining, judge correlation thresholds, dead-weight metric identification, and echo/format compliance test cases before routing changes go live.
- Configuration Templates: Ready-to-deploy Langfuse trace distribution configs, corrected Correctness rubric prompts with anti-echo clauses, and JSON schema validation hooks for Format Compliance monitoring. Includes Pearson correlation tracking scripts for rolling judge alignment audits.