TL;DR - Emissary enabled Middleware to build an AI system that judges every alert as Actionable or Likely Noise before the notification is sent. Powered by our novel Decision LM, a full alert assessment costs under $0.0001 - cheap enough to run proactively on every issue rather than a sampled subset, and fast enough (<1s) to land inside the Slack message before an engineer has even looked at it. Alert triage moved from a post-hoc dashboard exercise to the critical path of an AI SRE.

Introduction

Middleware is a full-stack observability platform: infrastructure monitoring, APM, logs, RUM, synthetic monitoring, and LLM observability in one unified experience. At the center of it sits OpsAI, Middleware's SRE agent, which detects, debugs, and auto-fixes production issues down to the exact line of code. Middleware serves 4,200+ engineers across retail, finance, gaming, and technology, ingesting telemetry at a scale where alert volume is measured in thousands per team per week across 100s of teams.

Challenge

Teams routinely receive over 1,000 alerts a week, and only 2-5% are worth acting on. That ratio is the single largest tax on an on-call engineer's attention, and it degrades everything downstream: real incidents get buried in alert storms, thresholds never get tuned, and MTTD stretches out while someone manually reconstructs whether a breach mattered.

Middleware set out to fix this with false positive detection in OpsAI - assessing each alert and attaching a verdict before the notification goes out. Building that on general-purpose frontier LLMs surfaced three hard constraints:

• Latency on the critical path. The verdict has to be attached before the notification is sent. Anything that doesn't return in time degrades to "Not evaluated", the alert still ships, but the feature silently stops working exactly when volume is highest. Frontier inference latency made the timing budget unworkable at scale.
• Per-alert economics that forced rationing. At frontier token pricing, assessing every Warning and Critical trigger across Metrics, Host, Logs, APM, RUM, LLM, Error Tracking, Anomaly, and Forecast rules is cost-prohibitive.
• Consistency on a narrow, repeated judgment. The decision is not open-ended generation. It weighs five specific signals — breach size, surrounding data over a 3x window, alert history across 24h/7d/30d, rule setup, and anomaly/forecast deviation — and returns one of four labels. Generative models were inconsistent on a task that needed a stable, scoreable verdict.

Solution - Decision LM Judge

Middleware leveraged Emissary’s Decision Language Models to develop an LM-judge purpose-built for alert triage. Using just a few samples of synthetic and internal data, including thumbs-up/thumbs-down feedback engineers give on live verdicts - passed through Emissary’s novel classification-conditioned generation training, they arrived at an 8B parameter model that is fast, trust-worthy and easy to improve over time.

The architectural shape is what makes the economics work. Classification-conditioned generation enables each request to return a decision <100ms, and generate reasons in <0.5secs that clearly explain the decision. No long context, no chain-of-thought padding, no retries.

Emissary handled the full ML infrastructure across training and inference - base model sweeps, model architecture modification, auto-scaling serverless inference, letting Middleware focus on what actually differentiates the product: the five-signal feature set, the calibration policy, and the feedback loop that scopes every correction to a single rule.

Impact

• Proactive assessments, no click to play. At <$0.10/M tokens, typical request costs ~$0.00006, enabling OpsAI to assess 500K+ supported Warning and Critical triggers proactively every month.
• In the alert, not after. Sub-second turnaround means the verdict is attached before the notification is dispatched, so "OpsAI detected this issue as Likely Noise" is sitting in the Slack message, the Teams card, the email, and the PagerDuty incident, the moment it arrives. No dashboard, no second login, no context switch.
• Judgements you can trust, and keep getting better: With a >85% true positive rate, improving with every set of feedback signals, the assessment is one you can rely on.

What's next?

Middleware is on a mission to making proactive SREs not just possible, but economically obvious. They are integrating Emissary's model routing into OpsAI to push beyond triage into proactive full-depth root-cause analysis on every issue, made possible by each step of the agent to the appropriate model and reserving frontier inference for the reasoning that actually requires it.