> For the complete documentation index, see [llms.txt](https://docs.stackai.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.stackai.com/guides-and-quickstarts/evaluations-and-observability/production-quality-monitors.md).

# Production quality monitors

Evaluation monitors score a sample of production responses over time. Use them to detect changes in quality and investigate requests that receive low scores.

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FSQ1Sgy933krTyK0S4JpP%2Fuploads%2F42HOZdsWCBCkfXKbx1Yu%2FG084.mp4?alt=media&token=6afc1a34-e022-4383-a074-cdff140198d7>" %}

## Before you start

Have a published project with run logging enabled, completed production activity, access to Evals, and a tested reference-free [evaluator](https://docs.stackai.com/guides-and-quickstarts/evaluations-and-observability/configure-an-evaluator.md). Agree who will investigate low scores and who should receive alerts. Monitoring provides evidence after a response; it does not block a poor answer from reaching a user.

Choose an evaluator suitable for the information available in production requests.

## Choose sampling and alert settings

A monitor samples completed production runs. Draft tests, evaluation experiments, failed executions, and runs with logging disabled are not monitored. Keep run logging enabled for the project. Choose a reference-free evaluator: production responses have no expected-answer field to compare against. Start with a tested LLM judge such as **Meets criteria** from [Configure an evaluator](https://docs.stackai.com/guides-and-quickstarts/evaluations-and-observability/configure-an-evaluator.md).

Sampling rate is the chance that an eligible run is selected. At 10%, roughly 10 out of 100 eligible runs are sampled over time; it is not a promise to select every tenth run. The daily maximum caps the number sampled per monitor per UTC day. Raising the cap does not create traffic or fill gaps retrospectively. Choose a sample rate and cap that can provide enough scored runs within your alert lookback.

For a pass/fail evaluator, Pass has score 1 and Fail has score 0. For numeric evaluators, scores at or above **Alert threshold** count as passes; lower scores count as failures. The alert checks the resulting pass rate against that threshold. Use pass/fail scoring first if you need a straightforward interpretation.

| Alert setting           | Meaning                                                                                   | Example starting value |
| ----------------------- | ----------------------------------------------------------------------------------------- | ---------------------- |
| Alert threshold         | Pass-rate floor used for degradation detection; also the score cutoff for counting passes | 70%                    |
| Warning threshold       | Shows Warning below this pass rate; a warning alone does not send an alert                | 80%                    |
| Window size (runs)      | Maximum number of recent scored runs considered in a check                                | 100                    |
| Lookback (hours)        | How far back a check looks for those runs                                                 | 24                     |
| Minimum samples         | Number of scored runs required to make a new assessment                                   | 30                     |
| Minimum drop (points)   | Minimum fall from the recent baseline, in percentage points                               | 5                      |
| Recovery threshold      | Pass rate required to recover from an alert                                               | 70%                    |
| Re-notify every (hours) | Reminder interval while qualifying degradation continues                                  | 6                      |

These values illustrate a policy; agree on your team's quality requirement before applying it. For example, use a tested pass/fail **Meets criteria** evaluator, 10% sampling, a cap of 100 runs per day, and the settings above. At fewer than about 300 eligible runs per day, 10% sampling may not produce the required 30 scores. Increase sampling or extend the lookback if appropriate; do not lower the quality bar just to make an alert appear.

Checks run hourly. A low pass rate must have high statistical confidence and a sufficient drop from the established baseline, and must persist across qualifying checks before an alert fires. One bad answer or the first batch from a new monitor is not an alert-delivery test. Recovery likewise requires two consecutive qualifying healthy checks. If there are too few samples, the monitor keeps its previous severity rather than proving that quality is healthy; check the newest score time too.

## Configure sampling

Open Evals → Monitors → New monitor and choose a suitable evaluator.

![New monitor evaluator and sampling settings](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/XhdU1eFlgSXzTRfZsTOw/assets/g084-g084-01-configure.png)

Set sampling, daily limits, and notification options for the production activity you intend to monitor.

## Generate and inspect production activity

Submit a normal request through the published interface and inspect its result.

![Published-interface request and response](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/fipueExlvozBOnpPdx5C/assets/g084-g084-02-alpha.png)

Sampling means that not every production run will necessarily receive a score.

Review a request with a different output against the same quality rule.

![Another response checked against the quality rule](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/aWcL6iHcyzbmMiM40ndO/assets/g084-g084-03-beta.png)

Successful execution and compliance with a quality rule are separate questions.

## Set alert conditions

Open the monitor's Alerts settings and review thresholds, windows, lookback, and minimum samples.

![Monitor alert settings](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/3PMTtoOseowS1nDo4aCl/assets/g084-g084-04-policy.png)

An alert depends on these conditions as well as the individual scores.

## Investigate evaluation results

Wait for sampled runs to finish scoring, then inspect the monitor table.

![Sampled-run scoring table](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/CWJMKIeAEFcRTWQ79bd3/assets/g084-g084-05-scores.png)

Interpret the score against the number and type of runs actually sampled.

Open a scored run and compare its execution status with its quality scores.

![Execution status and quality scores](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/G1qkQiwFxgZZ8cdaHlrj/assets/g084-g084-06-trace.png)

A successful run can still receive a failing quality score.

Expand the sampled output and compare it with the evaluator's rule.

![Expanded sampled output](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/uG7GX4TAFMVrsWB9oofZ/assets/g084-g084-07-output.png)

Use the content to explain the score rather than treating a quality failure as an execution exception.

## Disable a monitor when needed

Disable the monitor when continued scoring is no longer required.

![Monitor disable control and state](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/QzphCFUlMd9AFjwwWT8k/assets/g084-g084-08-disable.png)

Confirm its Disabled state while retaining historical results for review.

## Confirm who receives alerts

Enable the monitor's notifications. The default recipients are the project owner and additional builders, with organization administrators as the fallback. Review the evaluation-degradation notification's organization recipient rules and each recipient's delivery channels in [Notification preferences and inbox](https://docs.stackai.com/guides-and-quickstarts/personal-settings-and-support/notification-preferences-and-inbox.md); a monitor switch alone does not verify delivery.

After the monitor has a baseline, verify that a real qualifying degradation produces the expected alert and reaches the intended recipient. Use a dedicated test project with controlled traffic if you need to exercise this before rollout. Record the monitor, evaluator, score times, pass rate, alert settings, recipient, and received notification. Do not deliberately worsen an active user-facing workflow to test delivery.

If scores are missing, check production status, logging, sampling, daily cap, and scoring progress. If scores exist but no notification arrives, check sample count, baseline/drop conditions, hourly checks, monitor notifications, recipient rules, and personal channels. Keep inspecting individual failures even when the alert has not fired. Send unresolved delivery problems with these details through [Support and deployment diagnostics](https://docs.stackai.com/guides-and-quickstarts/personal-settings-and-support/support-and-deployment-diagnostics.md).

## Important behavior

Sampling controls which requests are evaluated. Monitor results do not necessarily include every production request.

Alert conditions depend on the threshold, lookback period, and minimum sample count. A small sample may not trigger an alert.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.stackai.com/guides-and-quickstarts/evaluations-and-observability/production-quality-monitors.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
