> For the complete documentation index, see [llms.txt](https://docs.stackai.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.stackai.com/technical-reference/evaluations-and-observability/evaluation-failures-and-result-comparison.md).

# Evaluation Failures and Result Comparison

An evaluation can fail while generating an answer or while scoring it. Separate these stages when comparing project changes.

{% embed url="<https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FSQ1Sgy933krTyK0S4JpP%2Fuploads%2FbXmKuo9wkUBUzHpTjgEu%2FG083.mp4?alt=media&token=81c67471-e568-4d60-b6fc-18de09e33814>" %}

## Before you begin

Access to the evaluation run, row results, and evaluator configuration.

## Key concepts

| Term            | Definition                                                                                                                                                                   |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Experiment      | An evaluation run with a selected dataset, project version, and evaluators. Compare equivalent configurations to attribute differences meaningfully.                         |
| Generation      | Execution that produces the output to evaluate. It can fail independently of the scoring stage.                                                                              |
| Scoring         | Applying an evaluator to a generated output and its required context. Missing scores and failed scoring are not passing results.                                             |
| Partial result  | Output available before all requested work completes successfully. Do not report the entire task as successful unless the operation’s contract supports that interpretation. |
| Project version | A specific saved configuration of a project. Select it explicitly when a result must be reproducible.                                                                        |

## Compare evaluation results

1. Inspect which rows produced outputs and which failed.
2. Review scoring errors separately from generation errors.
3. Compare runs that use the same dataset, criteria, and intended version selection.

![Keep the dataset, evaluators, and workflow version consistent when comparing results.](https://content.gitbook.com/content/SQ1Sgy933krTyK0S4JpP/blobs/O0OpQ1yparQY7beu3dBt/assets/g083-g083-01-setup.png)

Keep the dataset, evaluators, and workflow version consistent when comparing results.

The comparison distinguishes quality changes from execution or scoring failures.

## Behavior and limitations

A missing score is not a passing score. Changes to the dataset or evaluator can make aggregate comparisons misleading. Rescoring can add another result rather than replacing history. Inspect individual rows when an average hides a failure that matters to users.

## Related resources

[Run and inspect an experiment](https://docs.stackai.com/guides-and-quickstarts/evaluations-and-observability/run-and-inspect-an-experiment.md)

[Evaluation Datasets and Input Mapping](https://docs.stackai.com/technical-reference/evaluations-and-observability/evaluation-datasets-and-input-mapping.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.stackai.com/technical-reference/evaluations-and-observability/evaluation-failures-and-result-comparison.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
