For the complete documentation index, see llms.txt. This page is also available as Markdown.

POST /v2/evals/evaluators

Create Evaluator

post

Create a validated evaluator and its first immutable configuration version.

How to use

Choose a built-in template or validate a judge configuration through preview first. Supply evaluator_type, config, and the project node contract. This creates the first immutable configuration version.

Authentication and permissions

Required permissions: evals.create. Resource and owner access checks also apply.

Prepare request.json

Save the example body below as request.json for the curl sample. Replace example IDs, node keys, and dynamic configuration with values for your organization; these are illustrative, not pre-existing resources.

{
  "project_id": "YOUR_PROJECT_ID",
  "name": "Correct request category",
  "evaluator_type": "builtin_rule",
  "config": {
    "rule": "exact_match",
    "target_output_node_id": "out-0"
  },
  "node_contract": {
    "input_kind": "legacy_input_nodes",
    "input_keys": [
      "in-0"
    ],
    "output_node_ids": [
      "out-0"
    ]
  }
}
Authorizations
AuthorizationstringRequired

Personal API key (sk_...) or supported user access JWT. See Authentication and authorization for endpoint restrictions and scopes. Use exactly one Authorization header.

Body
project_idstring · min: 1Required
namestring · min: 1 · max: 200Required
descriptionstring · nullableOptional
evaluator_typestring · enumRequired

How an evaluator scores: LLM judge or built-in rule.

Possible values:
judge_providerstring · enum · nullableOptional

LLM provider identifiers.

Possible values:
judge_model_idstring · nullableOptional
Responses
201

Successful Response

application/json
post/v2/evals/evaluators
curl
curl --fail-with-body -sS \
  "$STACKAI_BASE_URL/v2/evals/evaluators" \
  -H "Authorization: Bearer $STACKAI_API_KEY" \
  -H 'Content-Type: application/json' \
  --data-binary @request.json
{
  "response_code": "text",
  "meta": {
    "ANY_ADDITIONAL_PROPERTY": "anything"
  },
  "data": {
    "evaluator_id": "123e4567-e89b-12d3-a456-426614174000",
    "project_id": "text",
    "name": "text",
    "description": "text",
    "evaluator_type": "llm_judge",
    "version_number": 1,
    "node_contract": {
      "input_kind": "start_node",
      "input_keys": [
        "text"
      ],
      "output_node_ids": [
        "text"
      ]
    },
    "config": {
      "ANY_ADDITIONAL_PROPERTY": "anything"
    },
    "score_kind": "numeric",
    "uses_input": true,
    "uses_expected_output": true,
    "judge_provider": "text",
    "judge_model_id": "text",
    "created_by": "123e4567-e89b-12d3-a456-426614174000",
    "updated_by": "123e4567-e89b-12d3-a456-426614174000",
    "created_at": "2026-01-01T00:00:00.000Z",
    "updated_at": "2026-01-01T00:00:00.000Z"
  }
}

Last updated

Was this helpful?