POST /v2/evals/evaluators
Create a validated evaluator and its first immutable configuration version.
How to use
Choose a built-in template or validate a judge configuration through preview first. Supply evaluator_type, config, and the project node contract. This creates the first immutable configuration version.
Authentication and permissions
Required permissions: evals.create. Resource and owner access checks also apply.
Prepare request.json
Save the example body below as request.json for the curl sample. Replace example IDs, node keys, and dynamic configuration with values for your organization; these are illustrative, not pre-existing resources.
{
"project_id": "YOUR_PROJECT_ID",
"name": "Correct request category",
"evaluator_type": "builtin_rule",
"config": {
"rule": "exact_match",
"target_output_node_id": "out-0"
},
"node_contract": {
"input_kind": "legacy_input_nodes",
"input_keys": [
"in-0"
],
"output_node_ids": [
"out-0"
]
}
}Personal API key (sk_...) or supported user access JWT. See Authentication and authorization for endpoint restrictions and scopes. Use exactly one Authorization header.
How an evaluator scores: LLM judge or built-in rule.
LLM provider identifiers.
Successful Response
Invalid request or pagination cursor
Missing or invalid credentials
Denied permission, project folder access, or API-key scope (meta.required_scope)
Resource not found in the caller's organization
Name conflict, concurrent operation, or dataset state prevents the operation
Invalid evaluation contract, row shape, evaluator configuration, or filter combination
Client error returned in the public API envelope
curl --fail-with-body -sS \
"$STACKAI_BASE_URL/v2/evals/evaluators" \
-H "Authorization: Bearer $STACKAI_API_KEY" \
-H 'Content-Type: application/json' \
--data-binary @request.json{
"response_code": "text",
"meta": {
"ANY_ADDITIONAL_PROPERTY": "anything"
},
"data": {
"evaluator_id": "123e4567-e89b-12d3-a456-426614174000",
"project_id": "text",
"name": "text",
"description": "text",
"evaluator_type": "llm_judge",
"version_number": 1,
"node_contract": {
"input_kind": "start_node",
"input_keys": [
"text"
],
"output_node_ids": [
"text"
]
},
"config": {
"ANY_ADDITIONAL_PROPERTY": "anything"
},
"score_kind": "numeric",
"uses_input": true,
"uses_expected_output": true,
"judge_provider": "text",
"judge_model_id": "text",
"created_by": "123e4567-e89b-12d3-a456-426614174000",
"updated_by": "123e4567-e89b-12d3-a456-426614174000",
"created_at": "2026-01-01T00:00:00.000Z",
"updated_at": "2026-01-01T00:00:00.000Z"
}
}Last updated
Was this helpful?

