Tests

Tests are individual test entries within an evaluation file. In normal authored eval YAML, prompts live in top-level prompts, case data lives in tests[].vars, and graders live in assert.

Basic Structure

prompts:
  - "{{ question }}"

tests:
  - id: addition
    vars:
      question: What is 15 + 27?
      expected_output: "42"
    assert:
      - type: llm-rubric
        value: "Matches the reference answer: {{ expected_output }}"

Fields

Field	Required	Description
id	Yes	Unique identifier for the test
vars	Yes when the prompt needs row data	Prompt-template variables for this row
criteria	No	Optional shared grader guidance for the case
vars.expected_output	No	Conventional reference-answer var for graders to consume explicitly
assert	Yes	Per-test graders; plain strings become llm-rubric rubric checks
execution	No	Per-case grader/default overrides such as skip_defaults; provider selection belongs in top-level providers or CLI --provider
environment	No	Per-case environment recipe (overrides suite-level)
metadata	No	Arbitrary key-value pairs passed to graders, setup commands, and lifecycle extensions

Prompt Data

In normal authored eval YAML, put prompt text in top-level prompts and per-case data in tests[].vars. The simplest form uses one prompt template and one string var:

prompts:
  - "{{ question }}"

tests:
  - id: addition
    vars:
      question: What is 15 + 27?

Structured object data can stay in vars for script graders and batch runners:

prompts:
  - "{{ input }}"

tests:
  - id: classify-ticket
    vars:
      input:
        request:
          type: classify_ticket
        ticket:
          title: Login button is broken

Top-level role is reserved for message objects. If your structured payload needs its own role field, nest it under another key.

For multi-turn or system messages, put the message array in prompts:

prompts:
  - - role: system
      content: You are a helpful math tutor.
    - role: user
      content: "What is {{ a }} + {{ b }}?"

tests:
  - id: addition
    vars:
      a: 15
      b: 27

Do not author tests[].input or top-level input in normal eval YAML. Those legacy direct-input fields are rejected; external raw-case imports are the only compatibility path for old input rows.

Criteria

criteria is optional case-level guidance for graders and prompt templates. Use it when the case needs shared evaluation context that several graders should see. If plain assertion strings already fully define the grading contract, omit criteria to avoid duplicating the same rubric in two places.

Do not confuse case-level criteria with a structured llm-rubric criterion’s outcome field. criteria describes the case-level grading context; outcome names one specific rubric item inside a llm-rubric criteria array.

Expected Output

Optional reference responses belong in tests[].vars.expected_output or default_test.vars.expected_output. Write the value as gold/reference data the target could have produced, not as a rubric or “the agent should…” criteria list. expected_output is passive by default: it is just a var. It does not choose a grader by itself. Add explicit assertion strings, llm-rubric, script, field-accuracy, or another reference-aware grader when you want the reference data evaluated.

A low-friction Promptfoo-compatible semantic check puts the reference in vars and consumes it from an explicit llm-rubric assertion:

prompts:
  - "{{ question }}"

tests:
  - id: addition
    vars:
      question: What is 15 + 27?
      expected_output: "42"
    assert:
      - type: llm-rubric
        value: "Matches the reference answer: {{ expected_output }}"

For structured expected output, keep the object under vars.expected_output and use a grader that knows how to compare it:

vars:
  expected_output:
    status: approved
    confidence: 0.95

Per-Case Execution Overrides

Override graders or local scoring settings for specific tests. Do not put provider selection in cases; use top-level providers, CLI --provider, separate eval suites, or tags/filters for provider-specific cases.

tests:
  - id: complex-case
    vars:
      input: Explain quicksort algorithm

    assert:
      - Provides a detailed explanation
      - name: depth_check
        type: llm-rubric
        prompt: ./graders/depth.md

Per-case assert graders are merged with root-level assert graders: test-specific graders run first, then root-level defaults are appended. To opt out of root-level defaults for a specific test, set execution.skip_defaults: true:

assert:
  - name: latency_check
    type: latency
    threshold: 5000

tests:
  - id: normal-case
    vars:
      input: What is 2+2?
    assert:
      - Returns the correct answer
    # Gets latency_check from root-level assertions

  - id: special-case
    vars:
      input: Handle this edge case
    execution:
      skip_defaults: true
    assert:
      - Handles the edge case
      - name: custom_eval
        type: llm-rubric
    # Does NOT get latency_check

Per-Case Environment Config

Override the suite-level environment recipe for individual tests. Test-level environment fields replace suite-level fields:

environment:
  type: host
  workdir: ./workspaces/default
  setup:
    command: ["bash", "-lc", "bun install && bun run build"]
    cwd: "."

tests:
  - id: case-1
    vars:
      input: Do something
    assert:
      - Completes the requested task
    environment:
      type: host
      workdir: ./workspaces/case-1
      setup:
        command: ["bash", "-lc", "bun install && bun run build && bun run setup:case-1"]
        cwd: "."

  - id: case-2
    vars:
      input: Do something else
    assert:
      - Completes the requested task
    # Inherits suite-level environment

See Environment Recipes for the full environment config reference.

Per-Case Metadata

Pass arbitrary key-value pairs to lifecycle commands via the metadata field. This is useful for benchmark datasets where each case needs repo info, commit hashes, or other context:

tests:
  - id: sympy-20590
    vars:
      input: Fix the diophantine equation bug in repo/.
    metadata:
      source_repo: sympy/sympy
      source_commit: "abc123def"
      test_patch: cases/sympy-20590/test.patch

The metadata field is included in the stdin JSON passed to lifecycle extensions as case_metadata. Use an environment recipe to materialize the checkout, then use a beforeEach extension when per-case metadata needs to drive patch application or fixture selection:

environment:
  type: host
  workdir: ./repo
  setup:
    command: ["bash", "./scripts/materialize-repo.sh", "./repo", "sympy/sympy", "abc123def"]
    cwd: "."

extensions:
  - file://scripts/apply-test-patch.py:beforeEach

Operational checkout state belongs in environment; matching metadata fields such as source_commit are informational only. For historical repo-state evals, pin the checkout in an environment setup recipe instead of only mentioning the SHA in prompt prose:

environment:
  type: host
  workdir: ./agentv
  setup:
    command:
      - bash
      - ./scripts/materialize-repo.sh
      - https://github.com/EntityProcess/agentv.git
      - 5e3c8f46d80fe66b1a75659e4fd94e38a7e09215
    cwd: "."
    timeout_ms: 120000

For benchmark task packs with source pins, patches, generated rows, and supporting files, see Benchmark Provenance.

Per-Test Assert

The assert field defines graders directly on a test. It supports both deterministic assertion types and LLM-based rubric evaluation.

Rubric Shorthand

For semantic or agent-behavior checks, prefer plain strings in assert. AgentV groups the strings into a rubric grader automatically:

tests:
  - id: bug-fix-review
    vars:
      input: Review this failing parser implementation.
    assert:
      - Identifies the root cause of the parser failure
      - Proposes a concrete code change
      - Adds or updates a regression test

Use this shape for qualitative requirements. It is less brittle than checking for exact substrings in an agent response. When these strings fully define the grading contract, do not add a criteria field that repeats the same rubric. Declare type: llm-rubric explicitly only when you need a custom prompt, custom grader target, or a deliberately separate grader panel.

Deterministic Assertions

Use deterministic assertions for exact machine-verifiable outputs. These graders run without an LLM call and produce binary (0 or 1) scores:

Type	Value	Description
contains	string	Pass if output includes the substring
contains-any	string[]	Pass if output includes ANY of the strings
contains-all	string[]	Pass if output includes ALL of the strings
icontains	string	Case-insensitive contains
icontains-any	string[]	Case-insensitive contains-any
icontains-all	string[]	Case-insensitive contains-all
starts-with	string	Pass if output starts with value (trimmed)
ends-with	string	Pass if output ends with value (trimmed)
regex	string	Pass if output matches regex (optional flags: "i")
is-json	—	Pass if output is valid JSON
equals	string	Pass if output exactly equals the value (trimmed)

Multi-word assertion types use kebab-case (contains-all, is-json, etc.).

tests:
  - id: json-api
    vars:
      input: Return the system status as JSON
    assert:
      - type: is-json
      - type: contains
        value: '"status"'

Array Assertions

Use contains-all or contains-any to check multiple values in a single assertion instead of repeating contains multiple times:

tests:
  - id: required-fields
    vars:
      input: "Confirm details: name is Alice, email is alice@example.com"
    assert:
      - type: contains-all
        value: ["Alice", "alice@example.com"]

  - id: greeting-variant
    vars:
      input: "Greet the user warmly."
    assert:
      - type: contains-any
        value: ["Hello", "Hi", "Hey", "Welcome", "Greetings"]

Assertion Modifiers

All deterministic assertions support these optional fields:

Field	Type	Description
negate	boolean	Invert the result (pass becomes fail, fail becomes pass)
weight	number	Relative weight when aggregating scores (default: 1)
required	boolean | number	Gate that must pass for overall test to pass. true uses 0.8 threshold; a number sets a custom threshold.
name	string	Custom name for the assertion (auto-generated if omitted)
flags	string	Regex flags for regex type (e.g., "i" for case-insensitive)

tests:
  - id: no-competitors
    vars:
      input: "Describe our product advantages."
    assert:
      - Response must not mention any competitor
      - type: contains-any
        value: ["CompetitorA", "CompetitorB", "CompetitorC"]
        negate: true

  - id: required-inputs
    vars:
      input: "Process customs entry for country BE."
    assert:
      - Agent asks for missing rule codes
      - name: asks-for-rule-codes
        type: icontains-any
        value: ["rule code", "rule codes"]
        required: true
      - name: mentions-format
        type: icontains-any
        value: ["true/false", "boolean", "expected value"]

Assertion graders auto-generate a name when one is not provided (e.g., contains-DENIED, is-json).

Advanced Rubric Assertions

Use type: llm-rubric with a value array when you need weights, required flags, or score ranges. Keep case-level criteria for shared grading context; each rubric item uses outcome for the specific desired behavior being scored:

tests:
  - id: denied-party
    vars:
      input: Screen "Acme Corp" against denied parties list
      expected_output: "DENIED"
    assert:
      - type: contains
        value: "DENIED"
        required: true
      - type: llm-rubric
        value:
          - id: accuracy
            outcome: Correctly identifies the denied party
            weight: 5.0
          - id: reasoning
            outcome: Provides clear reasoning for the decision
            weight: 3.0

Required Gates

Any grader in assert can be marked as required. When a required grader fails, the overall case status is fail regardless of the aggregate score.

Value	Behavior
required: true	Must score >= 0.8 (default threshold) to pass
required: true + min_score: 0.6	Must score >= 0.6 to pass (custom threshold between 0 and 1)

assert:
  - type: contains
    value: "DENIED"
    required: true          # must pass (>= 0.8)
  - type: llm-rubric
    required: true
    min_score: 0.6          # must score at least 0.6
    value:
      - id: quality
        outcome: Response is well-structured
        weight: 1.0

Required gates are evaluated after all graders run. If any required grader falls below its threshold, the case status is forced to fail.

Assert Merge Behavior

assert can be defined at both suite and test levels:

• Per-test assert graders run first.
• Suite-level assert graders are appended automatically.
• Set execution.skip_defaults: true on a test to skip suite-level defaults.

How Reference Fields and assert Interact

vars.expected_output is reference data, not a grader. It is provided to graders that know how to use it, but it does not create an LLM grading call by itself. A grader can use that data as an exact target, a semantic reference, a structured comparison object, or supporting context. Put the grading contract in assert.

Plain assertion strings are the default shape for semantic checks:

tests:
  - id: simple-eval
    vars:
      input: "Debug this function..."
    assert:
      - Assistant correctly explains the bug and proposes a fix

Suite-level transforms apply before graders see the target output. That matters when the agent output is a ContentFile block rather than plain text:

default_test:
  options:
    transform: file://scripts/transforms/xlsx-to-csv.ts

tests:
  - id: spreadsheet-eval
    vars:
      input: Generate the spreadsheet report
    assert:
      - Output includes the revenue rows

When assert is defined, only the declared graders run. No implicit grader is added because vars.expected_output exists. Declared graders such as plain rubric strings, explicit llm-rubric entries, or script graders receive the case context, including expected_output, as input automatically.

This means a case with vars.expected_output and only deterministic assertions evaluates only those deterministic assertions:

tests:
  - id: deterministic-reference
    vars:
      input: "What is 2 + 2?"
      expected_output: "4"      # reference data only
    assert:
      - type: contains          # only this grader runs
        value: "4"

For contract-style evals where assertion strings express every semantic check, keep those checks in assert:

tests:
  - id: verification-learning-capture
    vars:
      input: |
        Decide what durable repo change should be made after a PR closeout
        revealed reusable verification workflow lessons.
      expected_output: |
        The durable repo change is to update .agents/verification.md with the
        reusable verification workflow lessons.
    assert:
      - The answer recommends updating .agents/verification.md rather than leaving the learning only in PR comments or private evidence.
      - The answer avoids preserving one-off observations as durable guidance.

To combine deterministic checks with semantic checks, add both explicitly:

tests:
  - id: mixed-eval
    vars:
      input: "Debug this function..."
    assert:
      - Explains why the bug happens
      - type: contains
        value: "fix"

When you need a custom output conversion for only one grader, add transform directly to that grader:

tests:
  - id: mixed-eval
    vars:
      input: "Debug this function..."
    assert:
      - Response is helpful and mentions the fix
      - type: llm-rubric
        transform: file://scripts/transforms/xlsx-to-json.ts
      - type: contains
        value: "fix"

default_test.options.transform is inherited by every test. A tests[].options.transform entry overrides that inherited transform for one test, and assertion-level transform applies only to that assertion after the test/default transform has produced the candidate text for the rest of the graders.

Metadata

Pass additional context through the metadata field:

tests:
  - id: code-gen
    metadata:
      language: python
      difficulty: medium
    vars:
      input: Write a function to sort a list
    assert:
      - Generates valid Python

metadata is passed to lifecycle hooks as case_metadata, preserved in result records, and available to in-process custom assertions. AgentV does not interpret arbitrary metadata keys itself; use environment, extensions, prompts, vars.expected_output, and assert for operational behavior.