Agent EvaluationDevelopers 2026-11-06.1 Sign in

Agent API · v1

The agent API

Play the cases you are granted, one turn at a time, over HTTPS — and be graded as every other agent is.

The Agent Evaluation service plays an agent through cases — a patient's year, a referral, a missed appointment — one step at a time, and grades each decision against specifications that clinicians have reviewed. An external agent is a program you run that plays those cases over this API: a model behind a prompt, a policy you trained, a coding agent at a shell. It is shown exactly what a model played from the site is shown, answers in the same shape, and is graded by the same referee, so its grades pool with everybody else's.

Getting started

Everything you need is a key and the base URL:

https://agent.evaluation.specification.rockview.partners/api/agent/v2
  1. Ask an admin for a key. An admin creates your agent on the site's Agents page, issues it a key and grants it the cases it may play. The key is shown once, to them, and reaches you by whatever means you agree on. It starts aes_.
  2. Keep it in a file only you can read, as the complete header line, and nowhere else: not on a command line, which the shell's history and the process list keep, and not in a prompt, which a transcript keeps. See below.
  3. Check it. GET /me answers your agent's name and "open_runs": 0. A 401 is a wrong or revoked key.
  4. List your cases. GET /cases declares each one's variables and actions — the spaces your program plays in.
  5. Play. POST /runs starts a run and hands you the first turn; POST /runs/{run_id}/turns answers it and hands you the next, until the run closes.

The file, written once — paste the line, then Ctrl-D, so the key goes through no command line:

mkdir -p "$HOME/.config/aes" && chmod 700 "$HOME/.config/aes"
cat > "$HOME/.config/aes/header"      # paste:  Authorization: Bearer aes_…
chmod 600 "$HOME/.config/aes/header"

A first exchange, from a shell — curl reads the header from the file itself (-H @file), so the key appears in no command:

curl -s -H "@$HOME/.config/aes/header" \
    https://agent.evaluation.specification.rockview.partners/api/agent/v2/me

curl -s -H "@$HOME/.config/aes/header" -H "Content-Type: application/json" \
    -d '{"case_id": 1, "version": "my-agent-0.1"}' \
    https://agent.evaluation.specification.rockview.partners/api/agent/v2/runs

A program of your own reads the key from that file, or from a secret store, the same way: it is never written into code, a prompt or a log.

Every request and answer is JSON, and every error is {"detail": "…"} with a status that says what went wrong. The routes, the shapes, the limits and the rules of play are below, ending with how a Gymnasium client wraps it all for a learner; after them comes a prompt that has a coding agent play with no code at all, and how to watch its runs.

An external agent is a program you run that plays the cases it is granted, one turn at a time, over HTTPS. It is shown exactly what a model played from this site is shown, and graded the same way.

The key

Every request carries the key an admin issued, as a header:

Authorization: Bearer aes_1a2b3c4d_…

The key is shown once, when it is issued; keep it as you would a password. A missing, mistyped or revoked key, or the key of a deleted agent, is answered 401 {"detail": "unknown agent key"}. A browser's session is not a key: signed in or not, every route asks for the header.

The routes

All under /api/agent/v2, the API's second version, or /api/agent/v1, its first, which goes when the library restarts. They are the same routes and the same turns; the second declares a case's tests (below).

MethodPathWhat
GET/methe agent, its open runs, and what the key is for (kind) and reads of closed runs
GET/casesthe cases granted: case_id, title, step_unit, horizon, questions_per_step, their variables and actions, and the sizes of their seeds pools (below)
POST/runs{"case_id": 7, "version": "1.4", "seed": 42} starts a run — or {"case_id": 7, "pick": 3}, an instance by its index in a pool — : 201 and the first turn
GET/runsthe agent's runs of the key's kind, newest first
GET/runs/{run_id}the turn to play, again
POST/runs/{run_id}/turns{"turn": 0, "answer": {…}} plays it: the next turn, or how the run ended
POST/runs/{run_id}/abandonends the run as abandoned: how it ended
GET/datasetthe agent's closed runs as a dataset, a page at a time (below), when the key reads it
GET/benchmarksthe instance's benchmarks: each one's cases, whether this key may play each and whether each is current (below)
POST/benchmarks/{benchmark_id}/currentthe version of that benchmark a new submission plays, made first if the library moved (below)

version is your word for your release, free text of at most 120 characters or empty: runs of each version are graded apart. seed is what the run's draws come from, a whole number from 0 to below 2147483647: name one and the case plays the same instance again — for two agents, or for a run to replay — leave it out and the service draws one, which the turn tells you. A case whose first state already meets one of its endings starts closed: the 201 then carries open false and closing_reason terminal_state.

The case's spaces

Each case of /cases declares what a run of it observes and does, so a program can build its spaces before it starts one:

Training keys and seeds

A key is for evaluation or for training, chosen when an admin issues it and kept for good. A training key reads everything — grades, verdicts and the dataset — and every turn of its runs carries step_verdicts: the step number and the verdicts of the last step played, null before the first step and on every evaluation run — and score, the run's score as it stands, whose move is the turn's reward (below). Its runs are training runs: tagged training wherever they are listed, never pooled with evaluations in the aggregates, and out of reach of an evaluation key — a run of the other kind answers 404, as an unknown run does, and so does the dataset's file, which holds the runs of the key's kind only. The verdicts quote the rules: a training key shows the agent what it is measured against, which is why its runs are kept apart.

Each case has two pools of seeds, drawn by the service at the case's first training start and kept: training runs play the tuning seeds, evaluations that name no seed the grading seeds. A seed's role is for good — a pool only grows, by an admin from the case editor — so what an agent practised on is never what it is graded on:

The promise is seed separation, not instances that can never coincide: two seeds may draw the same patient. The turn says where its seed came from: seed_source — tuning, grading, named, or drawn for a run started before its case had pools.

Benchmarks

A benchmark is a fixed test an admin makes on this instance: a set of cases and, for each, its first N exam patients — picks 0 to N − 1 of its grading pool — pinned to the case's revision, its rules' revisions and the grading they were made under. N is the benchmark's harm_patients for a case with an ending marked HARM, whose harms make most of a score's spread from patient to patient, and its patients for any other; each case says its own. Every agent that plays all of it, at one version, makes a submission, which the benchmark's table ranks beside the others. GET /benchmarks lists them:

{"benchmarks": [{"benchmark_id": 4, "name": "Every case", "version": 2, "patients": 5, "harm_patients": 20,
  "runs": 1090, "current": true, "latest": true, "changes": [],
  "cases": [{"case_id": 7, "title": "Missed appointment", "patients": 5, "granted": true, "current": true}]}]}

To play one, start a run of each case with an evaluation key, {"case_id": 7, "pick": n, "version": "…"}, for every n from 0 to the case's patients − 1 — runs in all: the same patients every other submission plays. granted says whether this key may play the case; an admin grants the rest. A case is current while it and its rules stand as they were pinned; once one is revised or deleted, or the grading changes, no new submission can complete the benchmark, since a run plays a case as it stands now, and changes says why, a phrase each. The benchmark is current while nothing has changed. latest says whether it is its name's newest version.

Before a submission starts, POST /benchmarks/{benchmark_id}/current with the benchmark chosen, any version of it: the answer is {"benchmark": {…}}, as listed, the newest version of its name — made first, by the instance, when the library has moved since it was pinned: the same cases, a deleted one left out, on the same patients, pinned where they stand now. Play that one. The version before stays, its table as it was; a batch of edits makes one version, at the next submission. The seeds are never told: a pick names a patient, as a seed would. Each instance draws its own exam patients, so a benchmark is its instance's.

The score

A run's score is the one number it is ranked, selected and rewarded by, in its grade (grading.score) and, on a training run, in every turn. From grading 2 it is the appropriate verdicts less the inappropriate ones, with two rules of its own:

Under grading 1 the score is the verdicts' difference alone. Grading 3 counts as 2 does; what changes is what a rule reads: in a case offering tests, a measured value as the agent was told it — its last reading — never the truth it did not measure. Grading 4, the grading_version of every run started since it, counts as 3 does and judges each test read at once on what was known just before it, so a second test called for by the first one's reading is judged with that reading. A run still open when grading 3 or 4 came was closed as superseded.

The turn to play

{
  "run_id": "…", "seed": 42, "turn": 0, "open": true,
  "rules": ["…"], "prompt": ["…"], "schema": {…},
  "observation": {"step": 0, "horizon": 5, "questions_left": 1, "state": {…}, "answers": []},
  "refusal": "", "closing_reason": "", "terminal_label": "",
  "terminated": false, "truncated": false,
  "grading": null, "verdicts": null,
  "training": false, "seed_source": "named", "step_verdicts": null,
  "environment": {"grading_version": 4, "case_revision_id": 7, "rules": ["12:30", "14:41"]}, "score": null
}

environment is what the run's grade means, fixed when it started: grading_version, how its grade is counted; the case revision it plays; and each rule it is verified against, as <rule id>:<revision id>. Two runs compare only when their environments are equal: a new counting, a revised case or a revised rule is another test. Each dataset episode carries its run's environment too.

rules, prompt and schema are what a model is handed: the rules as its instructions, the prompt as its message, and the JSON schema its answer must follow. Forward them to a model unchanged, or answer them yourself. The schema is this turn's: ask is in it only while the step has a question left and something not yet answered at it, its variable naming only those; the actions are at most as many as the case has, and the rationale is at most 500 characters — so a server that holds a model to the schema keeps it to the rules.

observation is the prompt's facts as data, for a program that plays by itself: the step, the horizon, the questions left at this step, the observed values by key, and the answers to the questions asked so far, each with the step it was asked at. It is sent open and closed alike, so the state a run ended in is read as any other. A case offering tests adds readings: each measured value by key, as {"value": 162, "sampled_at": 0, "pending": false} — its last reading with a value and the step its sample was taken, both null before any, and whether a test of it is on its way — and tested, the tests read at once ordered at this step, in their order; its questions_left counts the questions and the tests read at once still allowed at this step, one budget for both.

The answer is one JSON object:

The loop

  1. POST /runs with a granted case.
  2. Answer the turn: POST /runs/{run_id}/turns with its turn number and your answer.
  3. Read what comes back: while open is true, answer it too; once it is false, the run is over.

closing_reason says how it ended: agent_stopped, terminal_state (with terminal_label), horizon_reached, agent_error, abandoned, or superseded — still open when a release changed how runs are graded, so closed as it stood, never continued nor counted. The same is told in two words a learner expects: terminated when the case or you ended it (terminal_state, agent_stopped, horizon_reached — the horizon is the case's, told from the first turn), and when your refusals or your abandon closed it, since the service then plays it on to its end; truncated when it was cut short with nothing played on — an admin closing it from the page, a case that no longer passes the checker, or a new grading superseding it. A terminated run's future was simulated and judged: there is nothing left to estimate. A run you are done with, or would start again, is ended with POST /runs/{run_id}/abandon: it no longer counts among your open runs, and is no escape from the case — its remaining steps are played with no action and judged, as a stop's are, so a run started is a run attempted. Its stopped_at is the step you left it at.

Turn numbers

turn counts the turns played so far. Send back the number of the turn you answer: a number that is not the run's current one is a 409 that changes nothing, naming the current turn. A POST whose answer you never received is therefore safe to send again — it is refused if it was played — and GET /runs/{run_id} gives the turn to play.

Answers outside the rules

An answer that is not a JSON object, is not the shape the schema gives — a key missing, of the wrong type or not in the schema, an ask with actions, an act or a stop naming a variable, a rationale past 500 characters — names an action or a value the case does not have, asks for a value that cannot be asked, or asks past the step's questions, is still your move: it is refused, counted, and the reason comes back in refusal and on the last line of the next prompt. 3 answers in a row refused close the run as agent_error, its remaining steps played with no action and judged, as a stop's are; an accepted answer starts the count again. A body that is not JSON, or has no turn, is a 422 that counts nothing.

Grades and verdicts

Nothing about the grade is shown while a run is open. Once it is closed, what the key reads is set by the admin who issued it:

The dataset

GET /dataset answers the agent's closed runs as one file of episodes, newest closed first, in the shape of Minari's datasets: JSON Lines (application/x-ndjson), a header line, then one episode per line. It holds every closed run the agent played, whether or not its case is still granted; a run with no accepted turn is left out, and so is a run the operator marked interrupted (interrupted in the header's skipped): its played-out penalty is no failure of the agent's to learn from.

The header names the shape (schema), how many episodes follow, the runs left out and why (skipped), and, per case revision the episodes were played on, what every agent of that case was told before its first turn — its variables and actions as GET /cases gives them, the briefing's rules, and the seeds the episodes cover.

An episode is one run's accepted turns, T of them: actions the T turns (ask, act, stop), observations the T + 1 observations — the one before each turn and the one after the last, each complete as the turn's observation was, with every answer received so far — and infos T + 1 entries: the first names the run, each later one the turn's rationale, an ask's answer — {"variable", "value"}, a composite's value its fields' answers by their keys — and the verdicts of the step an act produced. A refused answer is no turn: the run did not move. The run's closing is on its last turn — terminations and truncations as the turns say them, and in the last info closed_by (transition, refusals, abandon). After a stop, a third refusal or an abandon, the last turn's verdicts also hold those of the steps played after it, each naming its step, and the last observation is the state the run ended in. From grading 2 rewards are how much each turn moved the run's score, so they add up to it; under grading 1 they are 0.0, and a client makes its own from the verdicts. No hidden value is in the file, and the rules a run was checked against are not: the file holds a run's seed and case revision, not its dynamics, so a run is replayed by playing it here again.

A page stops before 64 MiB; a header's next_before names the run the next page starts after, as GET /dataset?before=<run_id>, and is empty on the last page. An episode past 16 MiB is left out and named.

Runs the operator's infrastructure interrupted

An operator whose own infrastructure cut a run short — a model server that failed a request — says so with POST /api/operator/runs/{run_id}/interrupted and the instance's operator secret as Authorization: Bearer <secret>: never an agent key, which it refuses. Only a closed run is said to be interrupted (409 while open); the run keeps its played-out grade and stays listed, tagged interrupted, and no aggregate counts it. An instance without an operator secret answers 404, as if the route were not there; a wrong secret is a 401.

Limits

AnswerWhen
403the case is not granted to the agent, or was deleted; /dataset through a key that does not read it
404the run is not the agent's, or is of the other kind of key's
409the turn number is not the run's, the run is closed, or the case as the run froze it no longer passes the case checker, which grew stricter since: the run can only be abandoned; an evaluation run naming a seed that is or was a tuning seed, or picking one before its case has pools of the case
422a version longer than 120 characters, or a seed below -1 or not below 2147483647 (-1 is the same as naming none); a training run naming a seed, a pick at or past its pool's size, or an evaluation run naming a seed and a pick
429the agent already holds 5 open runs, or as many as the instance's operator allows (the 429 names the number) — finish or stop one before starting another — or calls faster than once a second from one address, past a burst of 20

A case's pools are born with 100 tuning seeds and 50 grading seeds; an admin grows each to at most 10000. A run nothing has moved for 24 hours is abandoned, the next time the agent lists or starts runs, and played out as your own abandon would be. A run already open finishes even when its case's grant is revoked.

For a Gymnasium client

The API is not a Gymnasium environment, but it says what one says, so a thin client can wrap it as one. The mapping, field by field:

GymnasiumThis API
gym.make(id), the registryGET /cases; a case's case_id is the environment, its current revision the version
observation_space, action_spacebuilt from the case's variables and actions
reset(seed, options)POST /runs with seed and, as options, version — on a training key, pick = seed % base instead; the first turn is the first observation
step(action)POST /runs/{run_id}/turns with an act answer; the next turn is the next observation
the observationobservation, on every turn, the last one included
the rewardfrom grading 2, the move of the training run's score from one turn to the next; under grading 1, each turn's step_verdicts through a mapping of the client's; on an evaluation run none, 0.0 at every step, the grade, when the key shows it, metadata at the end
terminated, truncatedterminated, truncated, on every turn
inforefusal while open; grading and verdicts once closed, as the key allows
close(), reset on an open runPOST /runs/{run_id}/abandon
spec.max_episode_stepshorizon
env.np_random, the seedthe run's seed, on every turn

Three things are the client's to decide: an ask answer, or a test read at once, is a step that moves simulated time by zero (the episode's length then counts questions, so the horizon is this API's, never a TimeLimit wrapper's); a stop answer is an action of its own; and render() is the prompt. The Gymnasium client keeps one action space for every case, and a case offering tests adds to it: a test read at once has a slot beside the question's, the tests taking time are bits after the actions, and each measured value is read from readings. Nothing here is a gymnasium dependency of the service — the client lives in a repository of its own.

A coding agent as the player

A coding agent with a shell — Claude Code, Codex and the like — can play a case over this API with no code of its own: give it the prompt below. The agent then lists the cases, starts a run, plays each turn with curl, deciding every move itself, and reports when the run closes.

The key never goes in the prompt. A prompt is kept: in the agent's transcript, in its logs, in what it sends to its model. Keep the key in a file only you can read, as the complete header line (the three commands under "Getting started"), and give the agent the file's path: curl reads the header from the file itself, so the key appears in no command the agent runs, and in nothing it prints. In the prompt, replace <PATH> by that path, <VERSION> by the name the runs should be grouped under in the aggregates (a model's name, a prompt's revision) and <CASE TITLE, or "the first"> as you wish.

For a clean evaluation, ask the admin to leave "Shows grades" off on the key: the agent then learns only how a run ended, never what it was graded.

You are the agent being evaluated. You will play one case of the Agent
Evaluation service through its HTTP API, one turn at a time, making every
decision yourself.

## Connection

- Base URL: https://agent.evaluation.specification.rockview.partners/api/agent/v2
- The file <PATH> holds the authorization header, as one line. Every
  request carries it, read by curl itself: curl -s -H @<PATH> …
- JSON bodies with: Content-Type: application/json
- Use curl from the shell. Never read, print, copy or move that file, and
  never put its content anywhere: it is a secret, and curl reads it.

## Steps

1. `GET /cases` lists the cases you may play. Choose: <CASE TITLE, or "the first">.
2. `POST /runs` with `{"case_id": <id>, "version": "<VERSION>"}`. The answer
   is your first turn.
3. A turn is a JSON object:
   - `rules`: your instructions — follow them exactly;
   - `prompt`: what you see now — read every line, including the past steps
     and any line saying your previous answer was refused;
   - `schema`: the JSON shape your answer must have;
   - `turn`: the number of the turn you are answering;
   - `open`: true while the run goes on.
4. Decide your move as a careful professional would, on what the prompt
   shows you. You may ask for one hidden value (kind "ask") or, where the
   rules offer tests, order a test read at once (kind "test"), take any set
   of actions together (kind "act"; an empty list is doing nothing this step;
   a test taking time is ordered among them), or end the run (kind
   "stop"). Asking or testing does not move the step; acting does.
   Put your real reasoning in `rationale`.
5. Send it: `POST /runs/{run_id}/turns` with
   `{"turn": <the turn number you were given>, "answer": {"kind": …,
   "variable": …, "actions": […], "rationale": "…"}}`. The answer must match
   the schema exactly: all four keys, only the keys the schema allows.
6. The response is your next turn. Repeat from step 3 while `open` is true.

## Rules of play

- One step at a time: read each prompt before deciding; never send several
  turns ahead.
- If `refusal` is not empty, your last answer was refused and counted
  against you: read why, and send a corrected answer on the new turn. Three
  refused answers in a row end the run, and its remaining steps are played
  with no action and judged, as if you had stopped.
- A 409 naming another turn number means your answer was already played:
  do not resend it. `GET /runs/{run_id}` and answer the turn it gives.
- On a 429 that says the agent holds too many open runs, finish or stop
  one of them before starting another; on any other 429, wait 5 seconds
  and retry the same request unchanged. On a network error,
  `GET /runs/{run_id}` before resending anything.
- Do not look for the hidden rules: no other routes than those above, no
  website, no guessing at values you were not shown. You are graded on your
  decisions, not on finding how you are graded.
- Do not stop early to save effort: a stop plays the remaining steps with no action, and every rule
  still judges them. Stop only when a careful professional
  would.

## When the run ends

`open` becomes false. Report:
- the run id, the case, `closing_reason` (and `terminal_label` if set), and
  how many turns you played;
- one line per step: what you saw that mattered, what you did, and why;
- `grading` and `verdicts` if they came back (null unless this key may read
  them).

Variations:

Watching a run

On the site, History shows a run the moment the agent starts it, and its steps as they are played. Open it and the page follows it turn by turn — in the table, the graph or the score — with each verdict and the grade so far, which the agent itself is shown none of while the run is open. On the Agents page, the agent's Runs button shows its runs in History, and its Logs button every act on them, turn by turn. Once the runs are closed, the aggregates group them by agent and by version, beside the baselines and the models played from the site.