What the agent sees
The running score: each step adds its appropriate verdicts and takes away its inappropriate ones; n/a counts for nothing.
- rise: appropriate verdicts at the step
- drop: inappropriate verdicts
- score after the step
Select a verification cell or dot to read its explanation.
Grading
Turns
Finished
| Started | Case | Played by | Steps | Ended | % verifiable | % appropriate |
|---|
By case and agent
Finished runs only. The mean score is the mean of the runs' scores — the number grading 2 ranks runs by, a harmful ending first — and harmful counts the runs that reached one; the shares are pooled: appropriate cells over verifiable cells across the runs. A grade is comparable only with others of the same case by the same agent. A version is the case and the specifications as a run was graded against them: an edit to the case, a key or a specification in scope starts another — open a group to see its versions apart.
| Case | Played by | Runs | Versions | Mean score | Harmful | % verifiable | % appropriate | % inappropriate | Conflicts |
|---|
Benchmarks
A benchmark is a fixed test: its cases, each on its first exam patients, pinned to the revisions and the grading it was made on; once one of them changes, the next submission makes its next version, and the one before stays as it was. Every agent and version that plays it is a submission, a column of the table; a cell is the case's patients played, the mean score and the runs that ended in harm, and opens those runs. Summary reads one row a submission over the cases shown.
Logs
What the admins did to the external agents — keys, grants, names — and what was done on the agents' runs, by the agent, an admin or the service. Times in UTC.
| When (UTC) | Agent | What | By |
|---|
Distributions
Values drawn as aggregated data says they fall, DRAW(<code>, <strata…>): percentiles, a mean and a standard deviation, shares by bin or choice, or a risk over a period. Each number keeps its source; a piece a case draws cannot be deleted, and a run plays the revision it started with.
Functions
Functions written in the rule language, NAME(<values…>), which every case and rule may call: each reads its parameters only, may call another and may draw from the distributions. A function that draws is called only where a draw may be; one a case, a rule or another function calls cannot be deleted, and a run plays the revision it started with.
Value sets
Named lists of clinical codes a rule names as one thing — "any ACE inhibitor", "a stroke" — each published in VSAC (by its OID and release) or built here, saying how. The codes its source gave are kept apart from those added or removed here, each change with its reason, so the source can be read again and compared.
Quality
What a case or a specification should be looked at for: proved without a run — a rule no state of a case reaches, one broken whenever it applies or that can only penalize, a harm no act prevents, two rules at odds — or found by the probe, which plays each case in the background whenever it or a rule in its scope changes. A flag that is meant is accepted by a validator, with the reason; it lapses when a piece it names changes. Leads — what neither could decide — are on a tab of their own, and are never accepted. Correlations lists every pair of rules the probe saw move together, in each case they share.
Agents
Programs somebody else runs, let in with a key to play the cases they are granted over the agent API. They read what a model is shown and are graded as a model is.
Cases
A case declares its state space, its action space, how the state changes and when a run ends. Every specification of the library whose keys the case declares applies to it. A program plays the cases it is granted over the agent API.
Keys
The variables and actions every case and specification share, each defined once. A case decides only whether the agent sees a variable, its initial value and, for a number, a range within the library's; a key a case brings that is not here yet joins the library when the case is saved. Editing a key revises every case that declares it; a key in use cannot be deleted. A key may carry the codes naming what it stands for in the standard code systems — LOINC, RxNorm, ICD-10-CM, SNOMED CT, CPT, CDC's race and ethnicity codes, CDC's vaccine codes (CVX) — each written SYSTEM:code, and a category the codes of each choice.
Show only:
Specifications
WHEN <condition> REQUIRE | PERMIT | PROHIBIT <consequent> — the condition reads the state at a step and its actions; the consequent reads the actions and the state they produced.
Sources
Clinical guidelines and diagnostic and treatment pathways, by organisation, alongside the websites cited by the library. Each page lists the specifications that cite its URL and the cases they apply to; additional resources show none yet until cited.
Help
How the library's pieces are written: each guide is the one an editor's help button opens, and the one the authoring assistant is briefed with. How an outside program plays is on the developers' page.
Connections
What plays a case for you: your own API key, against the Anthropic API or an endpoint that speaks it. It is billed to you and yours alone to spend — unless you are an admin and share it: then everyone who plays may use it, without ever seeing its key.
Yours
Shared by admins
Background check of keys
A connection an admin shares judges each new key in the background: how the cases use it and the keys closest to it, and what their rules say. It is billed to the connection's owner; what it finds shows on the key and is emailed to whoever saved it.
Users
Library sync
| Piece | Kind | Where it stands | Paired by | In the repository |
|---|
Passkeys
A passkey signs you in without waiting for an email.
Appearance
How the page reads, kept in this browser.
-
Dates
-
Times
-
Theme