Behavior-Probe

Behavior-Probe

Can a model work out how one particular student gets subtraction wrong, then predict that student's next answers?

Each task shows a few worked answers from one simulated student who follows a hidden, systematically buggy procedure. The model must predict what that student will write on 40 new problems. Reproducing correct arithmetic earns nothing on the questions that count.

Setting
AnthropicOpenAIIdeal reasoner

Leaderboard

Accuracy is exact-match on the questions the student gets wrong, averaged over 20 tasks; ± is one standard error across tasks. 3-digit questions resemble the worked examples; 4–5-digit questions test whether the inferred rule transfers. Brier measures calibration of the stated confidence on the same questions (lower is better). Cost was not recorded for Codex CLI runs.

All settings

Accuracy on student-error questions, one run per cell

Per task

Each cell is one model's accuracy on one student, in percent. Hover a cell for the number of probes used. × marks a submission the grader rejected (it omitted answers), which scores 0. Hidden rules and answer keys are withheld to keep the test set clean.

Method

The task

A simulated student subtracts column by column but mishandles borrowing in a fixed, rule-governed way. The model sees a handful of that student's answers and the 40 test problems, and writes a JSON file with a predicted answer and a confidence for each.

ProblemStudent wroteCorrect

Two difficulty levels

  • Easy. The student has one of 13 textbook subtraction bugs. Six worked answers are shown and test problems have 3 or 4 digits.
  • Hard. The student combines two interacting bugs, each with its own trigger condition, drawn from 2,304 possible students. Only four worked answers are shown, chosen so that many different students fit them, and test problems go up to 5 digits. Test problems are picked where the students that fit the examples disagree.

Optional probes

In the probe settings the sandbox also has a probe A B command that returns this student's answer to a problem of the model's choosing, up to 10 times. Test problems cannot be probed. Using it is left entirely to the model, and the number of probes each model chose to use is reported.

Scoring

  • A prediction is correct only if it equals the student's answer exactly. There is no partial credit and no model-based judging.
  • Half of the 40 test problems are ones the student gets wrong. The headline accuracy counts only those 20, since any model that can subtract gets the other 20.
  • A submission that omits a question or is malformed scores 0 for that task.
  • The ideal reasoner keeps every candidate student that fits the evidence and answers by majority vote. Its score separates what the evidence allows from what the model achieved.

Setup and limits

  • Each model runs in its vendor's agent harness (Claude Code or Codex CLI) inside an isolated sandbox with a 30-minute limit. The grader, hidden rules and answer keys never enter the sandbox.
  • Every cell is a single run over 20 students. Treat gaps smaller than the ± column as noise.
  • The students are simulated. Results show whether a model can identify a rule-governed error pattern, not how it would model real children.