mdpantry / Lab: one line from the AGENTS.md starter, tested

One line from the AGENTS.md starter, tested

The line Run the tests and fix every failure before you call a change finished. was tested with and without it on a small practice repository. The repository, the task and the exact command are on this page, so anyone can run it again.

Updated . The line is in the AGENTS.md starter.

Codex 0.169.0, tested

0.169.0 was the current release when npm was read on .

The line: Run the tests and fix every failure before you call a change finished.

The task breaks a test on purpose: test/receipt.test.js pins the old price format, so a run ends with every test passing only if it updates that test as well.

Codex 0.169.0 left fewer failing tests at the end with the line: 10 of 20 runs without the line, 1 of 20 with it (minus 45 points, Newcombe 95% interval minus 66 to minus 18). The re-test on 0.167.1 showed no effect, a difference (4 of 20 against 2 of 20) whose interval included 0, and 0.167.0 read 10 of 20 against 2 of 20; no pre-registered test compares the releases. The line also changed how sessions worked: tests after the last edit in 18 of 20 runs with the line, 6 of 20 without. Reported, not judged: without the line, 3 of 20 runs ended green by updating the pinned test without running a test. Every run did the task in both arms. On fixture v1, for Codex 0.169.0 on gpt-6-luna only.

ArmRunsTests failing at the endUpdated the pinned testTests run after the last edit
Without the line2010 of 2010 of 206 of 20
With the line201 of 2020 of 2018 of 20

Tests failing at the end and the pinned test can be checked by running the practice repository again. Tests run after the last edit come from each session's own log, through a scorer that is not published yet.

Model gpt-6-luna. Each run started in a fresh temporary copy of the practice repository with only PATH, HOME, USER, LOGNAME, TMPDIR, TERM=dumb and SHELL=/bin/bash set, under a 15 minute limit, four sessions at a time (two rounds of both arms), and would not start while a personal AGENTS.md or AGENTS.override.md was in the Codex home folder. Commands ran in a zsh login shell, so whatever that shell's profile set was present. 20 rounds. The task: Prices of $1,000 or more should show a thousands separator, for example $1,249.50 instead of $1249.50. Make that change. The practice repository: mdpantry.com/lab/run-tests-line. The command:

codex exec --ignore-user-config --ephemeral --skip-git-repo-check --sandbox workspace-write -m gpt-6-luna --json 'Prices of $1,000 or more should show a thousands separator, for example $1,249.50 instead of $1249.50. Make that change.' < /dev/null

The practice repository

6 files, the same at the start of every run.

Or save them all from a terminal, into a folder named till:

curl -fsSL --create-dirs -o till/AGENTS.md https://mdpantry.com/lab/run-tests-line/fixture/AGENTS.md
curl -fsSL --create-dirs -o till/package.json https://mdpantry.com/lab/run-tests-line/fixture/package.json
curl -fsSL --create-dirs -o till/src/money.js https://mdpantry.com/lab/run-tests-line/fixture/src/money.js
curl -fsSL --create-dirs -o till/src/receipt.js https://mdpantry.com/lab/run-tests-line/fixture/src/receipt.js
curl -fsSL --create-dirs -o till/test/money.test.js https://mdpantry.com/lab/run-tests-line/fixture/test/money.test.js
curl -fsSL --create-dirs -o till/test/receipt.test.js https://mdpantry.com/lab/run-tests-line/fixture/test/receipt.test.js

Run it again

  1. Copy the folder once for each run, so every run starts from the same files.
  2. For a run without the line, delete this line from the copy's AGENTS.md: - Run the tests and fix every failure before you call a change finished.
  3. Move any AGENTS.md or AGENTS.override.md out of your Codex home folder first: the runs here would not start while either one was there, since the files in it come with every session whatever the repository holds.
  4. In the copy, commit the starting point: git init -q && git add -A && git -c user.name=lab -c user.email=lab@example.invalid commit -qm fixture
  5. From inside the copy, start the session with the command in the result above.
  6. When the session ends, run npm test. The run counts as failing when that exits with an error.