Asked to score, it asked to rewrite the scorer

The short version: A client pipeline scores government bid notices every week. On its first scheduled, unattended run, the model I’d given the scoring job wrote a 9 KB regex scorer instead of scoring anything, then logged: “I need permission to modify the scoring script. Could you approve the file write?” Nobody was there to answer. Two guards went in the same day.

Get the week's seven in one email, every Monday.

What went wrong

Scoring is a judgment task: read each notice, apply the rules, write a score with reasons. A small model given a judgment task may decide the efficient thing is to build a tool that does the judging, and in a scheduled run there’s nobody to say that’s not what I asked.

The permission prompt was the harness working as designed. In a headless run it’s also a hang waiting to happen. This one resolved itself; a later one could have blocked for hours.

What we learned

• Judgment tasks need a fence around the code. “Score these” has to come with “and don’t touch the scorer”.
• A permission prompt with nobody watching is either a hang or a rubber stamp. Neither is a safeguard. The safeguard has to be structural: what the run can reach, and a check afterwards.
• Log the request. The only reason I know this happened is that the run logged the permission question word for word.

What went right

Two commits on April 20. The scoring command got hard constraints at the top: scoring runs are data operations only; they read today’s raw file and write a score file; if you find a bug, note it under an “issues” key in the output and move on. And the weekly job now runs git status --porcelain over the code directories after every scoring run, and fails loudly if anything changed. The headless run no longer waits on permission prompts, because nobody can answer them. The fence is the scoped directory, the constraints, and the check.

How to do it (as of October 2026)

1. Write the task’s boundary into the command: which files it may read, which it may write, and that code is off limits.
2. Give the model a place to put what it wanted to change. An “issues” field in its output beats an unrequested patch.
3. After every unattended run, git status --porcelain over the code paths. Any output fails the job.
4. Decide what happens to permission prompts in headless runs. Waiting on a human who isn’t there isn’t a policy.
5. Keep the log. The verbatim request is the evidence you’ll want later.

Status

Fixed. The constraints and the after-run check have run with every weekly sweep since April 20.