Field notes · General

Five Open Models Matched a Paid One at Hard Coding Problems. The Judge Was a Bash Script.

Five copies of an open-weight 27B model, working as a team, matched a paid frontier model on ten hard coding problems. What the result shows, what it doesn't, and the test I ran on the script that decides pass or fail.

General Intermediate 12 min read
Five Open Models Matched a Paid One at Hard Coding Problems. The Judge Was a Bash Script.

A field note on the team-solve method. Facts as of Nov 2, 2026.

The short version

A research team published a method called GVS5H in August. Copies of one mid-size open model work a coding problem as a team. One plans, two pitch different approaches, one writes the code, one reviews the test results, and one applies the fix. With that setup, an open-weight model called Qwen3.8-27B went from 66.8% to 92.4% on 100 hard contest problems, according to the paper’s abstract. The paid frontier model they compared against, Claude Fable 5, scored 90.4% in a single call.

An open-source agent tool called Memdoor has rebuilt that team as eight small YAMLA human-readable data format used for configuration files, favored for its clean syntax with indentation instead of brackets and braces. Read more → files and posted every run in public. On the first ten problems of the authors’ list, five passes each, its version averaged 9.8 out of 10. The authors’ own code scored 9.6 on the same ten, and Fable 5 in one call scored 8.0. That’s from the team-solve README, and the Hacker News thread posted Oct 31 has drawn hundreds of points and comments.

The interesting part isn’t the score. The model didn’t get smarter. The loop around it got more disciplined, and a plain bash script decides whether the work passed.

I read the workflow, ran its pass/fail script against three programs I wrote, and read the installer before telling anyone to pipeThe | operator that sends one command's output directly into the next command's input, letting small tools be chained into a single job. Read more → it into bash. Here’s what holds up.

What this means if you don’t write code

Some terms first.

  • Open-weight model: an AI model whose files you can download and run on your own hardware. Qwen is one family of them.
  • 27B: 27 billion parameters. It’s a rough size label. Bigger models usually need bigger, more expensive hardware.
  • Frontier model: the top paid models from the big AI labs, used through their APIA set of rules and protocols that allows different software applications to communicate with each other and share data or functionality. Read more → at a per-tokenThe smallest unit of text that a language model processes, typically a word, part of a word, or a punctuation mark. Read more → price.
  • Benchmark: a fixed test set everyone scores against. LiveCodeBench Hard is a set of hard programming contest problems with hidden test cases.
  • Context windowThe maximum amount of text (measured in tokens) that a language model can read and consider at once when generating a response. Read more →: how much text a model can hold in view at once, measured in tokens.

Here’s why people care. If a team of smaller models can do the work of one big one, you can get strong coding help without paying frontier prices, and in principle on hardware you own, where your code never leaves your network.

A real example of the method, from the Memdoor sample problem: given a string of a, b and c characters up to 200,000 long, count the substrings with equal numbers of each. The planner writes a plan file. Two “ideators” each write a proposal into a shared notes file. The solver picks one and writes solution.py. A script runs it against sample inputs. A reviewer reads the actual test output and writes repair instructions. A repair worker applies them. A final script runs the samples again, and its exit codeThe number a program returns when it finishes, where 0 means success and anything else signals a specific kind of failure. Read more → is the verdict.

How the team works

From the team-solve README, the step order is:

plan ──┬── ideate-a ──┬── solve ── test ── review ── repair ── verify
       └── ideate-b ──┘
Flow diagram of eight steps, with ideate-a and ideate-b side by side after plan, and test and verify marked as shell commands.
Plan, two ideas in parallel, solve, test, review, repair, verify. Agents talk only through files on disk.

Each step is one YAML file in steps/. Six are agent steps with a prompt. Two, test and verify, are plain shellThe program that reads the commands you type, runs them, and shows the results; Bash is the default on most Linux systems. Read more → commands. A requires: key on each step builds the dependency graph, so the two ideators run in parallel once the plan exists. The engine that runs the graph is a separate project, mario.

Every agent prompt starts from a fresh context and is told to read the files on disk and not rely on chat history. The solver’s prompt, for example, says “Do not spawn agents or use chat context.” The agents talk to each other only through plan.md, notes.md, review.md and solution.py. The paper calls this “ledger-based” orchestration, and its abstract says transcript analysis “attributes the gain to decomposition and persistent context.”

That matches what I see on real ops work. A model that starts each step clean and reads a written plan drifts less than one dragging a long, messy conversation behind it.

The parallel ideators show the care that went into this. Both write to the same notes.md at the same time, so each one inserts its proposal right before its own marker, <!-- proposal-a --> or <!-- proposal-b -->, and is told never to rewrite the file. That’s the same problem two cron jobs appending to one log have, solved the same way.

Read the claim the way the author wrote it

The Memdoor README is unusually honest about its limits, and any write-up that skips them is overselling. In its own words:

  • Small sample. Ten problems, five passes, against the authors’ 100. “9.8 and 9.6 are within noise. The claim is parity with their harness.”
  • Possible contamination. The contests date from spring 2025, so the model may have seen them in training. “Not ruled out for any column.”
  • Uneven fight. Fable 5 got one call per problem, taken from the authors’ files. It wasn’t run as a team.
  • Not local. “Host: OpenRouter, not a local machine.” OpenRouter is a paid service that routes API calls to many model hosts.
  • The loop can break good code. On the one lost problem, “the manager invented a wrong counterexample and the team rewrote a correct program.” Twice more, a correct program was overwritten and later recovered.
A card listing four caveats from the team-solve README.
The author's own caveats. Ten problems, within noise, possible contamination, run in the cloud.

Memdoor reports a median of 40 minutes and $0.39 per run, and $35.79 for all 50 runs. A single call to a model takes seconds. You’re trading wall-clock time and many more tokens for accuracy.

The test I ran: what the judge actually checks

The whole method leans on one claim: a script, not a model, decides success. So I tested that script.

Both the test and verify steps call check-samples.sh. It runs python3 solution.py on each tests/N.in, compares the output to tests/N.out with diff, prints PASS or FAIL, and exits nonzero on any failure. That’s all it does.

I copied check-samples.sh and the five sample tests unchanged from the repo and wrote three solutions to the sample problem by hand:

  • fast.py: the right algorithm. Track running differences between the a, b and c counts and count repeats with a hash map. One pass, O(N).
  • brute.py: correct answers, wrong speed. Check every substring. O(N²).
  • wrong.py: a plausible bug. Count only three-character windows containing a, b and c.

No model ran in this test. It checks the judge, not the team. Box: Debian, Python 3.13.5, GNU bash 5.2.37, run Nov 2, 2026.

fast.py and brute.py both pass:

### solution = brute.py
PASS tests/1.in
PASS tests/2.in
PASS tests/3.in
PASS tests/4.in
PASS tests/5.in
Samples: 5; failures: 0
exit=0

wrong.py gets caught, with the diff to show why:

### solution = wrong.py
PASS tests/1.in
FAIL tests/2.in: output differs
FAIL tests/3.in: output differs
PASS tests/4.in
FAIL tests/5.in: output differs
Samples: 5; failures: 3
exit=1

Then the part the samples don’t cover. The problem allows strings up to 200,000 characters and says the solution “must run in O(N) or O(N log N) time.” I generated a random 200,000-character input and timed both passing solutions, with a two-minute cap:

-- fast.py
521393

real	0m0.088s
exit=0
-- brute.py

real	2m0.005s
exit=124
Two terminal panels. Both show five PASS lines and exit 0. Below, fast.py shows real 0m0.088s and brute.py shows real 2m0.005s, exit 124.
Both passed every sample. One finished in 0.088 seconds. The other was killed at two minutes.

Exit 124 is timeout killing it. Same verdict from the judge, PASS on every sample. One finished in under a tenth of a second. The other never finished.

What that tells you. The judge catches wrong answers on the cases it has. It doesn’t check speed, and it doesn’t know about cases it wasn’t given. The longest sample input is nine characters. The README says this plainly: “The loop sees the public samples; the score is the hidden tests.” The benchmark scores came from the authors’ hidden tests and their grader, not from this script.

So if you point this workflow at your own problems, exit 0 means “passed the tests you wrote.” Your test set is the real spec. Put a large input with a time limit in tests/, or add a timed check to verify, or the team can hand you a correct program that falls over in production.

That isn’t a knock on Memdoor. It’s the same rule as any CI pipeline. Green means your tests passed, nothing more.

Running it yourself

The README’s quick start:

curl -fsSL https://memdoor.ai/install.sh | bash
memdoor connect openrouter        # or anthropic, openai, gemini, chatgpt
# or no key: memdoor connect local   (Ollama, llama.cpp, vLLM, LM Studio)
git clone --depth 1 https://github.com/guregodevo/memdoor
mkdir -p ~/.memdoor/workflows
cp -r memdoor/examples/workflows/team-solve ~/.memdoor/workflows/
mkdir my-problem && cd my-problem
cp -r ../memdoor/examples/workflows/team-solve/sample/* .
memdoor workflow run team-solve --wait   # exit 0 when every sample passes

Your input is a task.md plus paired tests/N.in and tests/N.out files. It needs Python 3 and bash.

Gotcha 1: Ollama’s context window. The team-solve README warns that “Ollama needs a 32K window, not its 4K default.” Ollama’s own FAQ confirms the default: “By default, Ollama uses a context window size of 4096 tokens.” It’s set with the OLLAMA_CONTEXT_LENGTH environment variableA named value stored outside your code that configures application behavior, commonly used for API keys, database URLs, and deployment settings. Read more →, so for this workflow:

OLLAMA_CONTEXT_LENGTH=32768 ollama serve

Check the FAQ for your installed version, since defaults change. A model with too small a window doesn’t throw an error. It just loses the start of the plan and gives you worse answers, which is the worst kind of failure to debug.

Gotcha 2: the installer. I read install.sh before running anything. What I found, as of Nov 2, 2026:

  • It installs to ~/.local/bin by default with no sudoA Linux command that runs a single command with administrator (root) privileges, after checking that your account is allowed to do so. Read more →. --system puts it in /usr/local/bin and needs sudo.
  • It downloads the binary from https://memdoor.ai/dl/ and checks it against a SHA256SUMS file. It fails closed if the manifest can’t be fetched, the entry is missing or the hash doesn’t match.
  • The checksumA fixed-length string produced by running a file through a hash function, used to verify the file has not been corrupted or modified. Read more → file comes from the same dl/ directory as the binary. That catches a corrupt or half-swapped download. It won’t help if someone controls that whole directory, because they could replace both files. Building from source or running the install in a throwaway VM or LXC covers that case. Our checksums field manual covers why a signature beats a checksum from the same host.

None of that says the script is unsafe. It says read it before you pipe it, which applies to every curl | bash.

Gotcha 3: where your code goes. Per the README, files and commands stay on your machine, but whatever the model reads goes to the model provider. On OpenRouter, Memdoor sends data_collection: deny on every request, except for :free models, whose hosts “train on what they are sent.” With memdoor connect local, the README says nothing leaves the machine. If the code is private, use a local model or a paid key, not a free tier.

My take

This is the right direction for operators, and the reason is the design, not the score.

The method is portable. The workflow names no model. The useful idea is fresh context per step, a written plan on disk, and a deterministic check that decides done. You can lift that pattern into your own automation with any model, including the one you already pay for.

Treat the benchmark as “worth testing,” not “proven.” Ten problems, within noise, possible contamination, run in the cloud. The full paper covers 100 problems and is the stronger evidence. The Memdoor result shows the method reproduces, which is useful. It doesn’t show a homelab GPU replaces a subscription. Nobody has published that yet.

The time cost is real. A 40-minute median per problem is fine for an overnight batch of hard bugs. It’s useless for interactive coding. On local hardware you’re trading API dollars for hours and power, and I don’t have measured numbers for that tradeoff yet.

Your tests are the product. My test above shows it. The team optimizes for whatever the judge checks. Thin tests give you code that passes thin tests.

How I’d run it in production:

  • Pin the model and the quantization, and record both with each run.
  • Keep the workflow YAML in git and review changes like code.
  • Gate merges on the verify exit code, and make verify include a large timed case.
  • Log the reviewer’s notes. When the manager “invents a wrong counterexample,” you want the paper trail.
  • Start it on problems where a wrong answer is cheap.

What’s next

Running the full five-agent team on a local 27B model needs a GPU box, and I haven’t done that run yet. When I do, it’ll go in a follow-up with real wall-clock time, memory use and pass/fail on the same sample problem, one call against the full team on the same box. No numbers until they’re measured.

Sources

ailocal-modelsagentsworkflowsopen-weightcoding

← All field notes