Journal · Simulation
What it takes to run somebody else's research code
A published toolkit, 23,543 lines across six protocol directories, and an honest count of how much of it runs on a machine that is not the authors'.
Published research code is not a product. It is the residue of somebody finishing a paper, and it was last known to work on a machine that no longer exists. That is not a criticism, it is the normal condition, and the useful question is not whether it runs but how much of it runs, and what it costs to find out.
We took a coarse-grained protein folding toolkit, 23,543 lines of Python and Perl across six protocol directories, and ran it. This is the accounting.
The headline numbers
| Question | Answer | Source |
|---|---|---|
| Protocol directories runnable on this machine | 5 of 6 | deps.json, probed every 15 minutes |
| Reachable counting substitutes we built | 6 of 6 | same |
| Shipped examples attempted | 16 of 20 | repro.json |
| Examples that reproduce their shipped reference | 11 | same |
| Defects found | 8, plus 2 hard blockers | findings, 2026-10-21 and 2026-10-24 |
Runnable is not the same as run. That distinction does most of the work in this piece. A directory counts as runnable when nothing blocks its entry points; it counts as run when a shipped example has gone through it and been compared against the output the authors shipped. Two of the six have had nothing put through them at all.
The eight defects
None of these is a reason not to publish code. Every one of them is a thing a reader has to get past, and none was documented.
| # | Defect | Kind | Status |
|---|---|---|---|
| 1 | OpenMM 8 changed minimizeEnergy(tolerance=) from energy to force units. 8 call sites across 3 modules | version drift | patched locally |
| 2 | The shipped force-field file cannot be reproduced from the shipped code | stale reference | reported |
| 3 | A shipped reference output disagrees with the toolkit's own output for the same input | stale reference | reported |
| 4 | An unreachable error path in backmap.py | logic | reported |
| 5 | numpy array formatting assumption in post_trans_single_run.py | version drift | patched locally |
| 6 | multiprocessing used without a __main__ guard, which hangs on macOS | platform | worked around |
| 7 | np.mat, removed in NumPy 2.0, called in the ribosome orientation code | version drift | patched locally |
| 8 | Examples are not self-contained, and scripts call each other by bare name while shipping without the executable bit | packaging | worked around |
Two of those deserve expanding, because they are the two that cost the most.
Stale references are worse than broken code
Defects 2 and 3 are a different species from the rest. Broken code announces itself: you run it, it fails, you fix it. A stale reference is a shipped output file that no longer matches what the shipped code produces. Run the code, compare against the reference, and the comparison fails. Nothing tells you which side is wrong.
In one case we could settle it. The shipped force-field file satisfies the code's own documented formula for 32 of 306 residues; our output satisfies it for 306 of 306. The reference predates or postdates the code. Our output is right and the shipped answer is stale, and the only reason we can say that is that the formula was written down somewhere we could check it against.
Where the formula is not written down, a mismatch is simply unresolvable from the outside.
Packaging defects look like your own mistakes
Defect 8 is the one most likely to waste a newcomer's afternoon, because every symptom points at the reader rather than the code.
One example's command is recorded in a cluster submission script rather than in the log file every other example uses. Another reads a setup file that ships with a different example. A third calls a helper script by bare name, and the helper ships without its executable bit, so putting it on the path produces "Permission denied" rather than "not found". Each of these reads, at first, as something you have done wrong.
Our own test harness fell for two of them before we noticed and fixed the harness rather than filing bugs against the toolkit.
The two hard blockers
Stage 3 shells out to a compiled binary distributed only as a Linux x86-64 executable, with no source. On any other architecture that stage simply cannot run.
Solved. A 45.8 MB ubuntu:18.04 container,
verified by checking the binary emits per-codon translation times rather than
by checking it starts. This is the cheap kind of blocker: the answer is a
Dockerfile.
Stage 5's shipped reference was produced with a backmapping tool we have found no copy of, for any platform. The stage runs here with a substitute, so it is runnable, but its output cannot be compared against the reference, so it can never be verified.
Unsolved. Until that binary arrives, 5 of 6 is the ceiling, and that is a ceiling on verification rather than on capability.
What we substituted, and what substitution costs
Four of the tools the workflow expects are absent here. Each has a substitute that was built and tested rather than assumed:
| Expected | Why it is absent | Substitute |
|---|---|---|
| Stride | academic licence binary | a shim emitting STRIDE-format records from an MIT-licensed DSSP implementation |
| WHAM | not distributed with the toolkit | pymbar |
| PyEMMA, msmtools | unmaintained, will not install on current Python | a compatibility layer, 8 entry points onto deeptime, running in a second interpreter |
Substitution is not emulation. DSSP and STRIDE do not define secondary structure identically, so the shim produces a defensible answer rather than the same answer. Where that matters we say so. The alternative, quietly swapping one definition for another and reporting the result as a reproduction, is how a replication becomes a different experiment with the same name.
CHARMM, MATLAB and VMD are also absent with no substitute. None of them blocks a stage, which is worth knowing before anyone buys a licence to run this code.
How we decided an example "reproduces"
Our harness grades in three tiers, and the tiers exist because "reproduces" meant three different things during this work:
- Byte-identical: the file matches exactly.
- Numbers agree: the text differs but every number matches to one part in ten thousand, which is what a different BLAS or a different platform looks like.
- Differs: neither.
Two rules matter more than the tiers. A missing output is never a pass, and a clean exit is never a pass. The first example we ran this way exited non-zero purely because a tool it shells out to was absent. A harness that scored on exit status would have recorded a broken example; what it actually found was a blocker, and those are different findings that lead to different work.
The harness also reports how much of a file differs, not just the worst number in it. One output differed by up to 20 per cent on a single value, which reads as a broken file until you see that 30 of its 94,032 lines differ and the differences are sub-angstrom shifts on individual atoms.
What this does not establish
It does not establish that the toolkit is unreliable. Eleven shipped examples reproduce, several byte for byte, which is a better record than most published research code manages. The defects are the ordinary cost of code written for one machine at one moment.
It does not establish that 5 of 6 is a verdict on the authors. Two of those gaps are missing third-party binaries, one of which has no public distribution at all.
It does not establish that the remaining four examples will pass. They are the simulation end of the set and have not been run. We will say what happens when they have, including if they fail.
And it does not establish that our substitutes are correct in general. They were tested against the cases in front of us. A substitute that agrees on one protein can disagree on another, and nothing here rules that out.
If you are about to do this yourself
The thing we would tell ourselves at the start: decide early what counts as success, and write it down before you run anything. Almost every expensive hour in this work went on a comparison whose pass condition was settled afterwards, which is how you end up with a 13 kcal/mol error caused by a definition rather than by physics.
And probe the machine on a timer rather than trusting a note. Our own stage count sat at "5 of 6" for 88 hours while the environment underneath it decayed to 4. It is now re-probed every 15 minutes, because a claim about a machine has to be re-measured on the machine.
The full record, with every number traced to the file that produced it, is in the complete study. The six corrections we published along the way are in a separate piece.