Journal · Simulation
Six times we were wrong, and what caught it
We ran a replication. Part of it held and the part that could really fail did. Getting there meant being wrong six times in ways that each looked, at the time, like being right.
In October 2026 we took a published protein folding toolkit we had never used, got it running, calibrated its force field against one laboratory measurement, and then ran the calibrated strength. The model reached −6.039 ± 0.069 kcal/mol against a laboratory value of −6.0 ± 0.6, a z of −0.06. That is the value the calibration was aimed at, and the calibration runs either side of it already agreed with it, so it could not really have missed. The test that could miss, carrying the calibration to a second protein, did miss.
That sentence is worth almost nothing on its own. It is worth something only because of the six wrong answers we published on the way to it, and because each one was caught by a specific check rather than by noticing later. This piece is the sequence of corrections. The numbers are in the full record; what follows is what each error looked like from the inside.
We cannot grow faster than we learn, and we cannot learn faster than we are willing to be wrong where people can see it. So the failures get published, or the learning stops compounding.
1. A gate that passed on the wrong evidence
Replica exchange works by letting copies of a molecule at different temperatures swap places. The usual health check is the acceptance rate: what fraction of proposed swaps are taken. Ours looked fine, between 0.54 and 0.81 across the ladder, so we called the gate passed and moved on.
Acceptance is not the thing that matters. What matters is whether a replica actually travels from the cold end to the hot end and back, a round trip, because that is what mixes the ensemble. We had not counted them. When we did, the count was zero, in every one of the fifteen windows, across 2,002 cycles.
The acceptance average had hidden the problem. One pair in the middle of the
ladder was accepting at 0.080 while its neighbours ran at 0.54
to 0.81. The ladder was severed at that pair, and nothing crossed it. A single
number averaged over the whole ladder cannot show you a wall in the middle of it.
Source: mixing.json.
2. Then we over-corrected and nearly threw away a good result
Having found zero round trips, we concluded the run was unusable and said so. That was also wrong, and in the more expensive direction, because it would have discarded real compute.
Round trips are sufficient evidence of convergence, not
necessary. The question is whether the quantity you are measuring
has stopped moving. We went back and cut the run into quarters, discarding the
first as equilibration: the folded populations were stationary across the other
three, and the melting point came out 345.45 K three times running. The run had
delivered a melting point. It just had not delivered it by the route we were
checking for. A stable melting point is not the same as a reliable stability,
though, and the later 3CLA stability is marked provisional for exactly this
reason.
Source: tm_3cla_partial.json, which records the zero round trips
and the quarterly melting points side by side.
3. A 13 kcal/mol error that was not in the physics
This is the one that should worry a reader most, because nothing about it looked like a bug.
We reported a folding stability of −20.802 kcal/mol against a published −8.4315. That is not a disagreement, it is a different universe, and the natural reading is that the model is wrong.
The model was fine. To compute stability you must first decide which configurations count as folded. We had used a potential energy proxy, counting a structure as folded if its energy was below a threshold. The toolkit's own criterion is the fraction of native contacts formed, Q, with a threshold of 0.7321. Same trajectory, same windows, same physics. Changing only that definition moved the answer to −7.415 ± 0.859, which is z = 1.18 against the published figure and agrees within two standard deviations.
A definition we had chosen for convenience, not the simulation, was responsible
for a 13.4 kcal/mol error. The original number is still on file, marked
dG_298_INVALID with the reason attached, because deleting it would
remove the evidence that the criterion is the load-bearing choice.
Sources: stability_3cla_v2.json,
stability_3cla_Q.json.
4. A straight line through four points that were not straight
We needed to know how folding stability responds to the one tunable strength parameter in the force field. Three runs gave what looked like a clean linear relationship, and a line through three points always looks clean.
The fourth run broke it. The residuals from the straight-line fit come out
+0.182, −0.186, −0.183, +0.187, which is not noise,
it is a U. The relationship is curved, and the quadratic fits it essentially
exactly. Three points cannot show curvature; they can only fail to contradict a
line. Source: nscal_calibration.json.
This one mattered beyond tidiness. The linear fit puts the target at a strength of 1.897 and the quadratic at 1.890. We built the force field at 1.882 and tested it, which is the result at the top of this page.
And then one of those four points turned out to be wrong, which is why the numbers above are not the ones we first published. On 2026-10-26 we re-ran the lowest point and got −1.750 kcal/mol where we had published −1.436. The cause was not the simulation. The analysis had been pointed at a different protein's secondary-structure definitions, 3CLA's, applied to the 59-residue SH3 domain. That file decides which residues count as structured, which decides which contacts count as native, and the number of native contacts is the denominator of the folding coordinate. So it was 40 contacts instead of 50 and every configuration in the run was graded on the wrong scale. We confirmed it by reproducing the old published numbers exactly from the 40-contact set, to all four decimal places of the folded fraction.
What gave it away was a number that did not change. The effective sample size was identical to one decimal place across both versions, and that quantity depends only on the energies and the temperatures. So the simulation was never in question and only the grading could be wrong, which turned an open-ended hunt into a three-parameter one. Nothing in the output had recorded which definitions file was used, so a wrong-protein result looked exactly like a right one. Every analysis here now records that file and the resulting contact count.
The honest postscript: correcting it strengthened the curvature finding
rather than removing it, and moved the target strength from 1.882 to 1.890, which
is 0.008 away from the force field we had already built. The conclusion survived.
The number behind it did not. Source:
FINDINGS_ns1.1954_ssdef_2026-10-25.md.
5. An explanation that fit the facts and was still false
An earlier run missed a published value by 3.4 standard deviations. We had a good explanation: the laboratory construct and the deposited structure cover different stretches of the protein, so we were not simulating quite the same molecule.
It was a satisfying story and it was wrong. We went to the structure and counted: the relevant entry resolves 58 of its 67 residues, which is not the mismatch the explanation required. The construct could not account for the miss.
For a while we had no replacement explanation, and recorded the comparison as failed for reasons unknown rather than attach a plausible story. The explanation that the evidence later supported came from a different direction: the calibration runs showed the same model reaching the measured value at a stronger force field, so the miss is the strength the toolkit ships, not the construct.
6. A scaling law that rested on a single event
We published an exponent: mixing cost scaling as N to the 1.62 with protein size. It came from a fit, the fit came from data, and the data were real.
Underneath the fit, the number of round-trip events supporting one of the points was one. An exponent fitted to a single event is not a measurement of anything. The claim was withdrawn and replaced with a bound and an admission: the cost grows faster than N to the 3.5, and the exponent is properly unmeasured.
The pattern underneath all six
Read together, these are not six unrelated mistakes. Five of the six share a structure, and it is worth naming because it generalises well past molecular simulation:
A component reported success based on something other than the thing it claimed to measure, and the report was believed because it was the kind of report that is usually right.
Acceptance stood in for mixing. Energy stood in for folded. A line through three points stood in for the shape of a curve. A plausible story stood in for a cause. A fit stood in for a measurement. In every case the substitute was correlated with the real thing often enough to look trustworthy, and the check that caught it was the same each time: go and count the thing you are actually claiming.
That discipline is cheap to state and expensive to practise, because the substitute is nearly always the number you already have and the real thing nearly always needs new work.
What this does not establish
This is a piece about our own corrections, and it would be easy to let it imply more than it does.
It does not establish that the model is good. It reproduces two published values for two proteins and misses a third, and one of the successful comparisons has no stated uncertainty so no agreement can be tested at all.
It does not establish that we have found all our errors. Six is the number we caught. The honest reading of a list like this is that it is a lower bound, and the uncomfortable corollary is that the errors still standing look, right now, exactly like the six above looked at the time.
It does not establish that the method transfers. The force field calibrated on one protein reproduced the value it was calibrated against. The same calibration carried to a second protein, CI2, failed decisively.
The useful version of this for your own work
If you take one thing from the six, take the question rather than the answers. For any number a system reports to you as healthy, ask what it would look like if the system were broken in a way the number cannot see. Acceptance looks identical whether or not a ladder is severed in the middle. A clean linear fit looks identical whether or not the relationship is curved. A green check looks identical whether it checked the thing or something adjacent to it.
Then go and count the thing you are claiming. It is usually a few hours of work, and in this programme it was the difference between a replication and a press release.
The full working, with every number traced to the file that produced it, is in the complete record of the study.