RoamingPigs Research Notes
Menu

Reproduce

Check my work, and tell me where it fails

Everything needed to repeat the speech experiments is public: the code, the data at a pinned revision, the exact models and a plan written before the run. If you reproduce a result, or fail to, I will publish your report.

Outside reproductions so far: noneSelf-run checks do not count

What is available #

  • The code and results on GitHub, with the analysis plan as pushed on 2026-10-11 at 22:51:28 UTC, before the full run started. GitHub recorded that push time, not me. This is a dated plan. It is not a registered one, and I do not call it preregistration.
  • The audio, pinned to one revision and checked file by file against its manifest.
  • The six model checkpoints, pinned to commit hashes, in MODELS.md. This site describes them by size and kind. The repository names them so you can re-run them.
  • The per-clip results with digests, here at /results/ (model names replaced by size and kind) and on GitHub (with names).

The fast check, about five minutes #

This needs no models. It checks that the published files are intact and that the headline figures follow from the raw per-clip results.

git clone https://github.com/RoamingPigs/trcs && cd trcs
python3 -m pip install numpy
python3 verify_results.py --published results

It re-hashes every results file against SHA256SUMS and rebuilds analysis.json from the raw per-clip files. Expected: no mismatches, and the rebuilt analysis identical to the published one.

The full run, about three hours on a 48-thread CPU #

This downloads the audio and the six models, transcribes 5,760 clips, and compares your transcripts and rates with mine. The exact commands, versions and a one-model shortcut are in REPRODUCING.md. I ran Python 3.14.4, transformers 5.16.1 and torch 2.14.0+cpu on Intel(R) Xeon(R) Platinum 8160 CPU @ 2.10GHz.

What counts as agreement #

The tolerances were written before any outside run and are in expected/tolerances.json. On the same software and a similar machine, at least 95% of transcripts should match exactly. On different hardware or library versions, rare tokens can flip, so the test is whether each published headline rate lies inside your own 95% interval.

Decoding is deterministic, and the transcript for a clip does not depend on its batch: I decoded 24 clips per model one at a time and in batches, and 144 of 144 matched. Transcripts are not guaranteed to match across CPU and GPU or across library versions. That is a known property of floating-point inference, not a defect I am hiding.

Report what you got #

Open a reproduction report on GitHub. Say what you ran, on what, and what came out, and attach the raw output. A failed or partial report is as welcome as a success. Reports are listed in REPRODUCTIONS.md with your name or handle if you want one, and appear on this page.

Rules #

  • “Reproduced” means someone outside the project ran the published code on the pinned data. “Replicated” means they wrote their own implementation. I use these words as the ACM artifact badging scheme (v1.1) defines them.
  • I do not count my own runs, collaborators’ runs without saying so, or a run on different data.
  • I may not delete or edit a report. If one finds a bug, I fix it in a new release, mark the old claim superseded and credit the reporter.
  • Any headline number on these sites states how many outside reproductions back it, including none.

The full rules are in REPRODUCTION-RULES.md.