Cargo Cult Research

mlx-rl — reinforcement learning on a Mac

mlx-rl is a GRPO harness that RL-trains LoRA adapters on a single Apple Silicon machine — no cluster, no cloud. One M3 Ultra (96 GB) trains a 35B-parameter model against verifiable rewards: 64 sampled completions per step, judge-graded, ~4 hours per 200-step run.

base vs RL, live

The same 35B model answers twice: once as its pristine base self, once with an RL-trained calibrated honesty adapter (4.3M LoRA parameters — 0.01% of the model). The adapter was trained on trivia only; hedging on unanswerable questions transfers to people and papers it has never seen. Try to break it — flag what you find.

The live demo is offline — the machine's one model slot is busy with a research run. The GPU only fits one of us at a time; try again later.

base — Qwen3.6-35B

+ RL calibrated honesty

verdict: logged — thanks, this feeds the next training round

why this matters

Language models answer confidently whether or not they know. The RL recipe here teaches a model to act on its own uncertainty in free conversation: hedge on the unknowable, answer the known, stop asserting things that don't exist. Post-cutoff paper questions go from 3% hedged (base) to 88% — while famous-paper recall stays at 95–100% and an agentic-coding benchmark shows zero capability cost (45/72 vs 45/72, same harness, adapter the only variable).

base+RL adapter
hedges on made-up people0.200.95
hedges on post-cutoff papers0.500.92
still answers famous papers1.001.00
reads its own per-item uncertaintyAUROC 0.74
SWE-bench-style coding (of 72)4545

Replicated across seeds; the adapter is inert without its system prompt (register-bound by construction, so it cannot leak hedging into contexts that didn't opt in).

built on this harness

Calibrated abstention (the demo above)
The glove program: trivia-trained honesty that transfers to authors, years, and people it never saw. 16–22× per-item known/unknown separation.
SAGE replication
Independent reproduction of SAGE-RL against vanilla GRPO on the same budget — vanilla won (0.775 vs 0.650); SAGE's value turned out to live in decoding (oracle 0.806), now served as an experimental sampler.
Verifiable-reward math
Exact-match math tasks as the harness's regression suite — reward-shape changes are cheap to falsify before burning a long run.
Next: ear training
GRPO on audio-tower models with rewards synthesized from ground-truth audio — teaching a local model to actually hear.