Chasing the Small-Frontier on a Fixed Rig

A laboratory with two graphics processors and no cloud budget takes a widely reported 135-million-parameter model as its reference and asks what separates its own, slightly larger 158M model from it. The uncomfortable starting fact is that the bigger model scores lower on the lab's own harness, which invites the reading that the difference is one of recipe and skill rather than of raw scale. That premise - a smaller model beating a larger one - is what organizes the entire investigation.

The constraint is absolute and shapes every decision. There are two processors and no others, no cloud, and no additional hardware coming; a single processor sustains on the order of 74 million training tokens per hour, the trainer is not data-parallel, and models are capped at one gigabyte. Local computation is unlimited in marginal cost but not in throughput, so the question is never how to buy one's way to the frontier but how to reach it by skill.

The instructive result is not a state of the art but a sequence of corrections, each overturning a confident prior belief, and each surfaced not by additional training but by adversarial self-audit. Every figure in the work is produced by an analysis script over saved results and can be regenerated on a machine with no GPU at all.

The first and most consequential correction concerns the ruler, not the model. The headline composite averages unnormalized per-task smooth margins, so a task whose margins are intrinsically large dominates the unweighted mean: a single science task, sciq, contributes a margin comparable to the sum of every other task's contribution, making the aggregate a one-task measurement wearing the costume of a composite. A more serious defect afflicts BoolQ, a yes-or-no task scored on the likelihoods of the single tokens 'yes' and 'no', where every model answers largely according to its class prior.

The instability is visible without any per-class breakdown: across seventy logged runs, reported BoolQ accuracy swings with standard deviation 0.091 about the 0.50 chance line, while the genuine four-way task arc_easy is three times steadier at 0.031. The honest measure, balanced accuracy, sits at essentially one half for every model tested including the reference, so the champion's apparent BoolQ victory is an artifact of a prior that simultaneously inflated its standing and concealed the true shape of the gap.

Removing the coin-flip benchmark corrects only our own scores; scoring the reference through the same harness exposed a deeper fault. The two scoring paths computed different quantities - our checkpoints retokenized the answer as though it began a document, the reference's joint encoding silently dropped the first answer token on multi-token answers, and both normalized by token count across incomparable 16k and 49k vocabularies. The fix was a single shared scorer: one joint tokenization, an answer boundary recovered by longest common prefix, and normalization by the answer's byte length, which is tokenizer-invariant.

On the corrected byte-normalized ruler the picture is honest and split. The reference leads mean argmax accuracy 0.413 to 0.377, a gap of 0.036, and leads six of seven tasks; yet on the continuous byte-normalized log-likelihood margin - the capability-sensitive signal - our model leads, +0.197 to +0.163. A model that trails on the discrete decision while leading on the underlying margin is one whose knowledge is close but whose calibration at the decision boundary lags.

That a 135M reference still edges a 158M model on accuracy is not a ceiling but a skill-and-data gap, and the corrected measurement tells us it is a gap of confidence and coverage rather than of gross capability - a gap to close rather than a wall. The per-task breakdown shows the advantage is spread across knowledge and reasoning tasks, not carried by a single column, which is itself the honest reading the broken composite had hidden.

The comparison is deliberately re-measured on a single fixed scorer, n = 300 per task, with both models graded on the same instrument. The split between the two metrics is precisely the discrete-versus-smooth distinction the evaluation literature warns can manufacture or dissolve apparent capability jumps; only on the shared ruler is the comparison meaningful at all.

The second correction concerns the most elementary of hyperparameters. A controlled sweep of the optimizer's peak learning rate at matched token budget describes a clean inverted-U: every prior run, including the historical champion, used 6x10^-4, well cold of the optimum near 1.5x10^-3, where the smooth metric reads 0.204 against 0.069 - a factor of roughly three at identical compute. A small-scale proxy confirms the optimum transfers across scale.

The diagnosis is corroborated from an independent direction. The reference's own technical report discloses that its 135M model was trained at a peak learning rate of 3x10^-3, five times hotter than our historical default and of the same order as the optimum our sweep independently located - far above the 5x10^-4 used for the same family's much larger model. The elementary knob we had mis-set is one the reference had set hot on purpose.

The implication is uncomfortable and clarifying at once: every previous verdict of the form 'this data intervention does not help' was rendered at the wrong learning rate and is void until repeated at the correct one. Indeed, at least one intervention recorded as harmful reverses to helpful once the learning rate is corrected. A single mis-set knob had been silently taxing every experiment the lab ran.

The third correction was the most humbling. The measured deficit is at bottom a deficit of knowledge and reasoning, and the lab had been preparing to supply it by distilling a corpus from a local teacher model. Two independent adversarial audits reached the same objection: a teacher of moderate size emits on the order of a million tokens per hour, so the tens of billions of tokens required would consume years of compute, and the output of a weak generator is itself weak.

The corpus actually needed - curated synthetic textbooks, mathematics, and filtered web text - was the reference model's own openly published training data, re-tokenizable to the lab's vocabulary at the cost of a day of ordinary input and output. The lab had been on the verge of spending years to manufacture, badly, what could be downloaded in a day.

A controlled token-scaling series over the lab's single narrow corpus - six models identical but for the number of passes over the same data - reinforces the point. Held-out bits-per-byte falls monotonically from 1.161 to 1.126 as the model compresses the corpus ever more tightly, while fair equal-weight accuracy does not move at all, holding at 0.389 +/- 0.003, a spread indistinguishable from evaluation noise. Compression improves; competence does not. The lever is not more passes over the data one has but more unique, diverse, high-quality data - which is to be acquired, not fabricated.

Each correction was produced not by a training run but by convening a small panel of independent analytical agents, each charged with proposing a diagnosis and then adversarially auditing the others' diagnoses against the saved record. This process is what caught the BoolQ coin-flip artifact, the scale-domination of the composite, and the futility of manufacturing data. In a regime where each experiment is expensive, the cheapest and most reliable source of progress is the disciplined, adversarial re-examination of what one already believes.

The plan that survives is no longer a guess, because the reference's report supplies the recipe to match. The reference was trained on the order of two trillion tokens, a budget the lab does not have and does not pretend to; but the report is explicit that at this scale the lever is the mixture, not the token count - a roughly sixty-forty blend of educational and filtered web text, an educational-code slice, and above all a step-by-step mathematics corpus the report credits with doubling grade-school-math accuracy, every one of them downloadable.

The concrete plan is to reproduce the reference architecture to the parameter, train on a feasible budget of some one hundred billion unique downloaded tokens - on the order of a thousand tokens per parameter, roughly two processor-weeks - at the corrected hot learning rate over the same mixture, and to grade the result against the reference on the one harness this work has shown to be trustworthy. The objective is no longer to defend a tie but to erase a measured, honest gap of about three points of mean accuracy, with the two levers to do so - a hotter learning rate and a math-and-education mixture - now identified and in hand.