Large Binary Models

Nearly every code model in wide use is really a model of source text: the identifiers, keywords, and comments a person writes, tokenized from a subword vocabulary numbering in the tens of thousands. Yet source text is not the thing a machine runs. A compiler sits between the two, and the object it emits, a sequence of operations drawn from a fixed instruction set, is the program in the only sense that matters to the machine. Source modeling reaches only the human-facing shadow of computation; the emitted operations are the computation itself.

This work asks whether a model can predict and generate that binary directly, with no natural-language intermediary, and finds that it can, and readily. The corpus is the raw operation-byte stream of compiled CPython: every available source file of a standard installation compiled to a code object, with the raw operation bytes of each nested code object recursively extracted and concatenated, delimited by a code-object separator and a file separator. The alphabet is just 258 symbols, the 256 byte values plus those two markers. Applied to a standard installation the procedure yields 124.2 million tokens over 414,522 code objects drawn from roughly twenty thousand source files, none of which failed to compile; the final five percent is held out. No source text ever enters the model, which sees only the compiled operations and is trained to minimize the mean next-token negative log-likelihood over that stream.

Three decoder-only Transformers of 10M, 18M, and 29M parameters drive held-out cross-entropy from the uniform baseline of 8.01 bits per token down to 0.46 for the largest, a compression ratio of about 17.4x. The improvement is monotone in scale: the 10M model reaches 0.531 bits/token for a ratio of 15.1x, the 18M model 0.480, and the 29M model 0.460 for 17.4x. The model acquires the grammar of the virtual machine, the layout of the operation stream, the sequences of operations that constitute valid computations, and the regularities of their arguments.

The paper is careful that beating a uniform prior is a weak claim, since any model of a structured source does so; the honest question is whether the model beats a serious classical compressor on the identical held-out bytes. It does. On the same held-out five percent, an order-two byte n-gram attains 1.784 bits per token, gzip 1.716, and bzip2, the strongest of the three, 1.246. The model's 0.460 is 2.7x better than bzip2 and 3.9x better than the n-gram, a margin a uniform baseline cannot establish.

The character of the generations is more telling than the perplexity. Sampled and disassembled, they are not low-perplexity noise but genuine idioms of real programs: a generation may open with the prologue operation that begins every code object, load a global name and access an attribute upon it in the pattern of a qualified reference, and close with the operation that returns a value. The model even reproduces the approximately correct counts of the inline-cache placeholders that follow certain operations in the modern virtual machine.

That the model has learned the true distribution over operations, rather than a plausible-looking caricature, admits a direct test. Comparing opcode frequencies in the real corpus and in the model's own generations, measured identically on the raw byte stream, the Pearson correlation is r=0.998. The paper notes this test required care: an earlier comparison that measured the corpus via the disassembler's logical instruction stream while measuring generations from raw bytes reported a spurious r=0.14, because the disassembler hides the inline-cache placeholders that dominate the raw stream. Once both sides are measured identically, the agreement is near-perfect; it was the mismatch, not the model, that the low number had measured.

The tight marginal match is a consequence of how regular the substrate is: the bytecode obeys a near-deterministic grammar in which each operation sharply constrains its successor, so the mean conditional next-opcode entropy is small. This is low-entropy structure that a compact model learns readily and that a model of irregular human source text cannot exploit. It is also structure the marginals alone do not capture.

That the model has learned the grammar, and not merely the frequencies, is shown by comparing the opcode-to-opcode transition matrix of the real corpus to that of the model's own generations. The two are near-identical, at a transition-level Pearson correlation of r=0.956, a bigram-level agreement far stronger than, and not implied by, the unigram-level r=0.998. The paper stresses that this is a transition matrix and not a confusion matrix, so no diagonal is expected; the bright columns are the near-deterministic successors, such as every attribute access and global load routed through an inline-cache placeholder. It is careful, too, not to overstate: that nearly all generated bytes are legal operations is inflated by the ubiquitous cache placeholder, and this remains learned structure, not yet a verified-runnable program.

The generations so far are structurally plausible but not runnable: raw operation bytes spliced into a code object fail, because their arguments index the constant and name tables a model of co_code alone does not co-generate. The response is to stop discarding that metadata. The paper serializes the entire code object, operation stream together with its constant table, name table, argument counts, and stack size, into a single byte-token stream framed by a small set of structural markers, raising the vocabulary from 258 to 278, with integer constants encoded as sign-and-varint bytes. A verification harness reconstructs a genuine types.CodeType and executes it in an isolated subprocess under strict resource limits with an emptied builtin namespace, and a differential oracle independently decompiles the generated operations to an arithmetic expression and checks that the virtual machine's output equals that expression's value on eight probe inputs.

Because the oracle scores the ground-truth corpus at 150 of 150, a failure on a generation is a real defect. On straight-line integer functions a 14M-parameter model trained on this serialization moves the runnable rate from a categorical zero, unrunnable by construction, to one hundred percent by a thousand optimizer steps. Adding a conditional-branch rung on top, over four hundred sampled generations more than ninety-nine percent are correct, computing exactly what their own disassembly says on every probe input, and seventy-five percent are novel, their operation-and-constant signature absent from training. Novel-correct is seventy-four percent (Wilson 95% interval [69.5, 78.0]), up from sixty-two percent on the straight-line rung alone. Because the oracle is perfect and cheap, repeated sampling yields a valid novel program with probability 0.94 at two samples and 0.99 at three, though the paper is explicit that this is reliable yield under unconditional generation, not problem-solving pass@k.

Compiled binary is a viable and unusually efficient substrate for neural modeling. A small finite operation alphabet is a natural base vocabulary for neural computation, and because its generations can actually be run, assertions about them can be checked by execution rather than by perplexity or surface match. Coupled with a companion effort on a general-purpose neural computer that learns the semantics of such an alphabet from execution, the direct modeling of the distribution over operation sequences points toward systems that both write operation sequences and know what they do, operating on the true substrate of computation rather than its human-facing shadow.

The remaining work is stated plainly. The runnable result covers straight-line integer functions and a first control-flow rung; the agenda is to keep climbing the difficulty ladder one virtual-machine feature at a time, names and global lookups over a whitelist, loops with their exception tables, nested code objects, and finally real library functions, holding novel-correct as the metric at each rung. Beyond it lie learning a source-to-bytecode translation (a learned compiler), predicting behavior from bytecode (differentiable execution), and extending the substrate past one virtual machine to WebAssembly and physical instruction sets. The milestone that remains is the generation of programs that not only disassemble but run.