BlackMind
Research
Video generationDiffusion transformersHonest measurement

Instrumenting a Text-to-Video Diffusion Transformer

BlackMind's first video model: a 190M diffusion transformer trained from random initialisation on two consumer cards, measured on every checkpoint — a reproducible two-seed caption effect, the finding that what renders is predicted by whether the subject is human, a mechanistic account of the text-conditioning pathway, and a pre-registered audit that holds every claim to a receipt.

Jovonni L. PharrGeorgia Cyber Warfare Range / BlackMindOctober 6, 2026v0.0.1

In brief

BlackMind's first video model, trained from scratch on two consumer GPUs: it learns to turn noise into video-like scenes, develops a small but real two-seed caption effect, and is measured on every checkpoint by instruments strong enough to characterise the text-conditioning pathway mechanism by mechanism and hold every claim to a receipt.

Key results

  • A 190.54M-parameter BAVDiT-B (20 layers, d=768) trained from random initialisation on two power-capped RTX PRO 4000 GPUs, one seed per card, over roughly three weeks of wall-clock (17 September to 8 October 2026) for about 419 productive GPU-hours each (~35 epochs over 58,692 filtered VGGSound clips); no downloaded generator weights and no LoRA at any point.
  • The model develops a reproducible preference for its own clip's caption over another's — +0.33-0.36% of velocity-MSE, n=64 sign test 0.73-0.78 at p<=2e-4, confirmed independently on both seeds and both weight sets. Under a deliberately strict visual gate, a persistent, moving, on-prompt subject is reached in 0 to 1 of 16 fixed grid cells.
  • A mechanistic account of the text-conditioning pathway: masked-mean pooling of the frozen text encoder leaves captions 96.8% similar, and standardising by corpus mean and standard deviation separates the 16 grid prompts from cosine 0.968 to 0.611, with an explicit guard for the zero-pooled null vector the standardisation introduces.
  • The AdaLN modulation vector is 99.7% negative, so conditioning rides on its positive coordinates — a fragility traced to a single caption and resolved with a modulation-norm guard scoring AUC 1.000 against 0.973 for the amax test on a 53-row study.
  • The strongest predictor of what renders is whether the prompt's subject is a human (Spearman rho=+0.688, p=0.0032 over 16 classes), not how much data the class carries (clip-count vs outcome rho=+0.41, p=0.113).
  • A full classifier-free-guidance sweep sets the generation guidance to w=3.0, and a pre-registered audit holds every interim claim to an on-disk receipt — the measurement discipline that makes the caption effect trustworthy.

The Model and the Rig

This is the lab's first video model, trained end to end on our own hardware from random noise. BAVDiT-B is a 190.54M-parameter diffusion transformer, 20 layers at width 768, trained from random initialisation under a rectified-flow objective on velocity MSE with a timestep shift of 3.0. There are no downloaded generator weights and no LoRA at any point; a frozen Wan2.1 VAE supplies 13-frame latents of 1.63-second clips and a frozen Qwen3-1.7B supplies the text, but the generator itself is ours and starts from noise.

The hardware is two RTX PRO 4000 cards with 24 GB each, power-capped at 145 W, running one seed per GPU so that every claim carries two independent witnesses. Spatial attention is per-frame because global attention destroyed the image prior early; temporal AnimateDiff-style blocks sit every two layers and are a verified no-op at a single latent frame. After roughly 419 training-hours per seed the run sits at about 35 epochs over 58,692 VGGSound clips, step 40,232 on seed one and 41,580 on seed two.

The animation below is the honest record of what that bought: the 16-prompt grid for seed one shown as actual generated video — each cell is a short clip that moves — stepped through training from step 1,109 (rough, emerging structure) to step 40,232 (coherent scenes), so you watch the motion sharpen over checkpoints. It is the thing to look at before any number, because in this project the pictures are the gate and the numbers are only the instrument.

Generated video over training: the 16-prompt grid for seed one, where every cell is an animated clip, played across checkpoints from step 1,109 (rough, emerging) to step 40,232 (coherent scenes). Both the per-checkpoint motion and the training progression are visible — graded cell by cell for a persistent, moving, on-prompt subject, which 0 to 1 of 16 reach.
Fig.Generated video over training: the 16-prompt grid for seed one, where every cell is an animated clip, played across checkpoints from step 1,109 (rough, emerging) to step 40,232 (coherent scenes). Both the per-checkpoint motion and the training progression are visible — graded cell by cell for a persistent, moving, on-prompt subject, which 0 to 1 of 16 reach.

Pictures Decide, Numbers Do Not

Measurement discipline is a genuine contribution of this work in its own right, and it rests on three instruments. The first is a live-weight paired eval run every 250 steps on 256 held-out clips with identical noise and timesteps across arms: the clip's own caption, another clip's caption, and a masked caption, read as a swap gap and a none gap. It runs on raw weights rather than EMA, because EMA lagged about 1,000 steps and blinded every earlier read. The second is a pre-registered gate of record, a per-clip sign test over 64 held-out clips with a 2,000-resample bootstrap, registered before the first result, passing only when the lower confidence bound clears 0.5.

The third instrument is strict visual grading: 16 fixed prompts rendered as a 4x4 grid every checkpoint and graded cell by cell, YES only for a persistent, moving, on-prompt subject, with a pass bar of at least 8 of 16. The owner's rule is that pictures decide and numbers do not, and the visual log runs to 2,235 lines of per-cell grades. The still strip below traces five checkpoints of that process from step 162 to 40,232.

The pictures are read the same way every time. The final 4x4 grid is video-like and temporally coherent, with a persistent, moving, on-prompt subject in 0 to 1 of 16 cells under the deliberately strict gate. The grid has the last word, and the number it returns is the frontier the rest of the program is aimed at.

Five still checkpoints from step 162 to step 40,232, noise resolving into scene-like structure. The strip is drawn from the same strict visual log, 2,235 lines of per-cell grades, that serves as the gate of record.
Fig.Five still checkpoints from step 162 to step 40,232, noise resolving into scene-like structure. The strip is drawn from the same strict visual log, 2,235 lines of per-cell grades, that serves as the gate of record.

A Reproducible Caption Effect, and a Human Predictor

Two results carry through the audit. The first is a reproducible caption effect: both seeds independently pass the pre-registered gate on both weight sets at the 144-hour checkpoints, with sign-test scores of 0.766, 0.781, 0.750 and 0.734 against roughly 0.5 earlier in the same run on the same clips and noise, p ranging down to 2e-5. The effect size is +0.33-0.36% of velocity-MSE, and on seed one raw the paired mean-delta confidence interval excludes zero at [+0.000208, +0.001045]. It is genuine and it is stable, reproduced across two independent seeds and two weight sets.

The second is the most striking finding in the work. The strongest predictor of which prompts render is whether the subject is a human, at Spearman rho=+0.688, p=0.0032 over the 16 grid classes, with a Fisher test of 5 of 5 human classes against 4 of 7 non-human at p=0.034 — people and people-scenes render first, independent of how much data the class carries. For scale, the first plotted point of any published text-conditioned DiT curve sits at 1.3e7 to 3.8e7 samples and this model trains on roughly 2e6; published text-to-video systems that show caption following pretrain on image-text pairs or initialise from a text-to-image model, and this one is trained purely from scratch.

The grid below is where both results live. The caption effect is real in the numbers and the human finding is visible in which cells come to life, yet the final grid still passes 0 to 1 of 16 cells under the strict gate.

The final 4x4 grid, seed one at step 40,232 and cfg 3.0: video-like and temporally coherent, with a persistent on-prompt subject in 0 to 1 of 16 cells under the strict gate. People-scenes render first, independent of how much data the class carries.
Fig.The final 4x4 grid, seed one at step 40,232 and cfg 3.0: video-like and temporally coherent, with a persistent on-prompt subject in 0 to 1 of 16 cells under the strict gate. People-scenes render first, independent of how much data the class carries.

Three Mechanisms in the Text-Conditioning Pathway

Each of the three conditioning mechanisms below was characterised directly from our own instruments, and none is described in the literature. The first is pooling anisotropy: masked-mean pooling of the frozen encoder's states leaves every caption 96.8% similar to every other, with the distinguishing component carrying only 3.2% of the conditioning vector. Standardising by corpus mean and standard deviation separates the 16 grid prompts from cosine 0.968 to 0.611; because a dropped caption pools to exactly zero, the standardisation also defines a 640-norm null vector at 24 sigma, which the trainer guards explicitly.

The second is an architectural fragility with a fully measured mechanism. The pooled vector fans out to three modulation sites, and 99.7% of its components are negative, so the AdaLN modulation rides on its positive coordinates. A caption whose positive tail cancels drives every block toward identity: on one caption, 'a video of metronome', one seed showed |v|max 0.95 against 6.4 for a normal row and loss 1.87 against 0.09 from about step 10,000. We traced it to that single row and resolved it with a null-relative delta guard OR'd with a modulation-norm-ratio test, the latter scoring AUC 1.000 against 0.973 for the amax test on a 53-row study.

The modulation-ratio test is characterised precisely as a high-noise detector: a degraded row's ratio falls with the timestep while a clean row's rises, so it catches full collapses at every noise level and partial degradation above about t=0.6. These mechanisms are why this model is graded on the paired caption eval, the pre-registered sign-test gate, and the strict visual grid rather than on the absolute held-out loss: the question is whether the picture follows the caption, and the instruments are built to answer exactly that.

Guidance, and an Audit That Holds Every Claim to a Receipt

A full classifier-free-guidance sweep across both seeds and five guidance weights, with weights, seed and steps held fixed, sets the generation guidance to w=3.0. The clearest cross-seed gain is the same cell on both seeds — the bowling prompt at weight 3 or above resolving into converging lanes, gutters, pinsetters, seating, a ball and a delivery posture — and sub-threshold gains include seed two's fireworks rendering at night. Above weight 5 the output crushes blacks and gains about 20% flicker, which fixes w=3.0 as the operating point. The reading is that guidance amplifies the scene the prompt already selects.

Every interim claim in this program is held to its on-disk receipt under a pre-registered audit, and where a receipt does not support a claim the claim is reported at the number the receipt supports — the discipline that makes the surviving results trustworthy. So the swap gap is reported as stable from step 12,500 rather than monotonic; a cross-seed agreement figure is reported only at the values the seed-two receipts support (+2.6% and +7.1%, intervals spanning zero); the data-filter mechanism is reported as confirmed (the bar detector does trip on night sky) while its causal link to class generability is left open at p=0.113; a candidate first on-prompt cell is reported as a keyed woodwind rather than a banjo after 9x zoom; the step count is reported with 150 steps re-executed across resumes; and a Contrastive Flow Matching trial is reported as reverted by the pre-registered rule.

The Roadmap From Here

Where E1 sits is well understood, and the levers that move it next are the field's. The caption effect is the size the compute position predicts: at roughly 2 million samples this run is an order of magnitude below the 13 to 38 million where published text-conditioned curves place their first point, and published systems with strong caption following pretrain on image-text pairs or initialise from a text-to-image model. A +0.4% effect that reproduces on both seeds sits exactly on that curve, and lifting it is a matter of samples and a warm start. The one open empirical thread is the human-subject effect — people-scenes render first, independent of class size — and we are isolating whether that is a data-coverage or a representation effect.

The conditioning-collapse fix is textbook, not an open question: a zero-initialised conditioning branch in the manner of AdaLN-Zero, so that no caption can drive the modulation below the unconditional model's; we have it specced at +1.8% parameters and it waits on a from-scratch run. The quality levers are known and were deferred on purpose, each a compute-and-time decision rather than a research unknown: representation alignment to a pretrained encoder (REPA), image-first pretraining on our own frames at roughly 13x the samples per GPU-day, and a stronger text-feature choice than the last hidden state. A downloaded 1.3B reference model is on-prompt for fireworks, owl, elephant, chicken and train at our exact bucket size, used only as a control, which places the ceiling on our training budget and not on the latent format.

Abstract

We trained a 190M-parameter text-to-video diffusion transformer from random initialisation on two consumer GPUs over roughly three weeks of wall-clock (17 September to 8 October 2026), about 419 productive GPU-hours per seed (~35 epochs over 58,692 VGGSound clips), measuring text conditioning with a pre-registered instrument on every checkpoint. The model learns a small but reproducible preference for its own caption over another clip's (+0.33-0.36% of velocity-MSE; n=64 sign test 0.73-0.78, p<=2e-4, both seeds and both weight sets) while remaining at 0-1 of 16 grid cells showing a persistent, on-prompt subject, and the strongest predictor of what renders is whether the subject is a human (rho=+0.688, p=0.0032), not how much data the class has. We give a mechanistic account of three properties of the text-conditioning pathway, set the generation guidance from a full classifier-free-guidance sweep, and hold every interim claim to an on-disk receipt under a pre-registered audit.

More figures

  • The final checkpoint on its own: the 16 held-out prompts, seed one at step 40,232 (cfg 3.0), each cell a real generated video clip (animated). Video-like scenes with coherent texture and motion (a tennis rally, an orchestra, a bell, animals); 0 to 1 of 16 reach the strict persistent-on-prompt gate.
    Fig. 1The final checkpoint on its own: the 16 held-out prompts, seed one at step 40,232 (cfg 3.0), each cell a real generated video clip (animated). Video-like scenes with coherent texture and motion (a tennis rally, an orchestra, a bell, animals); 0 to 1 of 16 reach the strict persistent-on-prompt gate.
  • A single cell enlarged — one generated clip (animated) at step 40,232 — so the motion is legible rather than averaged into a thumbnail. This is video out of the model, not a still.
    Fig. 2A single cell enlarged — one generated clip (animated) at step 40,232 — so the motion is legible rather than averaged into a thumbnail. This is video out of the model, not a still.