Cellular-Automata Pre-Pre-Training Language Model Diagram

Overview

A two-panel diagram of pre-pre-training a language model on synthetic cellular-automaton data before ordinary pre-training. It is an editable PowerPoint diagram adapted from the original paper figure under CC BY 4.0.

What the diagram shows

Left, Pre-Pre-Training: a parameter θ_i sampled from a complexity band inside the space of automata defines a rule f_θi that is rolled out over time into a stack of grids X_i ∈ ℤ^H×W×T_<k; the grids become a green token stream (light → dark) that feeds the LLM, with low-, medium- and high-complexity grid examples below. Right, Pre-Training: three natural-language sources (a web-text card, a code card and a maths card with a short proof) become a yellow-to-orange token stream into the same LLM, followed by two small plots: loss vs. pre-training tokens (ours below the baseline) and accuracy vs. automaton complexity (code and web text peaking at different complexities). Every box, arrow, label and math term is a native shape you can recolor, resize and relabel for your own slides, posters or papers.

File format

The download contains .pptx (1 slide, 13.50 × 8.80 in; opens in PowerPoint, Keynote, WPS and LibreOffice Impress; Google Slides works too, but complex freeforms may shift slightly), .svg and .drawio. Includes preview.png (full resolution) and a 1200 px web preview.

Shapes and editing

The file has 596 native editable shapes: 535 auto-shapes, 25 text boxes, 7 freeforms, 29 lines/connectors (4 small groups). No embedded images. Everything is vector. You can edit all text and math (θ_i, f_θi, X_i ∈ ℤ^H×W×T_<k, the maths card with a₁, a₂, …, a_n and S_k), the complexity-band blob, the stacked grid cards, every token cell of both token streams, the down arrows, both LLM bars, the three text cards (web text, syntax-coloured code, maths), the loss and accuracy curves with axes and labels, and every cell of the four placeholder grids.

Embedded images and fonts

This file has no embedded images; everything is vector. The four cellular-automaton grids (the front frame of the rollout stack and the low / medium / high complexity examples) are simplified vector placeholder mosaics of flat colour cells (stripes, blocks and noise), not the paper's pixel textures; paste your own grids over them if you need real samples. Body text uses Georgia and formulas use Times New Roman; both are preinstalled on Windows and macOS (LibreOffice substitutes the metric-compatible Gelasio and Liberation Serif). The blackboard-bold ℤ and the midline ellipsis ⋯ use Cambria Math, which ships with Microsoft Office; other apps substitute a math font. Sub- and superscripts are real formatted text runs, not equation objects. The SVG keeps text as text; the draw.io file is a shape-by-shape conversion and may need small nudges. This is a faithful redraw for reuse, not the authors' original source file.