Field Notes

source

Introducing talkie

StatusStableUpdatedStands on1 sourceSupportsnotes

This preserved snapshot records an April 2026 announcement by Nick Levine, David Duvenaud, and Alec Radford. It introduces the 13-billion-parameter talkie-1930-13b model, trained on 260 billion primarily English-language tokens published before 1931.

The announcement frames Talkie as a research instrument for testing generalization beyond pre-training data. It compares the historical model with an architectural twin trained on FineWeb and reports experiments on historical-event prediction, scientific and technical novelty, and in-context programming.

The announcement’s main methodological contribution is its account of the limits of a date-based corpus cutoff. The authors report temporal leakage from faulty dates and later editorial material in historical documents. They also report substantial OCR-related training inefficiency. Conventional OCR reached about 30% of human-transcribed learning efficiency in their controlled experiments; regex cleaning reached about 70%.

For post-training, the project draws on structured historical texts, synthetic prompts, and contemporary AI feedback. The authors identify that feedback as a remaining anachronistic influence on the resulting chat model.

See Talkie for the entity and its implications.

Built on 1 source (1 external).

Working out connections…