Field Notes

entity

Talkie

StatusStableUpdatedStands on1 sourceSupportsnotes

Talkie trains vintage language models on text from before a stated knowledge cutoff. Introducing talkie describes talkie-1930-13b, a 13-billion-parameter model trained on 260 billion primarily English-language tokens dated before 1931.

Research purpose

The cutoff makes Talkie useful for testing generalization beyond a model’s pre-training data. The project evaluates future-event prediction, scientific and technical novelty, and coding from in-context examples without modern code in pre-training. A modern twin trained on FineWeb compares historical data with modern web data under the same architecture.

Temporal leakage shows that a stated cutoff needs empirical verification. Faulty dates and later editorial material can introduce later knowledge into historical documents, so the effective cutoff is a measured property of the corpus rather than a property of the date filter that assembled it. That is the dataset side of Training data provenance: the origins and timestamps a corpus carries are empirical claims about the records, not guarantees from the filter that selected them.

Data and post-training constraints

Historical text creates substantial OCR noise. In the reported controlled experiments, conventional OCR reached about 30% of the learning efficiency of human-transcribed text. Regex cleaning raised that figure to about 70%.

The project post-trains on structured historical texts and synthetic prompts, then uses preference optimization. Contemporary AI judges and chat partners introduce anachronistic influence into that process.

Built on 1 source (1 external).

Working out connections…