Real output from the finished 7B, excerpted. Asked about the cattle, it names the devil before contagion; asked about the fever, it prescribes a sweat — and ends its turn.
Porous
A language model that believes it is 1620.
Ask a modern assistant to role-play 1620 and you get the diction with today's assumptions underneath — it will say “Providence hath ordained” and then add that many would attribute it to natural causes. Porous goes the other way: it shifts what a model takes for granted, by pretraining on what that world actually printed. Named for Charles Taylor's porous self — a self open to spirits, omens, and meanings that live out in the world.
How it's built
- Corpus — 723M tokens across 11,240 texts of English print through 1660 — EEBO-TCP (hand-keyed TEI, not OCR) plus patristic and scholastic translations. Streamed XML→text extraction; mechanical orthography normalized (ſ→s, u/v), lexical spelling kept.
- A layman's mix — Every text tagged by genre, tradition, and register, then weighted toward a worldly gentleman's reading — 47% practical (medicine, conduct, law, travel, almanacs), 36% divinity, 17% lore — rather than a preacher's.
- Base model, not chat model — Continued pretraining of Qwen2.5-7B base. RLHF'd assistants carry a modern voice — hedged, pluralist — welded in too deep to fine-tune away, so the chat layer is instead built from the period's own Q&A: catechisms, cases of conscience, disputations.
- Measure before training — 66 probes score a period vs. a modern continuation of the same neutral stem (“The crops failed that year because…”). The metric is the delta-of-deltas against an untrained model, with a control that separates real learning from the model simply getting worse at English.
- Validate cheap, then scale — A $4.50 pilot on a 1.5B model moved the prior 80% of the way to parity on 3% of the corpus — and caught three broken probe designs before any real spend. The 7B run followed: 120M tokens on 8×H100, eval perplexity 12.8 → 9.6.
- Teach it to stop — Chat SFT on 16k examples of the period's own Q&A and colloquy — after which the model answered and then ran on into chapter headings. Two retrains at higher learning rates changed nothing, because the turn-end token had been masked out of the loss in every example: it had never once been taught to end a turn. One config line, verified on the labels for $0.12 before spending, and it ends its turn 5 of 5. The prior survived every step — −0.17 mean Δ against +1.06 untrained.
- Runs on a laptop — 4-bit GGUF through Ollama, 4.4 GB, behind a plain web chat. A three-turn exchange about the tides holds its thread, resolves “it” from two turns back, and closes with the colloquy's own formula: “and so much of the tides: now let us passe on to the next place.”
- 723M
- corpus tokens
- 11,240
- texts, all pre-1661
- 66
- worldview probes
- ~$72
- total compute
What it is, and isn't. A research artifact, not an oracle: it reconstructs the defaults of the printeddiscourse — clergy and pamphleteers, not the farmhand — and it will invent a plausible 1620 that never happened. Held faithfully, that worldview also includes the era's prejudices — unhedged, because hedging is the modern voice it was built to shed. It runs behind an email login at porous.sivori.xyz; a one-time code, no account.
Under the hood
- EEBO-TCP / CCEL / Gutenberg
- Python stdlib TEI pipeline
- Qwen2.5-7B base
- axolotl · FSDP · flash-attn
- Modal 8×H100
- llama.cpp · Ollama, 4-bit local