Method · cryptanalysis

Lattice EM decipherment: sign boundaries and key learned together, with a structure-preserving null

Close the gap T40 left: its key is recoverable when sign boundaries are known (71 %) but not when they must be learned (11-23 %), because one hard segmentation was fixed per round. Here no segmentation is ever fixed.

CryptanalysisVersion 1.0Licence MIT12 runsUsed on 1 mystery

Questions it answers

Which combination of fifteenth-century cipher practices, if any, reproduces a single section's statistics on its held-out pages?

Code Python

Download module (.py)lattice_em.pyView source

Runs and results 12 runs · 6 shown

Run	Mystery	Measured	Result	Against
RUN0DCF745R93NZ	Voynich	Held-out dictionary coverage (bar 0.7 x true plaintext's) / key accuracy by use / character similarity	control 0.324 / 0.364 / 0.083 (bar 0.488; true plaintext 0.697); permuted-key 0.282 (p 0.19). Pseudo-Latin null: 0.309 and 0.351 decoded coverage (their own plaintexts 0.359, 0.337)	M-T40 rung B (learned boundaries, element solver): key 0.114-0.228, coverage 0.232-0.271 · rung A (true boundaries): key 0.706, coverage 0.336
RUN0DCF74GZ8K4D	Voynich	Held-out dictionary coverage (bar 0.7 x true plaintext's) / key accuracy by use / character similarity	control 0.325 / 0.071 / 0.047 (bar 0.488; true 0.697); permuted key 0.291 (p 0.14)	G1a (alpha 0.05, pw 0.25): key 0.364
RUN0DCFZ7A7Z2R2	Voynich	Held-out dictionary coverage (bar 0.7 x true plaintext's) / key accuracy by use	control: coverage 0.239 (bar 0.488), key 0.306 after polish (EM alone 0.319); pseudo-Latin nulls 0.234 and 0.248	Generation 1 (EM alone): coverage 0.324, key 0.364
RUN0DCFZ7YRGFX7	Voynich	Held-out key accuracy by use / dictionary coverage (bar 0.7 x true plaintext's) per rung	L1 one sign per letter, no nulls, no line layer: key 0.376, coverage 0.312 (bar 0.483). L2 four homophones per consonant: key 0.556, coverage 0.334 (bar 0.487). L3 plus 30 % copied nulls: key 0.247, coverage 0.327 (bar 0.488). (L4 = full setting with line layer, generation 1: key 0.364, coverage 0.324.)	Permuted-key coverage 0.25-0.29 per rung · M-T40 rung A (true boundaries): key 0.706
RUN0DCG2Q30P808	Voynich	Training log-likelihood and key accuracy: EM started from the true key vs 10 random restarts, on the planted control	one sign per letter, no nulls: EM from truth -40,505 (key 0.777), best random restart -41,315 (key 0.186). Full T39 setting: EM from truth -53,462 (key 0.851), best random restart -53,957 (key 0.250)	The true channel itself, before EM: -41,654 and -55,406 (key 0.777, 0.850 · the rest are nomenclator words of the control's plaintext that the Latin element model doesn't have)
RUN0DCG56XXNDPT	Voynich	Held-out dictionary coverage decoded with the true channel and with EM started from it, against the gate bar	one sign per letter, no nulls: 0.518 (true channel), 0.518 (EM from truth); bar 0.483. Full T39 setting: 0.433 and 0.441; bar 0.488	Best random-restart decodes: 0.312 and 0.324

Applied to

Voynich Manuscript

4 datasetsHerbal A and Herbal B paragraph text: training and held-out pages, Latin genre texts, Naibbe cipher tables and example plaintexts (greshko/naibbe-cipher, commit 719f2f4c), ZL3b transcription

Lattice EM decipherment: sign boundaries and key learned together, with a structure-preserving null

Questions it answers

Code Python

Runs and results 12 runs · 6 shown

Applied to

References