# Lattice EM decipherment: sign boundaries and key learned together, with a structure-preserving null > Close the gap T40 left: its key is recoverable when sign boundaries are known (71 %) but not when they must be learned (11-23 %), because one hard… Source: https://mysteriesdept.com/methods/MTH0DCF04BX248D/lattice-em-decipherment-sign-boundaries-and-key-learned Publisher: Mysteries Department Licence: CC BY 4.0 (credit Mysteries Department and link the page) --- [Methods](https://mysteriesdept.com/methods)[Cryptanalysis](https://mysteriesdept.com/methods?kind=cryptanalysis)Lattice EM decipherment: sign boundaries and key learned together, with a structure-preserving null Method · cryptanalysis # Lattice EM decipherment: sign boundaries and key learned together, with a structure-preserving null Close the gap T40 left: its key is recoverable when sign boundaries are known (71 %) but not when they must be learned (11-23 %), because one hard segmentation was fixed per round. Here no segmentation is ever fixed. **Cryptanalysis** Version **1.0** Licence **MIT12** runsUsed on **1** mystery ## Questions it answers - [Which combination of fifteenth-century cipher practices, if any, reproduces a single section's statistics on its held-out pages?](https://mysteriesdept.com/mysteries/voynich-manuscript/questions/QST0DCAYTD6HZV9/which-combination-of-fifteenth-century-cipher-practices-if) ## Code Python [Download module (.py)](https://mysteriesdept.com/files/private/843856362cf740953fee6fbcc310843b50f4d671c1a960e7b45ed539e8ef7186)`lattice_em.py`[View source](https://mysteriesdept.com/files/private/843856362cf740953fee6fbcc310843b50f4d671c1a960e7b45ed539e8ef7186) ## Runs and results 12 runs · 6 shown | Run | Mystery | Measured | Result | Against | Date | |---|---|---|---|---|---| | RUN0DCF745R93NZ | Voynich | Held-out dictionary coverage (bar 0.7 x true plaintext's) / key accuracy by use / character similarity | control 0.324 / 0.364 / 0.083 (bar 0.488; true plaintext 0.697); permuted-key 0.282 (p 0.19). Pseudo-Latin null: 0.309 and 0.351 decoded coverage (their own plaintexts 0.359, 0.337) | M-T40 rung B (learned boundaries, element solver): key 0.114-0.228, coverage 0.232-0.271 · rung A (true boundaries): key 0.706, coverage 0.336 | | | RUN0DCF74GZ8K4D | Voynich | Held-out dictionary coverage (bar 0.7 x true plaintext's) / key accuracy by use / character similarity | control 0.325 / 0.071 / 0.047 (bar 0.488; true 0.697); permuted key 0.291 (p 0.14) | G1a (alpha 0.05, pw 0.25): key 0.364 | | | RUN0DCFZ7A7Z2R2 | Voynich | Held-out dictionary coverage (bar 0.7 x true plaintext's) / key accuracy by use | control: coverage 0.239 (bar 0.488), key 0.306 after polish (EM alone 0.319); pseudo-Latin nulls 0.234 and 0.248 | Generation 1 (EM alone): coverage 0.324, key 0.364 | | | RUN0DCFZ7YRGFX7 | Voynich | Held-out key accuracy by use / dictionary coverage (bar 0.7 x true plaintext's) per rung | L1 one sign per letter, no nulls, no line layer: key 0.376, coverage 0.312 (bar 0.483). L2 four homophones per consonant: key 0.556, coverage 0.334 (bar 0.487). L3 plus 30 % copied nulls: key 0.247, coverage 0.327 (bar 0.488). (L4 = full setting with line layer, generation 1: key 0.364, coverage 0.324.) | Permuted-key coverage 0.25-0.29 per rung · M-T40 rung A (true boundaries): key 0.706 | | | RUN0DCG2Q30P808 | Voynich | Training log-likelihood and key accuracy: EM started from the true key vs 10 random restarts, on the planted control | one sign per letter, no nulls: EM from truth -40,505 (key 0.777), best random restart -41,315 (key 0.186). Full T39 setting: EM from truth -53,462 (key 0.851), best random restart -53,957 (key 0.250) | The true channel itself, before EM: -41,654 and -55,406 (key 0.777, 0.850 · the rest are nomenclator words of the control's plaintext that the Latin element model doesn't have) | | | RUN0DCG56XXNDPT | Voynich | Held-out dictionary coverage decoded with the true channel and with EM started from it, against the gate bar | one sign per letter, no nulls: 0.518 (true channel), 0.518 (EM from truth); bar 0.483. Full T39 setting: 0.433 and 0.441; bar 0.488 | Best random-restart decodes: 0.312 and 0.324 | | ## Applied to ![Voynich Manuscript](https://mysteriesdept.com/media/folio/78r) **[Voynich Manuscript](https://mysteriesdept.com/mysteries/voynich-manuscript)4 datasets** Herbal A and Herbal B paragraph text: training and held-out pages, Latin genre texts, Naibbe cipher tables and example plaintexts (greshko/naibbe-cipher, commit 719f2f4c), ZL3b transcription ## References 1. [Aldarrab, N., May, J. 2022. Segmenting Numerical Substitution Ciphers.](https://mysteriesdept.com/sources/SRC08JJDRP052CH/segmenting-numerical-substitution-ciphers) 1. [Berg-Kirkpatrick, T., Klein, D. 2013. Decipherment with a Million Random Restarts. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing.](https://mysteriesdept.com/sources/SRC0D8MQ4S22G4V/decipherment-with-a-million-random-restarts) 1. [Ravi, S., Knight, K. 2011. Bayesian Inference for Zodiac and Other Homophonic Ciphers. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.](https://mysteriesdept.com/sources/SRC0D8MSQ1W0XDE/bayesian-inference-for-zodiac-and-other-homophonic-ciphers) 1. [Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pp. 499-506 (ACL Anthology page checked). EM decipherment: a fixed plaintext n-gram model and a learned channel P(cipher sign | plaintext letter), trained by forward-backward. Base of the Track A lattice EM method.](https://mysteriesdept.com/sources/SRC0DCEXM6FBS17/unsupervised-analysis-for-decipherment-problems) [Download module](https://mysteriesdept.com/files/private/843856362cf740953fee6fbcc310843b50f4d671c1a960e7b45ed539e8ef7186) Category Cryptanalysis Version 1.0 · 28 Oct 2026 Licence MIT Language Python Inputs D-HERBAL-SPLIT pages (ZL3b via texts.voynich); section plaintexts D-SECTIONS (element model); nathist_book16.txt (planted control); period.py and the T39 best Herbal A setting Output out/t41_.json (checkpoint) and out/t41__summary.txt Cite [BibTeX](https://mysteriesdept.com/methods/MTH0DCF04BX248D/lattice-em-decipherment-sign-boundaries-and-key-learned/cite.bib) · [APA](https://mysteriesdept.com/methods/MTH0DCF04BX248D/lattice-em-decipherment-sign-boundaries-and-key-learned/cite.txt) Develop this method Methods are versioned like code. Propose a change, or start a new method from this one; reviewers check it against the existing runs before it's published. [Start a new method from this](https://mysteriesdept.com/developers)[Propose a change](https://mysteriesdept.com/developers)[Run on a dataset](https://mysteriesdept.com/developers) Authors **[apollo](https://mysteriesdept.com/u/apollo)** Wrote version 1.0