Mysteries Department
Sign inRequest access
MethodsStatistical physics (Monte Carlo optimisation)Population-annealed EM for the lattice key, compared with independent restarts
Method · statistical physics (Monte Carlo optimisation)

Population-annealed EM for the lattice key, compared with independent restarts

Problem Track A can't solve with restarts: the diagnostic run on the planted control shows EM started from the true key reaches a higher likelihood than any random restart (one sign per letter: -40,505 against -41,315; full setting: -53,462 against -53,957), so the failure is in the search, not the model. Lasry-style annealing and random restarts don't reach that basin.

Statistical physics (monte carlo optimisation)Version 1.0Licence MIT3 runsUsed on 1 mystery

Questions it answers

Code Python

Download module (.py)lattice_em.pyView source

Runs and results 3 runs

RunMysteryMeasuredResultAgainstDate
RUN0DCGEKE5D3QCVoynichTraining log-likelihood, key accuracy by use, held-out dictionary coverage (bar 0.7 x true plaintext's), on the planted controlOne sign per letter, no nulls: deterministic annealing ll -42,066, key 0.000, coverage 0.477 (bar 0.483); population annealing ll -42,067, key 0.000, coverage 0.496 (above the bar), 55 % of words read as nulls. Full T39 setting: deterministic annealing ll -55,506, key 0.000, coverage 0.457; population annealing ll -55,598, key 0.000, coverage 0.414 (bar 0.488). Pseudo-Latin null under deterministic annealing: coverage 0.432Independent restarts on the same controls: ll -41,322 (key 0.376) and -53,904 (key 0.364) · EM from the true key -40,505 and -53,462
RUN0DCGXS4DNN5RVoynichHeld-out dictionary coverage; manuscript gate: coverage(Herbal A) - mean coverage(nulls) > coverage(control) - mean coverage(nulls), and key-permutation p < 0.01 (100 permutations)control 0.497 (bar 0.483; key 0.000, character similarity 0.062, permutation p 0.03); pseudo-Latin nulls 0.398 and 0.340 (mean 0.369; their own plaintexts 0.352, 0.346); Herbal A held-out 0.282, permuted keys mean 0.289, p 0.584Control margin over the null: 0.497 - 0.369 = 0.128 · Herbal A margin: 0.282 - 0.369 = -0.087
RUN0DCGYKWPD5D2VoynichTraining log-likelihood and key accuracy on the one-sign-per-letter control, against independent restartsll -41,455 (all 32 members converge to one state), key 0.218, coverage 0.328200 independent restarts, same control: ll -40,969, key 0.698

Applied to

Voynich Manuscript
4 datasetsHerbal A and Herbal B paragraph text: training and held-out pages, Latin genre texts, Naibbe cipher tables and example plaintexts (greshko/naibbe-cipher, commit 719f2f4c), ZL3b transcription

References

  1. Journal of the Physical Society of Japan 65(6): 1604-1608 (J-STAGE page checked). Parallel tempering (replica exchange). Track B candidate key search to compare with Lasry-style annealing.
  2. Neural Networks 11(2): 271-282 (bibliographic record with DOI checked on researchr.org). EM with the posterior computed from p(x, z) raised to a power beta that rises to 1, to escape poor local optima. Used in T41 generation 4 after the diagnostic showed the true key's basin has higher likelihood than any random restart.
  3. arXiv:1006.0252 (abstract page checked; journal reference not confirmed here). Population annealing: a population of states is annealed through a temperature ladder and resampled by Boltzmann weights at each step. Track B fresh idea in T41 generation 4, compared with independent restarts.
Private beta Downloads are for beta members

We're in a private beta while we get things right. Everything here is free to read; downloading datasets, sources and code is open to members for now.

Apply to the betaSign in
It takes a couple of minutes. We'll reply by email.