0
Language Modeling Is Compression
Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne et al., 2024. ICLR 2024.
Shows that a language model is also a compressor, and that compression rate is a way to compare models and data without a task-specific metric.
What it's used for here
The emerging-methods survey uses it to argue that model loss on the manuscript measures predictability, not meaning.
Cited by on this site 1 record
1 methods surveys
SRCMathematics
Cite it
Delétang, G., Ruoss, A., Duquenne, P. A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., Hutter, M., & Veness, J. (2024). Language Modeling Is Compression. ICLR 2024. https://arxiv.org/abs/2309.10668
@article{deletang2024language,
author = {Delétang, Grégoire and Ruoss, Anian and Duquenne, Paul-Ambroise and Catt, Elliot and Genewein, Tim and Mattern, Christopher and Grau-Moya, Jordi and Wenliang, Li Kevin and Aitchison, Matthew and Orseau, Laurent and Hutter, Marcus and Veness, Joel},
title = {Language Modeling Is Compression},
journal = {ICLR 2024},
year = {2024},
url = {https://arxiv.org/abs/2309.10668}
}TY - JOUR TI - Language Modeling Is Compression AU - Delétang, Grégoire AU - Ruoss, Anian AU - Duquenne, Paul-Ambroise AU - Catt, Elliot AU - Genewein, Tim AU - Mattern, Christopher AU - Grau-Moya, Jordi AU - Wenliang, Li Kevin AU - Aitchison, Matthew AU - Orseau, Laurent AU - Hutter, Marcus AU - Veness, Joel PY - 2024 JO - ICLR 2024 UR - https://arxiv.org/abs/2309.10668 ER -
Delétang, Grégoire et al.. “Language Modeling Is Compression.” ICLR 2024 (2024).