Transformer-based language models achieve state-of-the-art performance on many natural language processing tasks. However, such language models have grown very large, precluding their use to many NLP practitioners. In this work, simpler language models based on feed-forward and recurrent neural networks are explored and compared with transformer-based models on a custom benchmark. Results show that although transformer-based models outperform simpler models, their performance gain can be quite small and at the cost of a very strong regularisation.
Are Transformers Always Necessary for Statistical Language Modelling?
Capone V.
Conceptualization
;Casolaro A.Membro del Collaboration Group
;Iannuzzo G.Membro del Collaboration Group
;Camastra F.Supervision
2026-01-01
Abstract
Transformer-based language models achieve state-of-the-art performance on many natural language processing tasks. However, such language models have grown very large, precluding their use to many NLP practitioners. In this work, simpler language models based on feed-forward and recurrent neural networks are explored and compared with transformer-based models on a custom benchmark. Results show that although transformer-based models outperform simpler models, their performance gain can be quite small and at the cost of a very strong regularisation.File in questo prodotto:
Non ci sono file associati a questo prodotto.
I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.
