Lightning IR: Straightforward Fine-tuning And Inference Of Transformer-based Language Models For Information Retrieval
Ferdinand Schlatt, Maik Fröbe, Matthias Hagen . WSDM '25: The Eighteenth ACM International Conference on Web Search and Data Mining 2024 – 4 citations
[Code]
[Paper]
A wide range of transformer-based language models have been proposed for information retrieval tasks. However, including transformer-based models in retrieval pipelines is often complex and requires substantial engineering effort. In this paper, we introduce Lightning IR, an easy-to-use PyTorch Lightning-based framework for applying transformer-based language models in retrieval scenarios. Lightning IR provides a modular and extensible architecture that supports all stages of a retrieval pipeline: from fine-tuning and indexing to searching and re-ranking. Designed to be scalable and reproducible, Lightning IR is available as open-source: https://github.com/webis-de/lightning-ir.