Efficient Serbian Language and Question Answering Adaptation of LLaMA 3.2 model
- The Institute for Artificial Intelligence Research and Development of Serbia
Fruskogorska 1, 21000 Novi Sad
[adela.ljajic (corresponding author), marko.zubac, gorana.gojic, milos.kosprdic, nikola.milosevic, dragisa.miskovic]@ivi.ac.rs - Bayer A.G.
Muellerstrasse 178, 13353 Berlin, Germany
nikola.milosevic@bayer.com
Abstract
This paper presents a two-stage, parameter-efficient adaptation of the LLaMA 3.2 11B Vision model for Serbian language generation and question an swering. In the first stage, a Serbian-language adapter is trained with QLoRA on a large web-scale Serbian corpus and a curated collection of formal academic texts, adapting the model to Serbian using web-scale and formal academic texts. In the second stage, a task-specific QA adapter is stacked atop the language-adapted model and fine-tuned on a Serbian translation of the Natural Questions benchmark, with evaluation on both in-domain and out-of-domain Serbian QA datasets. The ap proach remains feasible on a single 40 GB GPU. The Serbian language adapter improves accuracy on a Serbian Cloze-style grammatical completion benchmark by 10 percentage points, while the full stacked model achieves strong gains in question answering, improving out-of-domain performance by up to 39.08% in F1, 4.31% in BERTScore, and 29.35% in ROUGE-L.
Key words
Causal Language Modeling, Supervised Fine-Tuning, Serbian Language, LLaMA, QLoRA.
Journal's Facebook page