Efficient Serbian Language and Question Answering Adaptation of LLaMA 3.2 model

Adela Ljajić1 Marko Zubac1 Gorana Gojić1 Miloš Košprdić1 Nikola Milošević1,2 Dragiša Mišković1

  1. The Institute for Artificial Intelligence Research and Development of Serbia Fruskogorska 1, 21000 Novi Sad
    [adela.ljajic (corresponding author), marko.zubac, gorana.gojic, milos.kosprdic, nikola.milosevic, dragisa.miskovic]@ivi.ac.rs
  2. Bayer A.G. Muellerstrasse 178, 13353 Berlin, Germany
    nikola.milosevic@bayer.com

Abstract

This paper presents a two-stage, parameter-efficient adaptation of the LLaMA 3.2 11B Vision model for Serbian language generation and question an swering. In the first stage, a Serbian-language adapter is trained with QLoRA on a large web-scale Serbian corpus and a curated collection of formal academic texts, adapting the model to Serbian using web-scale and formal academic texts. In the second stage, a task-specific QA adapter is stacked atop the language-adapted model and fine-tuned on a Serbian translation of the Natural Questions benchmark, with evaluation on both in-domain and out-of-domain Serbian QA datasets. The ap proach remains feasible on a single 40 GB GPU. The Serbian language adapter improves accuracy on a Serbian Cloze-style grammatical completion benchmark by 10 percentage points, while the full stacked model achieves strong gains in question answering, improving out-of-domain performance by up to 39.08% in F1, 4.31% in BERTScore, and 29.35% in ROUGE-L.

Key words

Causal Language Modeling, Supervised Fine-Tuning, Serbian Language, LLaMA, QLoRA.

Digital Object Identifier (DOI)

https://doi.org/10.2298/CSIS251215056L