Fine-Tuning Challenges of Large Language Models for Low-Resource Languages: an Analysis Based on the Azerbaijani Language
Article Sidebar
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Main Article Content
Abstract
Large Language Models have shown capabilities in natural language processing tasks. Their performance drops drastically on low-resource languages that lack large annotated corpora. We look at the challenges of fine-tuning Large Language Models for low-resource languages with Azerbaijani as a case study. We examine data scarcity, morphological complexity, tokenization problems and domain gaps that make model adaptation harder. We also review methods like data augmentation, cross-lingual pre-training and transfer learning and assess how well they work for Azerbaijani. The results show that small targeted datasets can lead to real improvements as long as tokenization and morphological modeling are handled properly.
Article Details
References
Bostrom, K., & Durrett, G. (2020). Byte pair encoding is suboptimal for language model pretraining. In Findings of EMNLP 2020 (pp. 4617-4624). https://doi.org/10.18653/v1/2020.findings-emnlp.414 DOI: https://doi.org/10.18653/v1/2020.findings-emnlp.414
Brown, T. B., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901.
Coltekin, C., & Rama, T. (2016). Discriminating similar languages with linear SVMs and neural networks. In Proceedings of the Third Workshop on NLP for Similar Languages, Varieties and Dialects
(pp. 15-24).
Conneau, A., et al. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL 2020 (pp. 8440-8451). https://doi.org/10.18653/v1/2020.acl-main.747 DOI: https://doi.org/10.18653/v1/2020.acl-main.747
Creutz, M., & Lagus, K. (2007). Unsupervised models for morpheme segmentation and morphology learning. ACM Transactions on Speech and Language Processing, 4(1), 3. https://doi.org/10.1145/1187415.1187418 DOI: https://doi.org/10.1145/1187415.1187418
Hedderich, M. A., Lange, L., Adel, H., Strotgen, J., & Klakow, D. (2021). A survey on recent approaches for natural language processing in low-resource scenarios. In Proceedings of NAACL 2021 (pp. 2545-2568). https://doi.org/10.18653/v1/2021.naacl-main.201 DOI: https://doi.org/10.18653/v1/2021.naacl-main.201
Hu, E. J., et al. (2022). LoRA: Low-rank adaptation of large language models. In Proceedings of ICLR 2022. arXiv:2106.09685.
Ismayilzada, S., & Ataman, D. (2022). Evaluation of Azerbaijani NLP resources and models. arXiv preprint arXiv:2212.10026.
Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020). The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of ACL 2020 (pp. 6282-6293). https://doi.org/10.18653/v1/2020.acl-main.560 DOI: https://doi.org/10.18653/v1/2020.acl-main.560
Petrov, A., La Malfa, E., Torr, P. H. S., & Bibi, A. (2024). Language model tokenizers introduce unfairness between languages. arXiv preprint arXiv:2305.15425. DOI: https://doi.org/10.52202/075280-1608
Sennrich, R., Haddow, B., & Birch, A. (2016). Improving neural machine translation models with monolingual data. In Proceedings of ACL 2016 (pp. 86-96). https://doi.org/10.18653/v1/P16-1009 DOI: https://doi.org/10.18653/v1/P16-1009