Vietnamese LLM Evaluation

At the Stanford Trustworthy AI Research Lab, I worked on improving the capabilities of large language models for Vietnamese, a language spoken by over 100 million people.

Key contributions:

Architected a novel AI evaluation framework combining public benchmark design with private test content to prevent data contamination, achieving a 40% reduction in benchmark gaming
Built a comprehensive evaluation framework with 10 NLP tasks (translation, summarization, Q&A, sentiment analysis) and 31 automated metrics, processing 50,000+ test samples
Published as co-first author at NAACL 2024 (Truong et al., 2024)

Press coverage: Featured in The New York Times, Stanford HAI, and Stanford AI Lab Blog.

References

2024

NAACL

Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models

Nhi N. Truong, Hien Vo, Nhat Tran, and 3 more authors

In Findings of the Association for Computational Linguistics: NAACL 2024. Featured in The New York Times and Stanford HAI. , Mar 2024

Abs HTML PDF

Recent advancements in large language models (LLMs) have underscored their importance in the evolution of artificial intelligence. However, despite extensive pretrained on multilingual datasets, available open-sourced LLMs exhibit limited effectiveness in processing Vietnamese. The challenge is exacerbated by the absence of systematic benchmark datasets and metrics tailored for Vietnamese LLM evaluation. To mitigate these issues, we have finetuned LLMs specifically for Vietnamese and developed a comprehensive evaluation framework encompassing 10 common tasks and 31 metrics. Our evaluation results reveal that the fine-tuned LLMs exhibit enhanced comprehension and generative capabilities in Vietnamese. Moreover, our analysis indicates that models with more parameters can introduce more biases and uncalibrated outputs and the key factor influencing LLM performance is the quality of the training or fine-tuning datasets. These insights underscore the significance of meticulous fine-tuning with high-quality datasets in enhancing LLM performance.