Publications

2026

1. ICML WS

   Uplifting Human Decision Making in AI Evaluation by Automating Benchmark Validity Analysis

   Rodolfo Corona, Sang Truong, Ritwik Gupta, Nhi N. Truong, Atnafu Lambebo Tonja, and 17 more authors

   In Combining Theory and Benchmarks Workshop, International Conference on Machine Learning (ICML), 2026

   Abs HTML PDF

   Evaluations of modern AI systems largely originate from English-speaking, Western nations, posing adoption challenges for other regions due to language resource scarcity, misalignment in cultural values, and blind spots to region-specific problems, knowledge, and perspectives. In this paper, we present a framework and automated pipeline to assess the applicability of AI benchmarks in the context of specific deployment use cases and target populations. Our framework is structured around the ontological, instance-level, and representational components of benchmark inputs and outputs, specifying the conditions under which benchmark evaluation validity would be violated if transferred to a different cultural or geographic context. To enable scalable validity analysis, our automated pipeline leverages large language models to evaluate benchmark-use-population triplets across our six validity dimensions. We validate this pipeline through human expert studies and apply it to assess 24 benchmark-use-population triplets spanning five global regions, surfacing systematic patterns in how porting strategies affect validity. We conclude with policy recommendations for actors to improve the benchmarking ecosystem in their respective regional contexts.

2024

1. NAACL

   Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models

   Sang T. Truong^*, Duc Q. Nguyen^*, Toan Nguyen^*, Dong D. Le^*, Nhi N. Truong^*, and 2 more authors * Equal contribution.

   In Findings of the Association for Computational Linguistics: NAACL 2024. Featured in The New York Times and Stanford HAI. , Mar 2024

   Abs HTML PDF

   Recent advancements in large language models (LLMs) have underscored their importance in the evolution of artificial intelligence. However, despite extensive pretrained on multilingual datasets, available open-sourced LLMs exhibit limited effectiveness in processing Vietnamese. The challenge is exacerbated by the absence of systematic benchmark datasets and metrics tailored for Vietnamese LLM evaluation. To mitigate these issues, we have finetuned LLMs specifically for Vietnamese and developed a comprehensive evaluation framework encompassing 10 common tasks and 31 metrics. Our evaluation results reveal that the fine-tuned LLMs exhibit enhanced comprehension and generative capabilities in Vietnamese. Moreover, our analysis indicates that models with more parameters can introduce more biases and uncalibrated outputs and the key factor influencing LLM performance is the quality of the training or fine-tuning datasets. These insights underscore the significance of meticulous fine-tuning with high-quality datasets in enhancing LLM performance.

2. ICLR WS

   Hybrid Transformer and Holt-Winter’s Method for Time Series Forecasting

   Nhi N. Truong, Duc Q. Nguyen, Jeffrey Gropp, and Sang T. Truong

   In Time Series for Health Workshop, International Conference on Learning Representations (ICLR), Mar 2024

   Abs HTML PDF

   Time series forecasting is an important research topic in machine learning due to its prevalence in social and scientific applications. Multi-model forecasting paradigm, including model hybridization and model combination, is shown to be more effective than single-model forecasting in the M4 competition. In this study, we hybridize exponential smoothing with transformer architecture to capture both levels and seasonal patterns while exploiting the complex non-linear trend in time series data. We show that our model can capture complex trends and seasonal patterns with moderately improvement in comparison to the state-of-the-arts result from the M4 competition.