LLMs for statistical methodology advice - findings and advice
Joni Karanka* 1, Kurban Bilal2, Wesley Yung3, Amilina Kipkeeva4
Abstract
The Advanced Data Science and Modern Methods (ADSaMM) is a UNECE sponsored group working on the impact of changing methodologies and techniques on the production of Official Statistics. Within our remit we were interested on how statisticians and statistical institutions might start utilising Large Language Models (LLMs) in their work. LLMs can collate and cross-reference large amounts of textual information, providing novel generated answers to queries made to them. Given the speed at which LLM performance has improved recently and their wider adoption in many industries, we can assume that statisticians will also start to use them.
We conducted research on the ability of LLMs to provide methodological advice to users for a set of common queries. These queries are a subset of representative problems that statisticians might want to resolve carrying their duties, such as how to impute certain type of data or estimate a variable in small geographical areas when only an overall written description of the task is given to the LLM. The set of queries were used as prompts for several LLMs: ChatGPT, Bing (in its Creative and Balanced modes) and Bard. We then assessed the responses provided to the problems by the LLMs with experts in our respective organisations. These assessments were qualitative and took the form of conversations or email exchanges, which allowed us to make general remarks and conclusions with regards to the suitability of the current generation of LLMs for the provision of methodological advice. Although we did notice differences in performance between the different models used, such a comparison was not the focus of this research.
We provide the main results as both findings and recommendations as although currently LLMs can provide suitable answers, these can be both biased and limited, and statisticians should be methodic and careful when using them.
1: Office for National Statistics - United Kingdom