美联储-全面召回？大型语言模型的宏观经济知识评价（英）

Finance and Economics Discussion SeriesFederal Reserve Board, Washington, D.C.ISSN 1936-2854 (Print)ISSN 2767-3898 (Online)Total Recall? Evaluating the Macroeconomic Knowledge of LargeLanguage ModelsLeland D. Crane, Akhil Karra, Paul E. Soto2025-044Please cite this paper as:Crane, D. Leland, Akhil Karra, Paul E. Soto (2025).“Total Recall?Evaluating theMacroeconomic Knowledge of Large Language Models,” Finance and Economics Discus-sion Series 2025-044.Washington: Board of Governors of the Federal Reserve System,https://doi.org/10.17016/FEDS.2025.044.NOTE: Staff working papers in the Finance and Economics Discussion Series (FEDS) are preliminarymaterials circulated to stimulate discussion and critical comment. The analysis and conclusions set forthare those of the authors and do not indicate concurrence by other members of the research staff or theBoard of Governors. References in publications to the Finance and Economics Discussion Series (other thanacknowledgement) should be cleared with the author(s) to protect the tentative character of these papers.Total Recall? Evaluating the Macroeconomic Knowledgeof Large Language Models*Leland D. Crane†Akhil Karra‡Paul E. Soto†June 24, 2025AbstractWe evaluate the ability of large language models (LLMs) to estimate historical macroe-conomic variables and data release dates. We find that LLMs have precise knowledgeof some recent statistics, but performance degrades as we go farther back in history. Wehighlight two particularly important kinds of recall errors: mixing together first printdata with subsequent revisions (i.e., smoothing across vintages) and mixing data forpast and future reference periods (i.e., smoothing within vintages). We also find thatLLMs can often recall individual data release dates accurately, but aggregating acrossseries shows that on any given day the LLM is likely to believe it has data in hand whichhas not been released. Our results indicate that while LLMs have impressively accuraterecall, their errors point to some limitations when used for historical analysis or to mimicreal time forecasters.*We thank Gary Cornwall, Anne Hansen, participants in Board brownbags, and participants at the 2025 SGEconference for useful comments. We thank Betsy Vrankovich for her technical expertise.Opinions expressed herein are those of the authors alone and do not necessarily reflect the views of the FederalReserve System or the Board of Governors.†Board of Governors of the Federal Reserve System‡Carnegie Mellon University1IntroductionThe rise of large language models (LLMs) has generated interest in how they can be used foreconomic analysis and forecasting (e.g., Korinek 2023). The utility of LLMs depends on theirunderstanding of economics-related facts and their ability to follow instructions precisely.We evaluate LLMs on several dimensions related to these capabilities. First, how well doLLMs estimate important macroeconomic variables from the past? Second, to what extentare LLMs’ estimates contami