Making America Hallucinate Again? GPTZero Detects New Errors in Major Government Report
On May 22, the U.S. Presidential Commission to Make America Healthy Again (MAHA), led by health secretary Robert F. Kennedy Jr., released a major report on the causes of chronic diseases in children. Yet within a week, news outlets including NOTUS, the New York Times and Washington Post reported that some of the publication’s hundreds of footnotes appeared garbled or fabricated, suggesting they’d been generated by AI. Using GPTZero’s Bibliography Scan tool, we identified nine fabricated sources in the report on May 29—the most comprehensive list available at that time. Our tool also revealed sources that failed to support the relevant claim, or that contained inaccurate components (including authors, titles, publications, or DOIs), streamlining the fact-checking and verification process.
Fake, AI-generated citations erode public trust and undermine institutional credibility. They’re also a common byproduct of AI tools that are hallucinating more, not less, as they increase in sophistication. There is no doubt that leadership in private sector and government want to avoid hallucinations, however, that's hard to enforce throughout their organization without the right tools. GPTZero helps journalists, researchers, and teams publishing high-stakes, time-sensitive reports reduce the risk of AI-fabricated sources. You can use the Bibliography Scan today (https://gptzero.me/sources) or reach out at team@gptzero.me.
AI-Generated Sources1
1A Growing Problem
Fake sources have become a common indicator of AI use over the past two years, showing up in legal documents, academic articles, and innumerable student essays. They indicate a continuing limitation of large language models – a failure to reliably generate accurate citations with working links – that has created headaches for teachers, reviewers, and editors across a variety of industries.
GPTZero has been powering investigations in the rising use and abuse of AI since its founding in January 2023 [1][2][3], and has pioneered several tools to help detect AI-generated text and citations. We first analyzed the MAHA report’s citations on May 29, using our Bibliography Scan tool, in response to a request from a media outlet. We later compared our findings to the changes made to the actual report between May 28 (before the story broke) and May 30 (after the NYT and WP had published their versions). The results indicated that Bibliography Scan had flagged the majority of the fabricated sources mentioned in the media, as well as several not previously identified. Our tool also revealed multiple inaccurate citations later fixed by post-publication alterations to the MAHA report.
How Bibliography Scan Works
Released earlier this year in response to the growing issue of hallucinated citations, Bibliography Scan attempts to verify citations from both directions. First, it checks if the source exists by conducting a web search to find the cited source, providing both a link to the most likely match and the percentage of similarity between key components of the matched source and the original citation. The result of this verification is signified by color: green (the source was found online), yellow (a similar source was found, but it’s not an exact match), and red (the source can’t be found).
Second, Bibliography Scan attempts to verify that the source is actually relevant and supports the appropriate claim. The result is once again coded by colour (green for “support”, yellow for “neutral”, and red for “contradict”) and includes a short summary of the model’s reasoning.
The accelerating problem of AI-generated citations across the web has made Bibliography Scan popular among those seeking a way to quickly verify factual content, from teachers to policymakers to analysts. Still, the 68-page MAHA report, with its 522 footnotes citing academic articles, government reports, and scientific studies, was a formidable test for our tool. After breaking the text into more manageable chunks, we identified seven sources that were red-flagged as potentially fabricated. Our tool also yellow-flagged dozens more sources, including one identified as fabricated by news media.
Fabricated Sources on May 28th
MAHA Report Citation (May 28) | Bibliography Scan Result | Key Media Mentions | Revised MAHA Report Citation (May 30) |
61. Cigna Corporation, "The loneliness epidemic: Insights from the 2024 loneliness in America survey," 2024. [Online]. Available: https://www.cigna.com/aboutus/newsroom/studies-reports/loneliness-epidemic. | No Change | ||
141. Guthrie, J. F., & Lin, B.-H. (2024). Peeling open U.S. fruit consumption trends (Economic Research Report No. 341). U.S. Department of Agriculture, Economic Research Service. https://www.ers.usda.gov/publications/pub-details/?pubid=110658. | No Change | ||
370. K. M. Keyes, N. T. Kreski, and G. Li, "Changes in mental health and substance use among US adolescents during the COVID-19 pandemic," *JAMA Pediatrics*, vol. 176, no. 12, pp. 1226–1234, 2022. doi: 10.1001/jamapediatrics.2022.3737. | “Panchal, N. (2024, February 6). Recent trends in mental health and substance use concerns among adolescents. Kaiser Family Foundation. https://www.kff.org/mental-health/issue-brief/recent-trends-in-mental-health-and-substance-use-concerns-among-adolescents/. “ | ||
371. W. Liu, Y. Zhang, and H. Li, "Chronic stress and inflammation: The role of cytokines in metabolic diseases," *Journal of Clinical Investigation*, vol. 130, no. 10, pp. 5123–5135, 2020. doi: 10.1172/JCI139553. | “Mehdi, S., Wani, S. U. D., Krishna, K. L., Kinattingal, N., & Roohi, T. F. (2023). A review on linking stress, depression, and insulin resistance via lowgrade chronic inflammation. Biochemistry and Biophysics Reports, 36, 101571.” | ||
420. Farber, H. J., Wang, G., Guerra, J., & Tsao, K. (2017). Overprescribing of Oral Corticosteroids for Children With Asthma. Pediatrics, 139 (Supplement 1), S58–S59. | “Farber, H. J., Silveira, E. A., Vicere, D. R., Kothari, V. D., & Giardino, A. P. (2017). Oral Corticosteroid Prescribing for Children With Asthma in a Medicaid Managed Care Program. Pediatrics, 139(5), e20164146. https://doi.org/10.1542/peds.2016-4146.” | ||
508. Lean, M. E. J., et al. (2023). Management of Type 2 Diabetes: ADA Standards of Medical Care in Diabetes. Diabetes Care, 46(Suppl 1), S1-S292. | “508 Woolston, C. (2019, March 6). The war on ‘prediabetes’ could be a boon for pharma—but is it good medicine? Science. https://www.science.org/content/article/war-prediabetes-could-be-boon-pharma-it-good-medicine.” | ||
520. Mintzes, B. (2005). Direct-to-consumer advertising of prescription drugs: implications for the patient-physician relationship. The Lancet, 366(9487), 789-791. | “Kravitz, R. L. (2000). Direct-to-consumer advertising of prescription drugs: Implications for the patient-physician relationship. JAMA, 284(17), 2244-2244. doi:10.1001/jama.284.17.2244-JMS1101-5-1. “ | ||
521. Shah, M. B., et al. (2008). Direct-to-consumer advertising and the rise in ADHD medication use among children. Pediatrics, 122(5), e1055-e1060. | “Schwarz, A. (2013, December 14). The Selling of Attention Deficit Disorder. The New York Times. https://www.nytimes.com/2013/12/15/health/the-selling-of-attention-deficit-disorder.html.” | ||
522. Findling, R. L., et al. (2009). Direct-to-consumer advertising of psychotropic medications for youth: A growing concern. Journal of Child and Adolescent Psychopharmacology, 19(5), 487-492. | “Thomas, C. P., Conrad, P., Casler, R., & Goodman, E. (2006). Trends in the Use of Psychotropic Medications Among Adolescents, 1994 to 2001. Psychiatric Services, 57(1), 63–69. https://doi.org/10.1176/appi.ps.57.1.63.” |
We believe this was the most comprehensive list of fabricated citations available in late May. However, we weren't certain it was an exhaustive list, so we decided to check our results against updates to the report published by the White House between Thursday and Friday.
After NOTUS broke the story by identifying several fake citations in the MAHA report on Thursday morning, the White House issued a series of updated versions throughout the afternoon and evening. These changes to “formatting issues” (as White House press secretary Karoline Leavitt called them) fixed or replaced several of the fabricated sources and modified many others. By comparing the version of the report we’d analyzed on Wednesday with the version published on the White House website as of Friday, we were able to identify 13 more sources that, while real, had significant errors We re-analyzed the corrected report and found that most of the errors corrected by HHS would have been caught in advance by GPTZero.
Hallucinated Sources Revised by Friday May 30th
MAHA Report Citation (May 28) | Bibliography Scan Result | Issue | Revised MAHA Report Citation (May 30) |
337. R. M. Barros, E. J. Silver, and R. E. K. Stein, "Recess, physical education, and elementary school student outcomes," Economics of Education Review, vol. 30, no. 6, pp. 1358–1364, 2011. | Inaccurate authors, issue. (Mentioned by WP) | Changed to “Barros, R. M., Silver, E. J., & Stein, R. E. (2009). School recess and group classroom behavior. Pediatrics, 123(2), 431–436. https://doi.org/10.1542/peds.2007-2825” | |
343. A. Mlynek, "How much recess should kids get?," U.S. News & World Report, Oct. 14, 2022. [Online]. Available: https://www.usnews.com/education/k12/articles/how-much-recess-should-kids-get | Inaccurate author name. (Mentioned by WP) | Author name was changed to “Rix, K” | |
344. J. McGinn, "Recess guidelines vary greatly from state to state," The Hechinger Report, May 12, 2022. [Online]. Available: https://hechingerreport.org/recess-guidelines-vary-greatly-from-state-to-state/ | Inaccurate author name and broken link. | Author name was changed to “Mader, Jackie.” Link was changed | |
351. N. Yuan, "Among teens, sleep deprivation an epidemic," Stanford Medicine News Center, Oct. 8, 2015. [Online]. Available: https://med.stanford.edu/news/all-news/2015/10/among-teens-sleep-deprivation-an-epidemic.html | Inaccurate author name. | Author name was changed to “Richter, R.” | |
[361] T. Kanagasabai, R. Dhanoa, J. L. Kuk, and C. I. Ardern, "Inflammation, oxidative stress, and antioxidant micronutrients as mediators of the relationship between sleep, insulin sensitivity, and glycosylated hemoglobin," Frontiers in Endocrinology, vol. 13, p. 897784, 2022. | Inaccurate coauthors, journal, and volume | Changed to “361. Kanagasabai, T., Riddell, M. C., & Ardern, C. I. (2022). Inflammation, oxidative stress, and antioxidant micronutrients as mediators of the relationship between sleep, insulin sensitivity, and glycosylated hemoglobin. Frontiers in Public Health, 10, Article 888331.” | |
362. M. G. Figueiro, B. Wood, B. Plitnick, and M. S. Rea, "The impact of light from computer monitors on melatonin levels in college students and children," *Pediatrics*, vol. 136, no. 4, pp. e916–e923, 2015. | Inaccurate journal, title, year, and page numbers. | Changed to different article: “362 Lund, L., Sølvhøj, I. N., Danielsen, D., & Andersen, S. (2021). Electronic media use and sleep in children and adolescents in western countries: a systematic review. BMC public health, 21, 1-14. | |
378. Farrell, A. K., Li, Y., & Vitoroulis, I. (2023). Loneliness and well-being in children and adolescents during the COVID-19 pandemic: A systematic review. Frontiers in Psychiatry, 14, 1087713. | Some inaccurate authors and wrong journal. | Changed to “378 Farrell AH, Vitoroulis I, Eriksson M, Vaillancourt T. Loneliness and Well-Being in Children and Adolescents during the COVID-19 Pandemic: A Systematic Review. Children (Basel). 2023 Jan 31;10(2):279. doi: 10.3390/children10020279. PMID: 36832408; PMCID: PMC9955087.” | |
380. Qualter, P., Brown, S. L., Munn, P., & Rotenberg, K. J. (2010). Childhood loneliness as a predictor of adolescent depression: A longitudinal study. Journal of Child Psychology and Psychiatry, 51(7), 803–811. | Inaccurate journal, volume, and pages. | Changed to “Qualter P, Brown SL, Munn P, Rotenberg KJ. Childhood loneliness as a predictor of adolescent depressive symptoms: an 8-year longitudinal study. Eur Child Adolesc Psychiatry. 2010 Jun;19(6):493-501.” | |
398. Bozaslan, H., & Karakaş, G. (2023). The effect of exposure to nature on children’s psychological well-being: A systematic review of the literature. Urban Forestry & Urban Greening, 81, 127856. | Inaccurate Authors | Authors changed to “Liu, J., & Green, RJ.” | |
407. Frances, A., & Batstra, L. (2013, August 20). Why are so many children being diagnosed with mental disorders? Psychiatric Times. https://www.psychiatrictimes.com/view/why-are-so-many-children-being-diagnosed-mental-disorders. | Inaccurate title, journal, and link | Changed to “407 Frances, A., & Batstra, L. (2013). Why so many epidemics of childhood mental disorder?. Journal of developmental and behavioral pediatrics :JDBP, 34(4), 291–292. https://doi.org/10.1097/DBP.0b013e31829425f5.” | |
412. Piper, B. J., Ogden, C. L., Simoyan, O. M., Chung, W., & Kim, M. (2018). Trends in use of prescription stimulants in the United States and Territories, 2006 to 2016. PLOS ONE, 13(11), e0206100. | Missing authors and one wrong author (Kim) | Changed to “412 Piper, B. J., Ogden, C. L., Simoyan, O. M., Chung, D. Y., Caggiano, J. F., Nichols, S. D., & McCall, K. L. (2018). Trends in use of prescription stimulants in the United States and Territories, 2006 to 2016. PloS one, 13(11), e0206100.” | |
495. Pradhan R, (2025). More than two-thirds of patient advocacy groups studied received funding from drug, medical device companies. KFF Health News. https://kffhealthnews.org/news/article/health-202-pharma-money-patient-advocacy-groups-public-citizen/. | Inaccurate title. | Title changed to “Millions of Dollars Flow From Pharma to Patient Advocacy Groups” | |
514 Goodnough, A. (2012, July 2). Glaxo agrees to pay $3 billion in fraud settlement. The New York Times. https://www.nytimes.com/2012/07/03/business/glaxosmithkline-agrees-to-pay-3-billion-in-fraud-settlement.html. | Inaccurate authors. | Authors’ names were changed to “Thomas, K., & Schmidt, M. S.” |
Several other garbled sources were identified by journalists but not actually fixed by the White House. These included an article (Woloshin & Schwartz, 2008) that was falsely attributed to The New England Journal of Medicine, a publication which RFK Jr. had recently called “corrupt.”
Garbled Sources Not Fixed by Friday May 30th
MAHA Report Citation (May 28) | Bibliography Scan Result | Key Media Mentions | Revised MAHA Report Citation (May 30) |
340. S. Spiegel, "How much recess should kids get?," U.S. News & World Report, 2022. [Online]. Available: https://www.usnews.com/education/k12/articles/how-much-rece | No change | ||
519. Woloshin, S., & Schwartz, L. M. (2008). Giving legs to restless legs: A case study of how the media helps make people sick. The New England Journal of Medicine, 358(8), 839-841. | No Change |
Balancing Accuracy with Fairness
As seen above, Bibliography Scan tended to green-flag sources with garbled citations, even if significant components (like author(s), publication, or even title) were inaccurate. This is because Bibliography Scan is engineered to avoid false positives by allowing users to exercise their own discretion over difficult cases.
For example, consider the tool’s evaluation of MAHA report footnote 362:
Make no mistake, this citation creates a conundrum for both humans and AI evaluators. While the authors and title generally match a real article, the publication, year, and page numbers do not. Does this mean the citation is an AI amalgamation cobbled together from the components of real scholarship or a careless human error? In this case, the former is more likely, but the latter cannot be fully disproved.
To further complicate matters, the MAHA report citation adds the phrase “and children” to the end of the article title and is linked to a claim about the impact of childhood screen time. However, the real study focused on college students, a contradiction that has caused the lead author, Mariana G. Figueiro, to argue that the report mischaracterizes her research.
In this case, Bibliography Scan green-flagged the citation because it had successfully identified Figueiro’s article in PubMed’s online database. It also identified important discrepancies between the citation and the matched source, indicating significant errors in components including the citation title, publication, and date. Finally, Bibliography Scan determined (accurately) that the cited article discussed melatonin production in young adults, not children, and therefore had a “neutral” stance to the claim.
By Friday, Figueiro et al. (2015) had been replaced by a different source that actually discussed the impact of screen time on children.
Making Sources Trustworthy Again
We’ve already significantly enhanced Bibliography Scan since our experiment with the MAHA report in May, making the tool significantly faster, more intuitive, and more accurate. As stories of AI-generated citations continue to multiply, we hope journalists, researchers, analysts and educators will turn to GPTZero to rapidly surface and scrutinize suspicious sources. Our goal is to save hours of laborious fact-checking, strengthen transparency, and preserve truth on the internet—one citation at a time.
For more information on Bibliography Scan, or to try the tool, visit https://gptzero.me/sources or reach out to our sales team at team@gptzero.me.