The task of fact-checking an all-knowing AI is falling to a workforce that is often overwhelmed, under-supported, and, in many cases, unqualified for the job. Insiders who rate and train AI models for a major tech company reveal that they are consistently asked to verify information on subjects they know nothing about, a practice that directly threatens the AI’s accuracy and reliability.
A core tenet of the job is to rate AI responses based on “factuality” and “groundedness”—whether the information is true and cites accurate sources. However, this principle is undermined by the realities of the workflow. Raters report being given tasks on highly specialized topics like advanced mathematics or medical treatments and being explicitly told not to skip them for lack of domain expertise.
This systemic flaw means that the AI’s knowledge base is being curated by non-experts. A person with a background in creative writing could be the final arbiter on the correctness of a chemotherapy regimen, a deeply unsettling prospect. The worker who faced this exact task said it “haunted” her, knowing that a real person might one day rely on the information she was forced to edit.
The recent public failures of AI, like generating bizarre and dangerous advice, are a direct reflection of this internal crisis. The human safety net is stretched too thin, staffed by people who are not equipped to catch every error. The result is an AI that can project confidence while being utterly, and sometimes dangerously, wrong.