A model can state something false with the same tone it uses for something true. That is not a glitch that a patch will eventually remove — it follows from how these systems are built, and understanding the mechanism is the only reliable way to know when to check an output before anyone relies on it.
Key takeaways
The term most people reach for is “hallucination,” and it describes something specific: an AI system generating a statement that sounds fluent and confident but is not grounded in fact. It is worth being precise about why that happens, because the fix that suits one situation — a better prompt, a bigger model, a connected search tool — does very little for another.
The Canadian Centre for Cyber Security defines generative AI plainly: it “generates new content by modelling features from large datasets that were fed into the model,” distinguishing it from older systems that only “recognize patterns or classify existing content.” (Cyber Centre, ITSAP.00.041) In practical terms, a large language model is predicting the next most statistically plausible piece of text given everything that came before it, over and over, until it has produced a full answer.
Nothing in that process checks the output against the real world as it stands today. The model is not looking anything up by default — it is continuing a pattern it learned during training, and a fluent, well-structured continuation is not the same thing as a true one.
This is the part people find counter-intuitive: a language model does not have a separate “I am not sure” mode that kicks in when it is guessing. It generates the next plausible word whether or not the underlying claim is solid, which is exactly why a fabricated statistic or a made-up citation reads in the same fluent register as a correct one.
The Cyber Centre’s own guidance names this directly among generative AI’s risks, alongside phishing, malicious code and biased content: outputs “can be incorrect,” “might not make sense” and “might not take certain factors into account.” Its instruction to users is unambiguous — “You should always be aware of and validate your sources to verify whether the content being presented is accurate.” (Cyber Centre, ITSAP.00.041) That is a federal cybersecurity agency telling Canadians, in plain language, not to take an AI output at face value.
Canada’s federal, provincial and territorial privacy commissioners tell organisations using generative AI to “evaluate the validity and reliability of the generative AI tool for the intended purpose,” adding that “tools must be accurate throughout the intended lifecycle of the tool and across the variety of circumstances in which they are used.” (OPC, generative AI principles) That guidance points, in a footnote, to the United States’ National Institute of Standards and Technology’s AI Risk Management Framework for how to think about those terms.
NIST’s own framework — a US resource, cited here only because Canada’s privacy regulators point to it — defines validity as confirmation “that the requirements for a specific intended use or application have been fulfilled,” reliability as the “ability of an item to perform as required, without failure,” and accuracy as “closeness of results of observations, computations, or estimates to the true values.” It then makes the point that matters most here: “Accuracy and robustness… can be in tension with one another.” (NIST AI RMF, AI Risks and Trustworthiness — United States) A system tuned to always produce a fluent, complete-sounding answer is, by construction, working against the goal of only ever saying things that are true.
Every model works within a fixed context window — the maximum amount of text it can hold in view at one time, covering both what a user has typed and what the model has already generated in the conversation. Once a conversation runs longer than that window, the system has to drop, compress or summarise earlier material to keep going. That is why a long back-and-forth can start contradicting something said ten minutes earlier: the detail has not been remembered, it has fallen out of view.
This is not a flaw a vendor forgot to fix. Holding an unlimited amount of text in view at full resolution is a genuine computational cost, and every generative AI product on the market manages that cost by capping how much context it keeps active. The practical consequence for a business is simple: a long, meandering conversation is a worse place to rely on the model remembering a fact than a short, freshly stated one.
Ask a generative AI tool the identical question twice and it can produce two different answers, sometimes with different numbers or a different conclusion. This is by design, not by accident: most systems introduce a controlled amount of randomness when choosing among several plausible next words, rather than always picking the single most likely one. That variation is part of what makes the output read as natural language instead of a rigid, repetitive template — and it means no single run of a prompt should be treated as the tool’s one fixed answer.
Two common ways businesses try to make an AI system more reliable work very differently, and neither removes the underlying risk on its own. Fine-tuning adjusts the model’s own internal weights using additional examples, which changes its style, tone and the topics it handles well — but it does not give the model a guaranteed, checkable connection to a current fact. The model still generates from a learned pattern; the pattern has just been reshaped.
Retrieval — often called retrieval-augmented generation — works differently: at the moment of a question, the system fetches a specific document or record and hands it to the model to work from, rather than relying purely on what it absorbed during training. That materially reduces the chance of an invented fact, because the model has something concrete in front of it to draw on. It does not eliminate the risk, though, for two reasons: the retrieved document has to actually be current and correct, and the model can still misread, misquote or over-generalise from what it was given.
Suppose a business builds a customer-facing tool to answer questions about its own return policy. One version is a model fine-tuned six months ago on a snapshot of the policy at that time; the other retrieves the current policy page at the moment a customer asks. If the return window changed three months ago, the fine-tuned version has no way of knowing that — it will answer fluently and confidently from what it absorbed during training, with nothing in its output signalling that the information is stale. The retrieval-based version, by contrast, is only as good as whether it actually pulls the current page and represents it accurately, which is a much narrower thing to test and monitor than “is the model right.”
Related: disclosing that a customer is talking to an AI system, and what happens when a chatbot promises something the business did not intend.
What a check step around this looks like once a system is running day to day is covered on the AI operations hub.
Not in the sense of a one-time patch. Canada’s Cyber Centre lists misinformation and disinformation among generative AI’s standing risks, not among its temporary defects, and its guidance to users — validate the sources yourself — is written as an ongoing practice, not a stopgap for an earlier product version.
It reduces the risk considerably when the retrieved source is itself current and accurate, because the model is working from a concrete document rather than a learned pattern alone. It does not eliminate the risk, because the model can still misread or overstate what that document actually says.
Most generative AI systems introduce a controlled amount of randomness when selecting among several plausible next words, so that the output reads as natural language rather than a fixed template. One consequence is that no single answer should be treated as the tool’s one true answer to a question.
A short call is enough to talk through where an unchecked output could actually reach a customer.
