A handful of confident claims about AI keep circulating in boardrooms and sales decks. Most of them do not survive contact with what Canada’s own cyber security and privacy guidance actually says about how the technology behaves.
Key takeaways
A generative system produces its answer by generating new content by modelling features from large datasets that were fed into the model, as Canada’s Cyber Centre puts it, describing the mechanism plainly rather than crediting it with comprehension. Modelling a feature of a dataset is a statistical operation: the system has learned which words, shapes or sounds tend to follow which others. That is enough to produce fluent, well-structured output. It is not the same operation as knowing what the output means, and nothing in how these systems are built gives them a way to check their own answer against the world the way a person checking their own memory can.
The practical test is asking what happens when the pattern runs out. A person who does not know an answer generally has some sense that they do not know it. A generative system has no equivalent internal signal by default — it will complete a sentence about a topic it has almost no training signal on with the same fluent register it uses for a topic it has seen thousands of times, because fluency and coverage are produced by the same mechanism and do not vary independently the way confidence and knowledge do in a person.
Confidence and correctness are produced by different parts of the process. The Canadian Centre for Cyber Security is explicit about this: outputs “can be incorrect”, “might not make sense”, “might not take certain factors into account”, and “can be biased.” None of those failure modes come with a built-in tell — a wrong answer is written in exactly the same fluent register as a right one. The Centre’s own instruction is the practical fix: you should always be aware of and validate your sources to verify whether the content being presented is accurate.
This is a genuinely counterintuitive habit to build, because everything about how people normally judge writing runs the other way: hedged, qualified prose usually signals uncertainty, and clean, declarative prose usually signals someone who checked their facts. A generative system's writing style tracks neither of those things reliably, which is exactly why the instruction above has to be a fixed habit — check the source, every time — rather than a judgment call made case by case based on how the answer reads.
The math is neutral; the data it was fitted to is not. Canada’s privacy commissioners describe the mechanism directly: developing a generative system means evaluating the training data sets to ensure that they do not replicate, entrench, or amplify historical or present biases – or introduce new biases. A model trained mostly on one kind of data, one language register, or one demographic’s writing will reproduce that skew as fluently as it reproduces anything else, and it will not flag the skew on its own.
This is why “it’s just doing the math” is not a reassurance. The math is exactly the mechanism by which a skew in the input becomes a skew in the output, applied consistently and without the kind of self-correction a person might apply on noticing their own assumption looked off.
Working and appropriate are different tests. Canada’s federal-provincial-territorial privacy principles for generative AI set out a category of use that a business should rule out before technical performance is even considered: developers and deployers should not put a system into service where it falls into what the principles call a “no-go zone” — profiling that may lead to unfair, unethical, or discriminatory treatment, or creating outputs that threaten fundamental rights and freedoms. A tool can perform its narrow technical task well — sorting, scoring, drafting — and still be the wrong thing to deploy against a specific purpose, which is a judgment a working demo cannot make for you.
The most recent Statistics Canada survey found the opposite of a stampede: 14.5% of businesses planned to use AI over the next 12 months, while two-thirds — 66.7% — reported no plans at all, and 18.9% were uncertain. Among those with no plans, the leading reason was not fear or lack of budget: 78.1% said AI simply was not relevant to the goods or services they currently provide. A business that has not adopted AI is, statistically, the median Canadian business, not a laggard.
The same StatCan survey found the pattern holds by firm size, too: businesses with 100 or more employees were more likely to plan to adopt AI (20.5%) than smaller ones — 15.0% among those with 20 to 99 employees, 14.4% among those with 5 to 19, and 14.2% among the smallest, with 1 to 4 employees (Statistics Canada, Canadian Survey on Business Conditions, third quarter of 2025). A business that has not adopted AI is the median case at nearly every size StatCan measures, not an outlier at any of them.
A model that tops a leaderboard was tested against someone else’s benchmark, on someone else’s data, for a task that may not resemble the one you need done. A bigger model is often also a slower and more expensive one to run at volume, which is a real cost even when the accuracy difference on your specific task turns out to be small or nonexistent. The mechanics of why a benchmark score can mislead — and what to ask for instead — are worked through in how AI benchmarks mislead.
A fifteen-minute demonstration is built, understandably, around the input that makes the product look its best. That tells you almost nothing about how it behaves on the messy file, the ambiguous email or the edge case your business actually generates in volume. What an AI demo hides covers the structural reasons a demo cannot substitute for an evaluation on your own material.
Taken together, these six myths point at one underlying habit: treat a claim about AI the same way you would treat a claim about anything else a vendor is selling you — ask what evidence supports it, whose test produced it, and what it would look like if it were wrong.
See also how AI benchmarks mislead and what an AI demo hides.
No. A search engine retrieves and ranks documents that already exist. A generative system produces new text one token at a time, which is why it can answer a question no page has ever answered — including with a wrong answer that no page ever contained.
Popularity tells you the tool works for whatever the majority of its customers use it for. It does not tell you it was tested against your documents, your terminology, or your failure tolerance — that is a separate question a business has to answer for itself.
Not by itself. The label covers everything from a hand-coded rules engine to a large language model, and those behave very differently. Ask which layer of the technology is actually doing the work before treating the label as a specification.
Myth-checking is the first filter, not the whole evaluation. For how to actually scope and test whether an AI capability is worth committing to, see the three questions that predict whether an AI pilot will scale.
