It's easy to list AI risks in the abstract. It's more useful to look at what has actually gone wrong, in named, documented cases — because the mechanism, the warning sign, and the fix look different every time, and none of them are prevented by the same single control.
Key takeaways
In July 2025, researchers documented a data-poisoning exploit in Microsoft 365 Copilot and similar retrieval-augmented generation (RAG) systems, in which threat actors injected poisoned documents to manipulate the AI's outputs on an ongoing basis, not just once (Cyber Centre, Top 10 artificial intelligence security actions, ITSAP.10.049). Why it failed: a RAG system trusts the documents it retrieves as source material by design — there was no ongoing check that retrieved content hadn't been tampered with. Who notices: nobody, until outputs drift far enough from expected that someone investigates — the exploit is built to be persistent and quiet, the opposite of a crash. The fallback: the Cyber Centre's own lesson from the case is that continuous testing and evolving guardrails have to be ongoing, not a one-time pre-launch check — the mechanics of the injection itself are covered in how hidden text hijacks an AI.
In 2024, security researchers working with Wiz and Hugging Face uncovered a risk in which malicious actors could upload poisoned data to Hugging Face's public dataset repositories, threatening the AI pipelines of every organization pulling models or data from them (Cyber Centre, ITSAP.10.049). Why it failed: weak tracking of data provenance and no anomaly detection on ingested data — the platform's openness, which is also its main value, was the same thing that let tampered data in. Who notices: whoever eventually audits training data against its original source, which is not a step every organization pulling from a public repository actually takes. The fallback: tracking where a dataset actually came from, curating versioned datasets rather than pulling the latest copy blind, and running anomaly detection on data before it's used to train or fine-tune anything.
In 2025, a documented case saw an HR technology company face reputational damage and legal scrutiny after removing human review from its AI-driven candidate screening process; unchecked algorithmic bias was allowed to influence hiring decisions as a result. The organization's response was to reinstate human-in-the-loop controls, add execution checks, and establish auditable decision trails so outcomes could be monitored and corrected (Cyber Centre, ITSAP.10.049). Why it failed: the failure wasn't the model existing — it was removing the human checkpoint that would have caught a biased pattern before it affected real candidates. Who notices: in this case, only after external scrutiny — the mechanism, covered in more depth in how bias gets into an AI system, doesn't announce itself the way a system crash does. The fallback: explainability tools, an auditable decision trail, and a genuine human checkpoint — not a rubber-stamp one — on the decisions that actually affect people.
Canada's federal banking regulator, the Office of the Superintendent of Financial Institutions (OSFI), has “identified AI model risk as a growing supervisory concern, particularly risks arising from data quality issues, model drift, lack of transparency, bias, and overreliance on automated outputs,” warning that “poorly governed AI systems can produce unreliable or unexpected results, potentially leading to operational disruption, financial loss, legal exposure, and reputational harm if not actively monitored and controlled” (Cyber Centre, ITSAP.10.049, citing OSFI). Why it fails: a model that performed well at launch can drift as real-world conditions change, and nothing about “still running” signals that it has drifted out of validated bounds. Who notices: only whoever is running continuous performance monitoring — if nobody is, the answer is nobody, until the downstream consequence shows up somewhere else entirely. The fallback: OSFI's own remedy, as described by the Cyber Centre, is continuous monitoring, explainability, human oversight for critical decisions, and clear fallback procedures — the same test covered from the trustworthiness angle in how to tell if an AI output is trustworthy.
A separate and increasingly common category doesn't involve the AI system failing on its own — it involves the AI system being the attacker's tool. Deepfake impersonation is one documented route: the Cyber Centre's guidance references a widely reported case in which a company lost roughly $25 million (U.S.) to a deepfake-enabled fraud, cited via the World Economic Forum, alongside its own guidance to deploy media authenticity checks and phishing-resistant identity verification (Cyber Centre, ITSAP.10.049). That case is not Canadian, and is cited here as a documented example of the mechanism rather than a Canadian statistic. Why it works: a convincing synthetic voice or video defeats the specific human habit — recognizing a colleague's voice or face — that most organizations rely on informally as their real authentication step for unusual requests. Who notices: whoever is trained to expect an out-of-band verification step regardless of how convincing the request sounds or looks. The fallback: a verification channel that doesn't depend on recognizing a voice or face at all — a callback to a known number, a separate approval system — because the whole point of the attack is that the human recognition step can no longer be trusted on its own.
None of the five was prevented by a single control. Each needed a different specific layer — continuous testing, data provenance tracking, a genuine human checkpoint, ongoing performance monitoring, or an out-of-band verification step — and each layer would have done nothing against the other four failures. The practical lesson is not “pick the right control” — it's that a single control never covers AI risk, because the five documented failures above didn't share a mechanism.
A mix. The organization behind the reporting — Canada's Cyber Centre — is Canadian, and OSFI's warning is a Canadian federal regulator's own statement. The specific named incidents (GitHub Copilot, Microsoft 365 Copilot, Hugging Face, the deepfake fraud case) are global, and are labelled as such rather than presented as Canadian statistics.
No. Each failure had a distinct mechanism and a distinct fix — continuous retrieval-content testing, data provenance tracking, human-in-the-loop review, ongoing drift monitoring, and out-of-band identity verification respectively. None of the five fixes would have prevented a different one of the five failures.
That's a broader question than these five cases answer on their own. What they consistently show is that unmonitored, unreviewed AI use carries these specific documented risks — not that no mitigation exists, since each case also names the specific control that was found to be missing.
Building the ongoing monitoring, review and verification layers these five failures point to is exactly the day-to-day work of running an AI system safely after launch.
