Most people picture a breach as a hack. Under Canadian privacy law, a breach is a breach of security safeguards involving personal information — and with an AI tool, the safeguard that fails is just as often a setting, a prompt, or a vendor's server as it is a stolen password.
Key takeaways
A generative AI tool is, from a privacy-law standpoint, usually a third-party processor: a business feeds it customer names, case details, transcripts or documents, and the vendor's infrastructure does something with that information. Canada's federal private-sector privacy statute, the Personal Information Protection and Electronic Documents Act (PIPEDA), answers the “whose problem is it” question directly, in Schedule 1, clause 4.1.3:
“An organization is responsible for personal information in its possession or custody, including information that has been transferred to a third party for processing. The organization shall use contractual or other means to provide a comparable level of protection while the information is being processed by a third party.” — PIPEDA, Schedule 1, clause 4.1.3
Read plainly: accountability does not transfer with the data. If an AI vendor's database, subprocessor or model configuration is the point where personal information gets exposed, the organization that collected it — not the vendor — is the one PIPEDA holds responsible for figuring out whether a breach occurred and what to do about it. A vendor contract can allocate cost and liability between the two businesses privately, but it cannot reassign a statutory duty.
PIPEDA does not require reporting every incident. Section 10.1(1) sets a specific test: an organization must report to the Privacy Commissioner “any breach of security safeguards involving personal information under its control if it is reasonable in the circumstances to believe that the breach creates a real risk of significant harm to an individual” (PIPEDA s.10.1(1)). Two things about that wording matter for an AI-specific incident. First, it is technique-agnostic — “any breach of security safeguards” covers a stolen laptop and a mis-scoped AI integration equally; the law does not ask how the data got out. Second, “significant harm” is itself defined, not left to intuition: s.10.1(7) lists “bodily harm, humiliation, damage to reputation or relationships, loss of employment, business or professional opportunities, financial loss, identity theft, negative effects on the credit record and damage to or loss of property.”
Whether a given incident clears that bar is decided by two named factors under s.10.1(8): “(a) the sensitivity of the personal information involved in the breach; (b) the probability that the personal information has been, is being or will be misused; and (c) any other prescribed factor” (PIPEDA s.10.1(8)). Both factors read differently once an AI tool is the point of exposure. Sensitivity can be higher than a business expects, because a chat-style interface invites people to type things a structured form never would — a health detail, a family situation, a financial specific — and none of it is flagged as sensitive at the point of collection, which is exactly the definitional question covered in what “personal information” means for AI. Probability of misuse depends on where the exposure actually went: contained inside a vendor's access-controlled infrastructure reads very differently from data that left the vendor relationship altogether.
Canada's Cyber Centre names “privacy of data” as one of eight risks of generative AI in its guidance ITSAP.00.041: “Users may unknowingly provide sensitive corporate data or personally identifiable information (PII) in their AI queries and prompts. Threat actors could harvest this sensitive information to impersonate individuals or spread false information.” That is the input side — information a person puts into a tool that was never meant to be a system of record for it.
The output side is newer and more active. The Office of the Privacy Commissioner's own generative-AI principles name two specific attack types under Principle 9, Safeguards, and define them in the text: “prompt injection attacks (in which carefully crafted prompts bypass filters or make the model perform unanticipated actions)” and “model inversion attacks (in which personal information contained in the model's training data is exposed)”, alongside jailbreaking, “in which privacy or security controls in the tool are overridden” (OPC, Principles for responsible, trustworthy and privacy-protective generative AI). These are not hypothetical categories. The Cyber Centre's Top 10 artificial intelligence security actions primer documents a live 2025 case in which a prompt-injection technique was used against GitHub Copilot to run code remotely, and a separate, related exploit against Microsoft 365 Copilot's document-retrieval pipeline that manipulated outputs by feeding it poisoned content — the mechanism the OPC's definitions describe, observed in production tools, not a lab demonstration. The mechanics of how a hidden instruction reaches a model in the first place are covered separately in how hidden text hijacks an AI.
PIPEDA is federal, but it is not the only private-sector privacy law in Canada, and it does not always apply. The Privacy Commissioner's own summary is direct: “Unless the personal information crosses provincial or national borders, PIPEDA does not apply to organizations that operate entirely within: Alberta, British Columbia, Quebec. These three provinces have general private-sector laws that have been deemed substantially similar to PIPEDA” (OPC, Summary of privacy laws in Canada). A business operating only within one of those three provinces answers to its own provincial Act instead — and each has its own breach-notification and enforcement mechanics, which this page does not attempt to state, because they are not PIPEDA's.
Quebec adds a further wrinkle that matters specifically for AI: where a decision about a person is based exclusively on automated processing of their personal information, Quebec's Law 25 requires the organization to tell the person, no later than when it informs them of the decision, and to give them the chance to have a staff member review it (Québec's privacy regulator (the CAI), summary of Law 25 changes). That is a Quebec-specific automated-decision rule with no federal PIPEDA equivalent — it is not the same thing as a breach-reporting duty, and the two should not be merged into a single “Canada requires” statement.
Suppose a business's customer-service chatbot logs full conversation transcripts by default. A misconfigured integration exposes those logs — including a customer's mailing address and a passing mention of a medical appointment — to an internal analytics channel with broader access than intended, for two weeks, before someone notices and locks it down. Applying the s.10.1(8) factors rather than a gut reaction: sensitivity is real (health information), but the probability-of-misuse factor is different from an external leak — the exposure stayed inside the organization's own access-controlled systems, to employees already bound by confidentiality obligations, and no evidence points to actual misuse. Whether that combination clears “a real risk of significant harm” is a judgment the organization has to document against the statutory factors, not guess at — and documenting that reasoning is itself part of the accountability principle.
Ontario businesses working through the mechanics of notification once the test is met — timing, content of the notice, recordkeeping — can start from the general PIPEDA breach process, which this page does not restate (treadstonelaw.ca, data breach notification rules for Ontario businesses). It is written for breaches generally, not for AI tools specifically, but the underlying reporting duty is the same one described above.
If it results in personal information under the organization's control being exposed, yes — section 10.1(1) covers “any breach of security safeguards,” and the statute does not carve out an exception for the technique used. The relevant question is not how the exposure happened but whether it creates a real risk of significant harm under the s.10.1(8) factors.
Not for the statutory duty to assess and report. PIPEDA Schedule 1 clause 4.1.3 makes the collecting organization responsible for personal information “including information that has been transferred to a third party for processing.” A contract can allocate financial liability between the business and the vendor, but it cannot reassign who owes the Privacy Commissioner a report.
Only if the personal information crosses a provincial or national border, or the business is federally regulated. An organization operating entirely within one of those three provinces is generally governed by that province's own private-sector Act instead, per the Privacy Commissioner's own summary of which law applies where.
Deciding what an AI tool is allowed to log, who can see it, and how an incident gets caught before it runs for weeks is an operational design question, not just a legal one.
