“Safe” is not a marketing claim a vendor gets to make on a business’s behalf. Under Canadian privacy law, whether customer data is protected once it enters an AI product is measured against a specific legal standard, and against a short, named list of risks the federal government’s own cybersecurity agency has published for exactly this kind of tool.
Key takeaways
PIPEDA’s Safeguards principle states it in one sentence: “Personal information shall be protected by security safeguards appropriate to the sensitivity of the information.” The clauses underneath name the three kinds of measure a business is expected to have: “(a) physical measures, for example, locked filing cabinets and restricted access to offices; (b) organizational measures, for example, security clearances and limiting access on a ‘need-to-know’ basis; and (c) technological measures, for example, the use of passwords and encryption.” (PIPEDA, Schedule 1, clause 4.7) An AI vendor’s contract and access controls are, functionally, an extension of the client organization’s own technological and organizational safeguards — not a separate system the standard stops applying to.
The Canadian Centre for Cyber Security’s awareness guidance on generative AI lists eight named risks, and one is written for exactly this question: “Privacy of data — Users may unknowingly provide sensitive corporate data or personally identifiable information (PII) in their AI queries and prompts.” (Canadian Centre for Cyber Security, ITSAP.00.041) The mechanism is rarely a dramatic hack. It is an employee pasting a customer’s file into a general-purpose chat window to get a faster summary, without knowing where that prompt is retained, logged, or used afterward. Whether that specific tool retains prompts, and for what purpose, is the first thing a business should be able to answer about any AI product it lets staff use with real customer information.
If personal information handled through an AI tool is exposed, PIPEDA’s breach provisions apply the same way they would to any other system: “An organization shall report to the Commissioner any breach of security safeguards involving personal information under its control if it is reasonable in the circumstances to believe that the breach creates a real risk of significant harm to an individual”, and must separately notify the affected individual on the same test. (PIPEDA, s.10.1(1) & (3)) “Significant harm” is itself defined in the Act and “includes bodily harm, humiliation, damage to reputation or relationships, loss of employment, business or professional opportunities, financial loss, identity theft, negative effects on the credit record and damage to or loss of property.” (PIPEDA, s.10.1(7)) An AI vendor’s own incident notice to its customers does not discharge this obligation on the business’s behalf — the client organization still has to make its own assessment against this test.
The joint generative AI principles fold safety into the necessity-and-proportionality principle directly: organizations should “evaluate the validity and reliability of the generative AI tool for the intended purpose”, and “tools must be accurate throughout the intended lifecycle of the tool and across the variety of circumstances in which they are used.” (OPC, generative AI principles) That is a standing evaluation, not a one-time procurement checkbox — a tool that was appropriately safeguarded when it processed general enquiries is not automatically still appropriate once a business starts feeding it more sensitive records.
The Cyber Centre’s guidance names seven other risks alongside privacy of data, and two of them turn directly on questions a business can actually put to a vendor. On training data: “Threat actors can inject malicious code into the dataset used to train the generative AI system… could also increase the potential for large-scale supply-chain attacks” — a reason to ask whether a vendor trains on data pooled across its customers, and how that training pipeline is secured. On the business’s own exposure: the guidance warns generally that outputs “can be incorrect” and that a user “should always be aware of and validate your sources to verify whether the content being presented is accurate” — which matters for data safety specifically where an AI tool is trusted to summarise or restate a customer’s own record back to them without a human checking that the restatement is correct. (Canadian Centre for Cyber Security, ITSAP.00.041)
A dental clinic’s front-desk software adds an AI feature that drafts appointment-reminder messages from patient records. Before turning it on, the clinic asks three questions the safeguards principle and the Cyber Centre’s risk list both point to directly: does the vendor retain the prompts sent to it, and for how long; is access to the feature restricted to staff who actually need it, matching clause 4.7.3’s “need-to-know” standard; and does the vendor’s contract commit to notifying the clinic promptly if the underlying data is exposed, so the clinic can run its own s.10.1 assessment rather than finding out from a news report. A vendor that cannot answer the first question has already failed the “appropriate to the sensitivity of the information” test for patient health data, whatever else its marketing claims.
Related: what applies when an AI tool processes data outside Canada, whether PIPEDA applies when a business uses AI, and whether customer records can be used to train a model.
Encryption is one of the “technological measures” clause 4.7.3 names, but the principle requires protection “appropriate to the sensitivity of the information” overall — for sensitive records that generally also means access controls and organizational measures, not encryption alone.
Vendor reputation is not a substitute for checking the specific answers — what the vendor retains, for how long, and under what contract terms — because PIPEDA’s accountability principle keeps the client organization responsible for what happens to data “transferred to a third party for processing,” regardless of that party’s size.
PIPEDA’s breach-reporting duty turns on whether it is “reasonable in the circumstances to believe that the breach creates a real risk of significant harm,” which is a case-by-case assessment — a genuine near-miss with no retention or exposure may not meet that threshold, but the organization still has to make and document that assessment rather than assume it away.
Operations covers monitoring, vendor review and incident response once an AI tool is live in production.
