“Explainable AI” gets sold as a single feature a system either has or lacks. It is actually a narrower, more specific claim than that — and Canada’s one government rule that spells out what an explanation has to contain draws the boundary sharply: an explanation of how a decision was reached is not, and was never meant to be, proof that the decision was right.
Key takeaways
Strip the marketing language away and “explainable AI” is a claim about traceability: that a system’s output can be connected back to the inputs, rules, or training patterns that produced it, in language a person can follow. That is a genuinely useful property. It lets a reviewer see which factors a model weighted, spot an input it should not have used, and challenge a specific step rather than the output as a whole. None of that is the same claim as “the output is correct,” and conflating the two is where explainability gets oversold.
Canada has no general law requiring a business to explain an AI-assisted decision to a customer. The closest thing to a worked example is the federal government’s own rulebook for itself: the Treasury Board’s Directive on Automated Decision-Making, which binds federal departments using an automated decision system to make an administrative decision about a client. It treats an explanation as two separate obligations, not one. Before a decision is made, the department must provide notice, “through all service delivery channels in use,” that the decision will be made or assisted by an automated decision system, delivered “prominently and in plain language.” After the decision, it must provide clients “a meaningful explanation … of how and why the decision was made,” with the depth of that explanation set by an Algorithmic Impact Assessment the department must complete and publish “prior to the production of any automated decision system,” then review and update “on a scheduled basis, including when the functionality or scope of the automated decision system changes.”
Read the scope carefully: this binds federal government departments making decisions about clients, not private Canadian businesses. No equivalent statute currently reaches an ordinary company’s use of AI. What the directive is worth to a business is as the most detailed template of what a real explanation obligation looks like in practice — notice first, a structured explanation after, and a documented assessment behind both — should a business choose to hold itself to a similar standard, or should a regulator eventually ask for one.
This is the distinction that matters, and it comes from a United States technical standards body rather than a Canadian one, so the source needs to be labelled plainly: the Office of the Privacy Commissioner’s own principles for generative AI point, in a footnote to their necessity principle, to the National Institute of Standards and Technology’s AI Risk Management Framework for more on “validity and reliability in AI systems.” That page defines accuracy as “closeness of results of observations, computations, or estimates to the true values or the values accepted as being true,” and it says plainly that “accuracy and robustness contribute to the validity and trustworthiness of AI systems, and can be in tension with one another.” A system can trace its own reasoning path in detail — a fully transparent chain from input to output — and still be reasoning from the wrong facts, an out-of-date policy, or a pattern that does not hold in this particular case. Explainability describes the path. It says nothing about whether the destination is correct, which is exactly the gap a confident-sounding AI answer trades on.
For businesses outside government, the nearest thing Canada has to an explainability norm is ISED’s Voluntary Code of Conduct on advanced generative AI systems. Its Transparency outcome commits signatories to publish “sufficient information … to allow consumers to make informed decisions and for experts to evaluate whether risks have been adequately addressed,” and its Accountability outcome expects organizations to “understand their role with regard to the systems they develop or manage, put in place appropriate risk management systems, and share information with other organizations as needed to avoid gaps.” Two limits matter here. First, it is voluntary and binds only its signatories — a list of roughly four dozen organizations, mostly technology vendors and large institutions, not a sweep of the Canadian economy. Second, the code says outright that it “does not in any way change existing legal obligations that organizations may have – for example, under the Personal Information Protection and Electronic Documents Act.” Signing it adds a transparency commitment on top of PIPEDA; it does not replace anything a business already owes under privacy law.
The honest use case for explainability is not proving a system is right — it is making a system reviewable by someone other than the person who built it. The Federal Court’s own interim principles for its use of AI commit the Court to “authorize external audits of any AI-assisted data processing methods that it embraces” under a principle it labels Transparency — not because an audit proves the system was right on any given file, but because an auditable system is one a third party can actually check. That is the same logic a diligence review applies to a target business’ AI tools: someone still has to sign off on what the system produced, and an explainable system is the one that makes that sign-off a real check rather than a formality.
A quick test for your own tools
If a system flagged, scored, or drafted something today, could a colleague who did not build the system point to the specific inputs that drove the result? If the honest answer is “no, we would just have to trust it,” that is a gap worth closing before the output touches a decision that matters — not because explainability makes the output correct, but because it makes the output checkable.
No. Explainability and accuracy are separate properties. NIST’s own AI Risk Management Framework states that accuracy and robustness “can be in tension with one another,” and a system can trace its reasoning in full detail while still reasoning from a wrong or outdated input.
Not under any general law currently in force. The Treasury Board’s Directive on Automated Decision-Making requires federal departments to give a meaningful explanation after a decision, but it does not extend to private businesses. ISED’s Voluntary Code asks signatories to be transparent; it is voluntary and binds only the organizations that signed it.
As Canadian federal policy uses the terms, transparency is about what an organization discloses before and around a system’s use — that AI is involved at all, and enough about how it works for an outside expert to assess the risks. Explainability is narrower: it is the ability to trace a specific output back to the specific inputs that produced it.
Related: who is accountable when AI decides, and why AI confidence is not accuracy.
A short call is enough to map which of your tools could survive an outside review.
