TrustCyber / machine
/insights/canadian-ai-procurement-aia/index.mdtext/markdown7.2 KB ~1,832 tokens
Open raw

The AIA Starts Before the AI Contract Is Signed

For covered federal decisions, procurement must secure the information and rights needed to complete and maintain the impact assessment.

Canonical HTML page

An assessment question can become a procurement problem surprisingly quickly. If the buyer needs to explain how a system was tested for unfair impacts, but the supplier will provide only a marketing summary, the missing evidence cannot be repaired by writing a more confident answer in the form.

Canada’s Directive on Automated Decision-Making makes the Algorithmic Impact Assessment part of the governance of covered federal systems. The procurement implication is practical: identify the information and review rights the buyer will need while there is still an opportunity to secure them.

First establish whether the directive applies

The directive concerns production systems used to make an administrative decision or a related assessment about a client; systems used solely for research, experimentation or testing are excluded (sections 5.1–5.2). Applicability also depends on institutional provisions in section 8 and timing provisions in section 1. Provincial and municipal buyers must examine their own instruments; federal requirements are not a universal Canadian AI procurement rule.

For a covered system, section 6.1.1 requires completion, approval and publication of the final AIA before production, subject to the institutional provisions. Section 6.1.3 requires scheduled review, approval and updates, including when functionality or scope changes. The responsible institution needs a way to obtain the supplier information those duties require.

Translate the assessment into procurement deliverables

Need to establishSupplier contributionBuyer responsibility
What role automation playsWorkflow and component documentationConfirm how the actual service uses the output
How performance was testedMethods, results and known limitationsAssess relevance to the intended population and use
How decisions can be examinedVersion information and accessible recordsVerify a sample decision can be reconstructed
What changed after acceptanceChange notices and updated documentationReview the assessment and affected controls

A completed supplier-dependency record

Hypothetical record DEP-01 concerns a federal department procuring software that recommends eligibility decisions about clients, with applicability already confirmed. This is a separate use from the internal-report example. The delivery and acceptance terms below are proposed procurement mechanisms, not prescribed contract wording.

DEP-01 — evidence needed for the institution’s assessment

Record fieldCompleted example
Assessment issueCan the institution explain the relevance and limitations of testing for accuracy and unfair impacts on its intended client population?
Supplier-controlled evidenceVersioned model/build manifest V-3 and test methods/results TEST-3, including population coverage and known limitations. Received: aggregate results and a build name; missing: coverage breakdown and reproducible version reference.
Buyer owner and delivery pointProgram director owns assessment completeness; evaluation lead reviews evidence. Procurement lead makes the missing materials due before acceptance review and before the final AIA approval/publication required for production.
DecisionHold production acceptance. Supplier must provide TEST-3 detail and V-3 identification under agreed review/access safeguards; evaluation lead must determine whether further institution-specific testing is needed.
Refresh dependencyRecord supplier notice and refreshed evidence for functionality/scope changes, plus the institution’s scheduled review. The responsible federal official retains the AIA review, approval and update duty.

Acceptance should include a traceable decision

Select a representative case and follow it from input to outcome. Identify the software and model versions, the information used, the rules applied, and any human intervention. Examine whether the records support the explanation the institution needs to give. A dashboard showing a success rate cannot answer all of those questions.

The directive includes testing for accuracy, unintended bias, and unfair impacts, together with ongoing outcome monitoring. Procurement teams should work with service, policy, privacy, security, and legal owners to determine the test access and documentation needed for their system. The supplier’s own benchmark may inform that work without being sufficient for the institution’s decision. See sections 6.3.1–6.3.3.

A contract dependency: If an assessment answer relies on evidence controlled by the supplier, identify that evidence, its delivery date, the right to review it, and the obligation to refresh it.

Write for the second version

The first accepted release will change. A supplier may replace a model, revise a decision rule, or introduce a new data source. Access to released component versions and appropriate review, testing, monitoring, and audit rights help the institution understand those changes. Confidentiality safeguards can be part of the arrangement without making the evidence inaccessible. The component-access requirements appear in sections 6.2.5–6.2.6.

Keep unresolved supplier dependencies visible before contract award. Where evidence cannot be obtained, the institution needs to decide how that affects the proposed use, procurement terms, or acceptance. The AIA is easier to maintain when the contract has already made its questions answerable.

Make supplier dependencies visible

AI governance advisory can help connect assessment evidence needs to ownership and procurement decisions.

Selected primary sources