Xcelerate
Menu
Get in touch
All articles on ai integration

AI integration

An AI supplier scorecard for New Zealand procurement

Compare AI suppliers on the same New Zealand service task. Score reviewer effort, serious errors, running costs and evidence buyers can check at handover.

Evaluators compare supplier scores in a software evaluation matrix.
AI-generated editorial image.

Two suppliers demonstrate an AI tool for classifying maintenance requests. One produces attractive summaries. The other shows how a planner can correct a routing error. Which demonstration answers the brief? A buyer can make the comparison fairer by specifying the decision, the evidence each supplier receives and the errors that would prevent rollout.

What goes into a shared evaluation pack?

Choose work examples that reflect the spread of requests. Include short messages, a job with several issues, a message mentioning two locations and one outside the maintenance team's remit. Invent names and details for the assessment. Have experienced planners mark the acceptable destination, alternatives and cases where a supplier should ask a person.

Give bidders the field definitions, permitted destinations and scoring rules. Keep a separate set of equivalent examples for the scored exercise. Use the same instructions and timing for each bidder. NZ Digital Government’s GenAI procurement guidance recommends explicit evaluation criteria and exit planning. Recording those decisions before demonstrations makes the eventual purchase easier to explain.

Choose scoring examples with the service team rather than relying on a vendor’s sample library. In maintenance triage, a planner might accept either of two destinations for the same routine job if both teams share the queue. Mark both as acceptable before comparing suppliers. Otherwise the score rewards a guess about local conventions.

How do you score a costly mistake?

Distinguish an error that needs a quick edit from one that could leave an urgent request in an ordinary queue. Agree on escalation and minimum performance for consequential classes before assigning weights. A strong average can conceal weak results on uncommon jobs. Record how many examples supported each result so assessors can see the limits of the test.

For a worked comparison, suppose a supplier routes 38 of 40 invented messages correctly. If both remaining messages belonged in the urgent queue, the 95 per cent headline hides the most important result. Another supplier may refer those messages for review. Count the referrals, time the planner's work and decide whether that workload fits the proposed service.

What should the evaluators take away?

Ask for item-level results with reasons, time to a checked outcome and operating costs at your expected volumes. Include the cost of exceptions, vendor support, monitoring and changing models. Use the existing triage method on the same material so the team can see where a simpler change would deliver a comparable result.

Hold a final moderation session with procurement, the service owner and the people doing the work. Preserve disagreements and explain how they were resolved. The NIST Generative AI Profile recommends testing in conditions relevant to deployment and sharing results with those who authorise release. Include the evaluation pack, known weaknesses and rerun rights in the handover. The buyer then has a way to assess future model changes after the demonstration team has gone.

Sources and further reading