How to Evaluate a Pathology AI Tool Before You Buy

Vendor demos of pathology AI tools are designed to impress, and a short demonstration on hand-picked slides tells you little about how the tool will behave in your lab. This guide is for pathologists, lab directors and procurement teams who need a structured way to evaluate a pathology AI tool before committing budget, covering the questions to ask, how to test locally and what to watch for in the contract.

Define the problem and intended use first

Write down, in one or two sentences, what you want the tool to do and how its output will be used. For example: an aid for scoring a specific immunohistochemical marker, reviewed by a pathologist before reporting. This statement becomes your evaluation yardstick.

Be explicit about:

• The specimen types, stains and diagnoses in scope, and those excluded.
• Whether the tool screens, triages, measures or suggests a diagnosis.
• Who reviews the output and at what point in the case.
• What success looks like: better reproducibility, faster sign-out, fewer missed findings, or less typing.

Without this, evaluations drift into general impressions and are hard to defend later. It also helps you judge whether the tool fits the way your pathologists work; our article on fitting AI-assisted diagnosis into the workflow covers that side in detail.

Involve at least one pathologist who is skeptical of the tool. Their questions tend to surface weaknesses early, and their eventual support carries weight with colleagues.

Questions to ask the vendor

Evidence and regulatory status

• What is the regulatory status of the tool in our country, and for exactly which intended use?
• What studies support performance, and were test sets drawn from institutions not used in training?
• Are results reported by subgroup, such as tissue type, grade, scanner or demographic group?
• What are the known failure modes, and how does the tool behave on images it was not designed for?

Data and technical fit

• Which scanners, file formats and magnifications are supported, and which have actually been tested?
• Where is image data processed and stored, and does any of it leave our environment?
• How does the tool integrate with our viewer and laboratory information system?
• How are model updates delivered, and will we be told when the model changes?

Support and accountability

• What training is provided for pathologists and technical staff?
• How are errors reported and investigated, and how quickly?
• What logs are available for audit: inputs, outputs, model version, user actions?

Be cautious of answers that rely on headline accuracy numbers without describing the test population, or that cannot name any limitations.

Testing a pathology AI tool on your own slides

Published performance describes someone else's slides. Local testing is where most surprises appear.

1. Assemble a representative case set. Use retrospective cases from your own archive that reflect your real case mix, including difficult and borderline cases, poor-quality slides and less common entities. Avoid cherry-picking clean examples.
2. Fix the reference standard in advance. Decide whether you compare with signed-out diagnoses, a consensus read or another standard, and how disagreements will be adjudicated.
3. Include your variation. Test slides from different days, staining batches and, if relevant, more than one scanner.
4. Measure what matters to you. Agreement with the reference standard, time per case, number of flags requiring review, and pathologist confidence are often more informative than a single accuracy figure.
5. Study the failures. Review every discordant case with a pathologist. Understanding why the tool failed tells you whether failures are rare and recognizable or systematic and hidden.
6. Document everything. Record the protocol, case selection, results and decision. This becomes part of your method validation file.

If your lab follows ISO 15189, this evaluation fits within the verification you would perform for any new examination procedure. See what ISO 15189 changes day to day for how method changes are typically managed.

Usability and the pilot phase

Accuracy alone does not predict adoption. A short pilot with a few pathologists using the tool on live cases, in parallel with usual practice, reveals practical issues quickly.

Ask pilot users to note how many clicks the tool adds, whether output appears where they need it, whether it slows the viewer, and whether they trust it after a week of use. Pay attention to the histology team as well, since their slide quality affects performance.

Use the pilot to estimate total cost of ownership as well. Licensing is only part of it: computing infrastructure, storage for additional outputs, integration work, validation time and ongoing monitoring all cost money or staff hours. A tool that saves a few minutes per case but needs a dedicated person to maintain it may not pay off in a smaller lab.

Red flags

Some signals suggest a tool is not ready for your lab, or that the vendor relationship will be difficult:

• No willingness to support a local evaluation, or pressure to sign before testing.
• Performance claims without a description of the test data.
• Output that cannot be reviewed visually, with no indication of where on the slide a result came from.
• Silent model updates, or no versioning in the logs.
• Unclear data handling terms, especially for patient images leaving your jurisdiction.
• Integration that requires pathologists to work in a separate application.
• Reluctance to share examples of cases where the tool performed poorly.

Contract and lifecycle considerations

Before signing, make sure the agreement covers data ownership and retention, notification of model changes, access to audit logs, service levels, and an exit plan that lets you retrieve your data. Plan internally for ongoing monitoring: who reviews discordant cases, how often performance is rechecked, and what triggers revalidation. These responsibilities should sit within your quality management system, not in one person's head.

Agree, too, on what happens if the vendor withdraws the product, changes its pricing model or is acquired, so your lab is not left with an unsupported tool in routine use.

Key takeaways

• Write a clear intended use before any demo, and evaluate against it.
• Ask for evidence from independent test sets, subgroup results and known failure modes.
• Test locally on representative, unselected cases with a predefined reference standard, and study every failure.
• Judge usability in a real pilot, not a demo.
• Secure model versioning, audit logs and data terms in the contract, and plan for ongoing monitoring.

Choosing tools is easier when the underlying platform keeps reports, images and audit logs in one place. PATHOLOGY+ is a cloud laboratory platform with audit logging and role-based access control built in. Book a demo to see how it fits your evaluation.