Srileo LearnProfessional Learning Platform

MODULE 5 · LESSON 2

Free — no login required

Sign in to track progress, save quiz attempts and enrol in the full course.

Sign in to track progress / enrol

Evaluating Vendors Without Being Sold To

The demo is the good weather

AI for Non-Technical Teams established this: a demonstration shows the easy majority of the distribution. Every vendor demo uses documents that work, questions the system answers well, and a dataset curated over months.

None of that is dishonest. It is simply uninformative about your operation, which is why the central discipline of vendor evaluation is: never evaluate on their data.

The question set

Ten questions. A vendor with a shipped product answers all of them readily; one with a prototype and a sales team becomes vague, and the vagueness is your finding.

On performance

  1. What is your accuracy on documents like ours, and how was that measured? Watch for accuracy quoted without a dataset. "94% accurate" is meaningless without knowing on what.
  2. What does the system do when it is unsure? You want a confidence signal and a route to a human. A vendor whose product cannot express uncertainty is selling something that will fail silently.
  3. Show me your worst cases. The single most revealing question in the set. A mature vendor has a ready answer and is comfortable discussing it. A defensive response tells you they have not looked.

On your reality

  1. Which of our systems do you already integrate with, and which customers can confirm it? Integration is the biggest hidden cost from module 4.
  2. How does the system handle a new document layout or a new category it has not seen? Establishes whether you can maintain it or must return to them for every change.
  3. What do we have to supply, and how long does that usually take? Their honest answer to this predicts your build cost better than their quote does.

On operations

  1. How do we monitor whether it is still working, and what do we see? Module 8 depends on the answer.
  2. What happened at your last significant incident? Every vendor has had one. How they describe it tells you how they will treat you.

On commercials

  1. How does the price change as usage grows? Usage metering from module 4 — model your actual volume, not the sample tier.
  2. What does leaving look like? Covered fully in the next lesson, and asking it early changes the tone of everything that follows.

Designing a proof of concept that decides something

Most proofs of concept fail to produce a decision because nobody agreed in advance what would constitute passing.

📅 Timeline
Use your dataYour real documents, your real tickets, including the awkward ones. If commercial sensitivity blocks this, that is a genuine constraint to solve, not a reason to accept their sample.
Use a held-back setGive the vendor a training portion and keep an evaluation portion they never see. Otherwise you are measuring memorisation.
Define the pass mark firstWritten down before the vendor starts, derived from your baseline. "Above 85% field-level accuracy on the held-back set, with confidence flagging the errors."
Time-box itFour to six weeks. An open-ended proof of concept becomes a slow implementation nobody approved.
Include the exception pathEvaluate what happens to the cases it cannot handle, because that is your permanent operating cost.

Run two vendors on the same held-back set where the decision matters. The comparison is worth far more than either result alone, and it changes the commercial dynamic in your favour.

Claims that should not survive scrutiny

  • "It learns from your data automatically." Ask what that means concretely. Does it retrain, and on what cadence? Who reviews what it learned? Frequently this means nothing at all.
  • "Accuracy improves over time." Only if corrections are captured and fed back. Ask how, and ask to see it.
  • "No integration required." Usually true and usually irrelevant — it means someone uploads files manually, which is not a production process.
  • "Our model is proprietary." Often a wrapper around a widely available model. Not disqualifying, but it should change what you pay and how you view lock-in.
  • "Used by [large recognisable company]." Ask for what, at what scale, and for how long. A pilot in one department is a legitimate answer, and a different one from the impression created.

Vendors supply references who will say good things. You can still get an accurate picture, provided you ask about process rather than satisfaction.

Useless: "Are you happy with the product?" They were selected to say yes.

Useful, in this order:

  • "How long from signing to the first real usage?" Compares against the vendor's implementation estimate. A gap of several months is common and tells you what to budget.
  • "What surprised you?" Open enough that people answer honestly, and it surfaces the thing no sales process would have disclosed.
  • "What did you have to do that you had not expected?" This is where the hidden costs live — data cleaning, an integration built internally, a process changed to suit the tool.
  • "What proportion of cases go to a human, and did that match expectations?" Your permanent operating cost, from someone with no reason to flatter it.
  • "How did they handle it when something went wrong?" Every implementation has a bad month.
  • "If you were choosing again, what would you do differently?" The most productive question in any reference call, because it invites reflection rather than judgement, and people answer it candidly even about products they like.

One more move that is worth the effort: ask the vendor for a reference who stopped using the product. Most will decline, which is itself informative. The occasional vendor who agrees is demonstrating unusual confidence, and that call will be the most useful conversation in your evaluation.

❓ Knowledge Check

Why must a proof of concept use a held-back evaluation set the vendor never sees?

📚 Flashcards1 / 6
Term

Never evaluate on their data

Click to flip
Definition

A demo shows the easy majority of the distribution, curated over months. It is uninformative about your operation, so evaluation must run on your documents, tickets and awkward cases.

Click to flip back
💡Key Takeaway

Never evaluate on the vendor's data: run a time-boxed proof of concept on your own cases with a held-back set and a pass mark written down first. Work through the ten questions, and treat vagueness as the finding rather than an inconvenience — particularly on worst cases, uncertainty handling and what leaving looks like. In reference calls, ask about process rather than satisfaction, and ask for a customer who left.