The Vendor Evaluation Questions a Feature Comparison Will Never Ask

Every vendor RFP asks the questions a vendor is well-rehearsed to answer well. The questions that actually reveal risk are the ones a good salesperson visibly hesitates on — and most evaluation processes never ask them, because they don’t fit neatly into a feature-comparison scorecard.

I’ve written about the build-vs-buy framework that has to survive contact with sales, and about measuring vendor lock-in before signing. This is the specific set of questions that surface the answers a standard RFP process reliably misses — because they’re not about features, they’re about what happens when something goes wrong, which is precisely the part of the relationship a sales process is least equipped to discuss convincingly.

Questions about failure, not success

“Walk me through your worst outage in the last two years — what happened, and what changed afterward.” Every vendor has had one. The answer that matters isn’t the outage itself — it’s whether the vendor can describe it specifically, with a concrete remediation, versus a vague, rehearsed reassurance that avoids naming anything real. A vendor unwilling or unable to discuss a genuine failure specifically is telling you something about how they’ll handle the next one with you.

“What’s the SLA remedy, in practice, if you miss it — and has a customer ever actually invoked it?” Most SLAs promise service credits that sound meaningful until you calculate what they’re actually worth against the cost of the outage they’re meant to compensate for. Asking whether the remedy has ever actually been paid out, and what the process looked like, separates a real commitment from boilerplate nobody has tested.

Questions about dependency, not capability

“Which of your own critical functions run on a single sub-provider, and what happens to my service if that sub-provider fails?” A vendor’s own concentration risk becomes your concentration risk the moment you depend on them, and most vendors have never been asked to disclose their own fourth-party dependencies, because most customers never ask. The answer, or the refusal to give one, tells you whether the vendor has actually thought about this or is passing the same unexamined risk downstream.

“If you were acquired tomorrow by [a specific plausible acquirer], what in this contract protects me?” Vendor acquisition is one of the most common ways a stable relationship becomes an unstable one, and most contracts are silent on what happens to pricing, roadmap commitments, and support quality after a change of control. A vendor who has genuinely thought about this will have a specific answer. Most don’t, and the silence is the answer.

Questions about exit, asked before entry

“Show me, concretely, what data export actually looks like — not a policy statement, an actual sample export.” Requesting a genuine sample export, in the format the contract promises, before signing, catches the gap between “we support data export” and export that’s actually usable by a replacement system — a gap that’s invisible in a policy document and immediately obvious the moment you look at an actual file.

“Who’s the last customer who left you, and can I talk to them?” A vendor confident in their exit experience will have a good answer. A vendor who’s never lost a customer either has a genuinely retentive product — worth understanding why — or has never actually been tested on the exit promises in their own contract, which is itself useful information before you become the test case.

Why these questions get skipped

None of these questions fit a standard feature-comparison scorecard, which is precisely why they get skipped in favour of questions a vendor’s sales engineering team has polished answers for. The uncomfortable questions are the ones worth asking specifically because they’re uncomfortable — a vendor’s composure, specificity, and willingness to engage honestly with a hard question is itself diagnostic information a features list will never give you.

What one uncomfortable question actually surfaced

During a procurement review for a core operations platform, asking a shortlisted vendor’s sales team the sub-provider dependency question directly produced a long pause and a promise to “follow up with the technical team” — a red flag in itself, since a mature vendor should know its own critical dependencies without needing to check. The eventual answer, three days later, revealed a single-region dependency on a specific cloud availability zone that the vendor’s own marketing materials had never mentioned and that materially changed the concentration-risk picture for a customer already using the same cloud provider for other critical functions. The feature comparison had rated this vendor highest. The uncomfortable question was the only thing in the entire evaluation that surfaced the actual risk.

Building a genuine vendor-evaluation question set for a specific procurement — the questions that actually surface risk rather than confirm what the sales deck already claims — is exactly the kind of due-diligence work a technology control assessment is built to run before a contract is signed, not after.