Questions to ask an AI supplier
Nine questions that separate a supplier who has thought about care from one who has not, with the answers that should worry you and the ones that should reassure.
When we researched eight AI tools for our directory, the most useful thing we learned was not which tools are good. It was how differently suppliers behave when you ask a direct question.
Some publish everything. Some publish nothing and expect you not to notice. The questions below are the ones that sorted them.
Take the printable question sheet to the meeting.
1. Where is our data held?
Why it matters. It belongs in your privacy notice, your DPIA, and any conversation with commissioners.
What we found. Two of the tools we checked state plainly that they host on US infrastructure. One social care tool states all processing happens in the UK. One major supplier does not publish a hosting region at all, and offers no UK data residency option.
Good answer. A named country or region, and whether UK residency is available and on which plan.
Worrying answer. “In the cloud.” “On secure servers.” Both mean the supplier either does not know or would rather not say.
2. Do you train your models on our data — and can anyone read it?
Two questions, deliberately. They have different answers and suppliers often conflate them.
What we found. One transcription supplier’s own privacy policy says it trains its AI on “transcriptions (which may contain Personal Information)”, with no distinction between free and paid plans. Meanwhile a supplier that does not train on your data may still allow humans to read it: one states that “specialized third-party contractors” may review business conversations for abuse.
Good answer. A clear no to training on the plan you are buying, plus a straight answer about human review, and a link to where it says so.
Worrying answer. “Your data is safe with us.” That is not an answer to either question.
3. Will you sign a data processing agreement, and can I see it now?
Why it matters. Without a DPA there is no contract governing what the supplier does with personal data. It is the document that makes you the controller and them the processor.
What we found. Every DPA we could confirm was business-tier only. For two tools we could not confirm one existed at all from any published page.
Good answer. Yes, here it is — or it is built into our commercial terms, here they are.
Worrying answer. “We can look at that at contract stage.” You are being asked to trial with real data before the contract exists.
4. What certifications do you hold, and which products do they cover?
The second half is the important half.
What we found. One supplier holds SOC 2 Type 2, ISO 27001, ISO 42001 and more — but its certification matrix covers only its Team, Enterprise and API products. Its consumer plans do not appear at all. Another marks SOC 2 and ISO as “No” against every individual plan.
For UK care, UK Cyber Essentials is worth asking about specifically. So is anything that helps with your DSPT return.
Good answer. Named certifications, with the product scope, and a certificate you can see.
Worrying answer. A logo on a website with no scope attached.
5. Can you help with our DSPT return?
Why it matters. The Data Security and Protection Toolkit is NHS England’s self-assessment against the National Data Guardian’s ten data security standards. As its own site puts it: “All organisations that have access to NHS patient data and systems must use this toolkit.”
If you file a DSPT return, a new AI tool touching that data becomes part of what you are assuring.
Good answer. A supplier who knows what the DSPT is and has helped other providers with it.
Worrying answer. “What’s that?” It does not disqualify them, but it tells you how much UK care work they have done, and how much of the explaining will fall to you.
6. Is it “CQC approved”?
This is a trick question, and the answer tells you a lot.
CQC’s own words: “We do not assess or approve specific technologies.” There is no such thing as a CQC-approved AI tool.
Good answer. “No — CQC doesn’t approve tools. Here’s how we help you meet the regulations instead.”
Worrying answer. “Yes.” A supplier claiming CQC approval is either misinformed about the regulator they are selling into, or willing to say things that are not true. Neither is what you want handling data about the people you support.
7. What happens when it gets something wrong?
Why it matters. CQC’s principles require “clear mechanisms for addressing issues or harm caused by AI” and systems “to recognise, report and investigate when something goes wrong”.
Good answer. A named route for reporting errors, something that happens when you do, and honesty about known failure modes.
Worrying answer. A supplier who cannot name a single thing their tool is bad at has either not looked or will not say. Every tool in our directory has a “watch out for” section, because every tool has one.
8. What does it cost, fully?
Ask specifically: does the price include VAT, does it need another licence underneath, what is the minimum term, and what does integration cost?
What we found. One major tool quotes prices excluding VAT and requires a separate licence underneath it. Another supplier publishes no pricing at all.
See paying for AI for what else to budget.
9. Who else in social care uses this, and can I speak to them?
Why it matters. Most AI tools were built for offices, not care. Evidence from a large council adult social care team may not carry across to a twelve-person domiciliary service.
Good answer. Named reference sites of roughly your size and type, and a willingness to connect you.
Worrying answer. Case studies you cannot verify, or percentage improvements with no independent evaluation behind them. Reported figures from a supplier and its customer are a claim, not a finding.
After the meeting
- Get the important answers in writing. A sales call is not a contract.
- Put them into your DPIA while they are fresh.
- Note the date. Supplier terms change, and two of the eight we checked had renamed or restructured within the year.
- If the answers were vague, that is itself an answer. There is no shortage of suppliers.