Safeguarding and AI
Where AI touches safeguarding, the rules get stricter, not looser. What must never be automated, the quiet risk in AI-smoothed records, and what to do if details reach an unapproved tool.
Everywhere else on this site we try to be balanced about what AI can and cannot do. This page is less balanced, because safeguarding is where the consequences of getting it wrong land on a person rather than on a process.
The line that does not move
AI must never decide whether to raise a safeguarding concern.
Not as a first filter, not as a prompt, not as a score. The decision that something might be abuse or neglect, and that it needs raising, stays with a person.
CQC’s first principle for good use of AI is “AI to support, not to replace: AI can enhance, but not replace human decision making.” In most areas that is a judgement about where to draw a line. Here it is the line.
There is a practical reason as well as a principled one. These tools work by predicting likely text. Safeguarding concerns are, by their nature, the unlikely thing — the detail that does not fit, the account that does not quite hold together. A system optimising for the probable is the wrong instrument for noticing the improbable.
Never put safeguarding details into an unapproved tool
The general rule on this site is that nothing identifying goes into a tool your organisation has not approved. For safeguarding material the rule is the same, and the consequences of breaking it are worse.
Consider what the suppliers themselves say. Google’s privacy notice states that “a subset of chats are reviewed by human reviewers (including Google’s trained service providers)”, and that those conversations are retained up to three years, with deletion of the chat not removing them.
Now apply that to a paragraph describing an allegation of abuse, naming a person, a family member and an address. A stranger at a contractor may read it. A copy exists for three years. You have no data processing agreement, no lawful basis, and no way to retrieve it.
That is not a hypothetical policy breach. It is a disclosure of the most sensitive category of information a care service holds, to an organisation with no relationship to you.
The quiet risk: smoothing
This one is less obvious than the others, and in our view more likely to actually happen.
AI is good at making writing sound professional. Applied to care records, that tends to soften:
- “He shouted and threw the cup” becomes “he became agitated”
- “She flinched when her son came in” becomes “she appeared uncomfortable”
- “He refused to let me in and I could hear someone crying” becomes “access was declined”
Each rewrite is defensible on its own. The problem is cumulative. Safeguarding is very often recognised through a pattern across records over time, and a pattern of escalation that has been smoothed, entry by entry, into a series of mild events can become invisible.
The person who later reads those records — a safeguarding lead, a social worker, a coroner — reads what was written, not what happened.
What to do. Where the original language was strong, keep the original language. Direct speech should survive into the record. This is a specific thing to look for in spotting AI mistakes, and a specific thing to say in training.
What AI can reasonably do
Once a person has decided to raise a concern, AI can help with the writing — structuring a referral you have drafted, or checking that a report covers the headings your local authority asks for.
Two conditions. The decision was already made by a person. And the tool is an approved one with a data processing agreement, because the content is still safeguarding material.
It can also help with things that touch safeguarding without containing it: drafting training materials, rewriting your safeguarding policy into plainer English, or producing an easy-read version of how to raise a concern. None of those involve a real person’s details.
If it has already happened
Someone will eventually paste something into the wrong tool. Plan for it now, so the response is not invented under pressure.
- Treat it as a potential personal data breach. Follow your normal breach process, including the assessment of whether it is reportable to the ICO.
- Record what was entered, when, and into which tool and account. You cannot assess the risk without this, and memory degrades quickly.
- Delete what you can, while understanding the limits. Deleting a chat does not necessarily remove a copy that has been reviewed or retained.
- Consider the risk to the person, not just to the organisation. That is the question that actually matters, and it is the one most breach processes get to last.
- Do not discipline first. If staff believe reporting this will get them in trouble, the next one will not be reported. CQC asks for systems “to recognise, report and investigate when something goes wrong so that lessons are learned”.
What we are not saying
We are not saying AI has no place in a service that does safeguarding — which is every service. We are saying that safeguarding material is the category where the general rules stop being guidance and start being the point.
If you take one thing from this page: people decide, and safeguarding details never go into a tool you have not approved.
See the golden rules, and put the “never paste this” poster where staff can see it.