What AI is, and what it is not, in plain English

Most explanations of AI are written by people who find it interesting. This one is written for someone who has to decide whether to let it near a care record.

What the tools actually do

When you type into ChatGPT, Claude, Gemini or Copilot, the tool is doing one thing: predicting what text is likely to come next.

It has been trained on an enormous quantity of writing, and from that it has learned which words tend to follow which other words, in which contexts. Given your question, it produces a response one piece at a time, each piece chosen because it is a likely continuation of what came before.

That is the whole mechanism. There is no database of facts being consulted. There is no understanding of your service, your residents or your regulator.

Why this explains almost everything else

Nearly every surprising thing about these tools follows from that one fact.

Why they write so well. Fluent, well-structured prose is exactly what “likely next words” produces. The tools are genuinely excellent at form.

Why they are confidently wrong. A plausible-sounding falsehood and a true statement look identical to a system predicting likely text. There is no internal signal that says “I am unsure here”. CQC’s own risk list puts it as: AI “can hallucinate and present untrue information, authoritatively as fact”.

Why they invent specifics. Asked for a dose, a date or a regulation number, the tool produces something that looks like a dose, a date or a regulation number. Leaving a gap is not a likely continuation. Filling it is.

Why they are good at rewording. Rewriting something you supplied into plainer English is a task where the facts are already on the page. The tool is doing form, which is what it is for.

Why they are bad at judgement. Deciding whether someone’s risk has changed is not a text-prediction problem. It requires knowing the person.

Three things people believe that are not true

“It learns from our organisation as we use it.” Generally no. The model does not update itself from your conversations. Some tools remember things within your account, and some suppliers use your text to train future versions — which is a data protection question, not an improvement to your service. See free and paid AI tools.

“It gets things right most of the time, so the risk is small.” The risk is not the error rate; it is that errors are hard to spot. CQC notes AI mistakes “can be difficult to spot, sometimes only after people experience harm”. A rare error in a care record found six months later is worse than a frequent one found immediately.

“It is basically a search engine.” A search engine finds a document that exists and shows it to you. These tools generate text that did not exist before. When a model gives you a regulation number, it has not looked it up.

What they are genuinely good at

Worth saying, because the honest answer is not “nothing”:

• Rewording. Turning dense text into plainer English, or towards easy read.
• Structuring. Taking something rambling and organising it under headings.
• Drafting from scratch where the facts do not matter much yet — a job advert, a first pass at a policy, interview questions.
• Summarising something you supplied, with the warning that summaries drop outliers, and in care the outlier is usually the point.

What they are bad at

• Facts you cannot check. If you would not already know the answer was wrong, do not ask.
• Anything requiring judgement about a person. See deciding eligibility or risk.
• Knowing what it does not know. There is no reliable “I’m not sure” signal.
• Arithmetic and counting, more often than people expect.

A note on the word “AI”

“AI” covers a lot. A tool that transcribes speech, a tool that drafts text, and a tool that scores someone’s risk of falling are all called AI, and they carry completely different risks.

When someone says “we’re using AI”, the useful next question is always: to do what, with whose data? That question does more work than any definition.

What this means for you

You do not need to understand the technology to use it safely. You need to hold two ideas at once: it is good at writing and bad at knowing.

Everything in the golden rules follows from that. A person checks the output because the tool cannot tell you when it is wrong. People make the decisions because the tool has no judgement. Nothing identifying goes into an unapproved tool because the writing goes somewhere.

Start with where are you starting from, or ways to use AI if you want to see what it looks like on real tasks.