Personalization makes AI considerably more useful. It also gives the system more opportunities to form conclusions about the person using it.
What feels like a million years ago, OpenAI’s Playground gave many of us a very different way of interacting with language models.
It was built as a testing environment where developers could experiment with prompts, models, and parameters before moving their work into the API. Compared with ChatGPT today, the experience was remarkably impersonal. A session had context, but there was no long-term understanding of the person sitting behind the prompt. Start again, and all that context disappeared with it.
As models improved, we discovered that this creates a very particular kind of frustration, one that almost completely disappears once an AI remembers you.
There is something genuinely convenient about no longer having to explain the same background every time. The system knows how you prefer things written, which project you mean when you say “the website,” perhaps which foods you avoid or what problem you were trying to solve last week. You can continue a conversation rather than reconstruct it.
It feels like progress because, in a very practical sense, it is. Personalization is gradually turning AI from a tool we repeatedly instruct into something closer to a system that carries context.
What changes once AI starts remembering?
Once personalization enters the picture, though, the system is doing more than just responding to what a user says. It may rely on something the user said months ago, a pattern it noticed across previous conversations, or an assumption it has made about something that is no longer relevant today.
This is one of the questions we are interested in examining in the Future Intelligence Think Tank (FITT) LinkedIn group: how much context does an AI need before personalization becomes genuinely useful, and where should the limits sit?
Current AI products are already separating different forms of personalization. ChatGPT, for example, distinguishes between memory, chat history, model-improvement settings, and temporary conversations. But even a temporary conversation can use existing personalization without creating new memories, which gives some sense of how many different decisions are hidden inside the simple idea that an AI “knows you.”
How does an AI learn something about you?
There are three ways an AI model can learn things about you:
Disclosure. The user explicitly provides information.
Memory. The system retains that information and uses it later.
Inference. The system reaches a conclusion that the user never directly stated.
These are often treated as slightly different versions of the same process, but they bring quite different privacy problems, and inference is the one we find particularly interesting.
A system can build a meaningful picture of someone without that person ever providing the picture directly. Once inference becomes part of personalization, privacy concerns the conclusions generated by the system as much as the information deliberately disclosed by the user.
The European Data Protection Board has already recognized part of this broader problem in its opinion on AI models, stressing that the use of personal data needs to be understood in context, including whether people could reasonably have expected their information to be used in a particular way.
How much context should survive the conversation?
The central design problem is not whether additional context can improve AI. In many situations, it clearly can, but should every useful piece of context remain available for every future interaction?
One of the GDPR’s core principles, data minimization, is where this tension with personalization becomes especially clear. Personal information should be adequate, relevant, and limited to what is necessary for a defined purpose. The framework also includes purpose and storage limitations, which place boundaries around why information is collected and how long it should be retained.
For a general-purpose AI assistant, that becomes surprisingly complicated because the purpose keeps changing. The same account may be used to prepare something for work in the morning, discuss a financial decision at lunch, and ask a health-related question later that evening.
What should follow you from one conversation to the next?
The ICO makes this problem explicit in its work on agentic AI. Its guidance warns against giving AI systems access to personal information simply because the data may prove useful at some point. There should be a justifiable reason for the system to access it. It seems like a simple principle until the assistant’s purpose is to be useful across different situations.
People are not stable datasets
Persistent personalization also carries a quiet assumption that information about us continues to describe us.
People change jobs, habits, and relationships. Interests disappear. A question about a medical diagnosis may have been prompted by a parent, and a purchase may have been a gift.
An inaccurate answer is easy enough to challenge in a conversation. An inaccurate assumption that sits somewhere in the system and quietly shapes later answers is much harder to notice.
Once that assumption begins influencing future conversations, it can also become strangely convincing. The AI behaves as though the conclusion were true, the personalization around it becomes consistent, and eventually the system appears to have evidence for something that began as an inference.
This is why useful AI memory probably needs some concept of correction, expiry, and selective forgetting. Forgetting, however, becomes much less straightforward once systems can infer new information.
Privacy becomes more complicated when the AI is useful enough
There is a reason this trade-off will be difficult to resolve through warnings and consent screens, because the benefits arrive immediately.
Give an AI access to your email, and it can help draft a reply using the previous conversation. Give it access to your documents, and it can answer questions about your work. Give it enough continuity, and the repetitive explanations that once made digital assistants feel strangely unintelligent begin to disappear.
Now, assume that an inference turns out to be wrong. Perhaps information provided for one reason quietly becomes relevant to another. Perhaps an increasingly capable assistant ends up with access to parts of someone’s life that were previously kept separate. None of these have to produce an obvious failure.*
In fact, the system may feel better precisely because those boundaries have become less visible, and that is exactly what makes personalization such an interesting privacy problem. The tension appears at the point where the technology is working well.
Should an AI know everything?
Another assumption worth questioning here is whether better personalization must always come from accumulating more information.
Context can be temporary, memory can be limited to a particular project, access can be granted for a single task rather than indefinitely, and some processing can occur locally rather than be part of a central profile.
The architecture matters because “the AI knows this” can describe several very different realities.
Perhaps the system knows something for the next thirty seconds.
Perhaps it has stored it for later.
Perhaps another service has the information, and the AI has temporary permission to use it.
Perhaps the system never saw the raw information at all but received the result of a calculation based on it.
To the person asking the question, all four could produce exactly the same useful answer. That suggests the future of personalization may depend less on maximizing what an AI knows and more on deciding what it needs to know here.
Inference does not disappear with less data
However, even careful limits on stored data do not make inference disappear, since a sufficiently capable model can reach conclusions using fragments that seem harmless on their own. This makes privacy more difficult to think about because the sensitive information may never have existed in the original dataset in explicit form.
The ICO has already raised this issue regarding agentic systems, noting that they may rapidly generate new personal information through inference and that, in some cases, those inferences may amount to profiling. It also raises the possibility that inaccurate information could spread through systems and influence later decisions.
At that point, controlling what we tell AI is only part of the problem. We may also need ways to understand what AI has concluded about us.
Perhaps the useful boundary will eventually be less about whether an AI is allowed to remember and more about whether people can see, correct, and limit the profile of themselves that emerges from that memory.
We spent the first years of generative AI trying to give models enough context to understand what we were asking, but now that they are getting better at remembering that context, we need to ask another question:
How much should an AI be allowed to understand about the user?
We’re curious where you would draw the line.
Share your perspective in the comments or continue the discussion in the Future Intelligence Think Tank LinkedIn group.