What feels like a million years ago, OpenAI’s Playground gave many of us a very different way of interacting with language models.

It was built as a testing environment where developers could experiment with prompts, models, and parameters before moving their work into the API. Compared with ChatGPT today, the experience was remarkably impersonal. A session had context, but there was no long-term understanding of the person sitting behind the prompt. Start again, and all that context disappeared with it.

As models improved, we discovered that this creates a very particular kind of frustration, one that almost completely disappears once an AI remembers you.

There is something genuinely convenient about no longer having to explain the same background every time. The system knows how you prefer things written, which project you mean when you say “the website,” perhaps which foods you avoid or what problem you were trying to solve last week. You can continue a conversation rather than reconstruct it.

It feels like progress because, in a very practical sense, it is. Personalization is gradually turning AI from a tool we repeatedly instruct into something closer to a system that carries context.

What changes once AI starts remembering?

Once personalization enters the picture, though, the system is doing more than just responding to what a user says. It may rely on something the user said months ago, a pattern it noticed across previous conversations, or an assumption it has made about something that is no longer relevant today.

This is one of the questions we are interested in examining in the Future Intelligence Think Tank (FITT) LinkedIn group: how much context does an AI need before personalization becomes genuinely useful, and where should the limits sit?

Current AI products are already separating different forms of personalization. ChatGPT, for example, distinguishes between memory, chat history, model-improvement settings, and temporary conversations. But even a temporary conversation can use existing personalization without creating new memories, which gives some sense of how many different decisions are hidden inside the simple idea that an AI “knows you.”

How does an AI learn something about you?

There are three ways an AI model can learn things about you:

• Disclosure. The user explicitly provides information.

• Memory. The system retains that information and uses it later.

• Inference. The system reaches a conclusion that the user never directly stated.

These are often treated as slightly different versions of the same process, but they bring quite different privacy problems, and inference is the one we find particularly interesting.

A system can build a meaningful picture of someone without that person ever providing the picture directly. Once inference becomes part of personalization, privacy concerns the conclusions generated by the system as much as the information deliberately disclosed by the user.

The European Data Protection Board has already recognized part of this broader problem in its opinion on AI models, stressing that the use of personal data needs to be understood in context, including whether people could reasonably have expected their information to be used in a particular way.

How much context should survive the conversation?

The central design problem is not whether additional context can improve AI. In many situations, it clearly can, but should every useful piece of context remain available for every future interaction?

One of the GDPR’s core principles, data minimization, is where this tension with personalization becomes especially clear. Personal information should be adequate, relevant, and limited to what is necessary for a defined purpose. The framework also includes purpose and storage limitations, which place boundaries around why information is collected and how long it should be retained.

For a general-purpose AI assistant, that becomes surprisingly complicated because the purpose keeps changing. The same account may be used to prepare something for work in the morning, discuss a financial decision at lunch, and ask a health-related question later that evening.

What should follow you from one conversation to the next?

The ICO makes this problem explicit in its work on agentic AI. Its guidance warns against giving AI systems access to personal information simply because the data may prove useful at some point. There should be a justifiable reason for the system to access it. It seems like a simple principle until the assistant’s purpose is to be useful across different situations.

People are not stable datasets

Persistent personalization also carries a quiet assumption that information about us continues to describe us.

People change jobs, habits, and relationships. Interests disappear. A question about a medical diagnosis may have been prompted by a parent, and a purchase may have been a gift.

An inaccurate answer is easy enough to challenge in a conversation. An inaccurate assumption that sits somewhere in the system and quietly shapes later answers is much harder to notice.

Once that assumption begins influencing future conversations, it can also become strangely convincing. The AI behaves as though the conclusion were true, the personalization around it becomes consistent, and eventually the system appears to have evidence for something that began as an inference.

This is why useful AI memory probably needs some concept of correction, expiry, and selective forgetting. Forgetting, however, becomes much less straightforward once systems can infer new information.

Privacy becomes more complicated when the AI is useful enough

There is a reason this trade-off will be difficult to resolve through warnings and consent screens, because the benefits arrive immediately.

Give an AI access to your email, and it can help draft a reply using the previous conversation. Give it access to your documents, and it can answer questions about your work. Give it enough continuity, and the repetitive explanations that once made digital assistants feel strangely unintelligent begin to disappear.

Now, assume that an inference turns out to be wrong. Perhaps information provided for one reason quietly becomes relevant to another. Perhaps an increasingly capable assistant ends up with access to parts of someone’s life that were previously kept separate. None of these have to produce an obvious failure.*

In fact, the system may feel better precisely because those boundaries have become less visible, and that is exactly what makes personalization such an interesting privacy problem. The tension appears at the point where the technology is working well.

Should an AI know everything?

Another assumption worth questioning here is whether better personalization must always come from accumulating more information.

Context can be temporary, memory can be limited to a particular project, access can be granted for a single task rather than indefinitely, and some processing can occur locally rather than be part of a central profile.

The architecture matters because “the AI knows this” can describe several very different realities.

• Perhaps the system knows something for the next thirty seconds.

• Perhaps it has stored it for later.

• Perhaps another service has the information, and the AI has temporary permission to use it.

• Perhaps the system never saw the raw information at all but received the result of a calculation based on it.

To the person asking the question, all four could produce exactly the same useful answer. That suggests the future of personalization may depend less on maximizing what an AI knows and more on deciding what it needs to know here.

Inference does not disappear with less data

However, even careful limits on stored data do not make inference disappear, since a sufficiently capable model can reach conclusions using fragments that seem harmless on their own. This makes privacy more difficult to think about because the sensitive information may never have existed in the original dataset in explicit form.

The ICO has already raised this issue regarding agentic systems, noting that they may rapidly generate new personal information through inference and that, in some cases, those inferences may amount to profiling. It also raises the possibility that inaccurate information could spread through systems and influence later decisions.

At that point, controlling what we tell AI is only part of the problem. We may also need ways to understand what AI has concluded about us.

Perhaps the useful boundary will eventually be less about whether an AI is allowed to remember and more about whether people can see, correct, and limit the profile of themselves that emerges from that memory.

We spent the first years of generative AI trying to give models enough context to understand what we were asking, but now that they are getting better at remembering that context, we need to ask another question:

We’re curious where you would draw the line.

Share your perspective in the comments or continue the discussion in the Future Intelligence Think Tank LinkedIn group.

Leave a comment

There is one problem. They cannot explain why it is correct.

That situation is becoming an ordinary part of knowledge work. Developers review generated code. Analysts inspect generated conclusions. Clinicians increasingly work alongside systems that identify patterns or suggest diagnoses.

The obvious question is whether AI improves the result or reduces the time required to produce it. A harder question follows: which abilities still need regular human practice once AI can perform the underlying task?

Organizations need an answer before convenience answers it for them.

The argument emerging from a recent Future Intelligence Think Tank (FITT) discussion is practical. Some professional capabilities should be deliberately preserved, even when using AI is faster. The strongest candidates are the abilities people need to verify AI output, exercise judgment in uncertain situations and recover when automation fails.

That suggests a new management responsibility: AI skill retention.

Join our LinkedIn group

Efficiency changes what professionals practice

AI can reduce the amount of direct work required to complete a task. That advantage is easy to measure.

The less visible consequence is a change in what the professional repeatedly practices.

A longitudinal study of software engineers by Annie Vella and Kelly Blincoe found that AI coding assistants were changing the balance of engineering work. Participants reported spending less time writing code and more time directing, evaluating and correcting AI output, a category the researchers call supervisory engineering work. Perceived productivity remained high while aspects of developer experience, including cognitive load and flow, worsened for some participants over time.

Another study of 442 developers by Zixuan Feng, Sadia Afroz and Anita Sarma found that GenAI adoption could increase job demands associated with burnout, while supportive resources and positive perceptions could mitigate some of those effects.

These findings complicate a simple productivity calculation.

If a developer writes less code and reviews more generated code, the organization has changed the work. It has also changed the practice through which expertise develops.

That distinction appeared repeatedly in the FITT discussion behind this article. One participant, data engineer Andronic Tudor, argued for deliberately keeping part of the work without AI as practice. His concern was that reviewing generated output does not exercise exactly the same capability as producing the work yourself.

Gheorghe David, Co-Founder and Managing Partner at ASSIST Software, pushed the idea further. Professional expertise has traditionally developed through doing difficult work, making mistakes and building judgment. If AI removes some of that difficulty, organizations may gain immediate productivity while reducing opportunities to develop the expertise they will need later.

This creates a management question that deserves more attention than tool adoption rates:

What should people continue doing themselves because the act of doing it develops a capability the organization still depends on?

Verification requires knowledge that cannot be outsourced completely

The strongest reason to preserve a skill is verification.

A professional cannot reliably supervise an AI system using knowledge they no longer possess.

Software offers an obvious example. Generated code may compile, pass tests and look convincing. A developer still needs enough understanding of architecture, security, edge cases and system behavior to recognize when the solution is locally plausible and globally wrong.

Medicine provides evidence that the problem extends beyond software.

A 2025 observational study examined endoscopists who had begun routinely using AI during colonoscopies. Researchers compared their performance on colonoscopies conducted without AI before and after AI was introduced into their practice. Adenoma detection during those unaided procedures fell from 28.4 percent to 22.4 percent. The authors described the result as evidence of a possible deskilling effect, while also noting the observational design of the study.

A separate experiment involving 27 radiologists examined what happened when readers received incorrect AI suggestions while assessing mammograms. Accuracy fell substantially when the AI recommendation was wrong, including among highly experienced radiologists.

These studies come from specific clinical settings, so their results should not be generalized mechanically to every profession. They still expose an important design problem for AI supported work: supervision depends on the human retaining enough independent capability to challenge the machine.

A review process is valuable only when the reviewer can detect something worth challenging.

Failure reveals which skills still matter

There is another useful way to decide what should remain practiced: imagine the AI disappears.

Which abilities become immediately important?

For some tasks, very little is lost. If AI drafts a routine internal summary and the system becomes unavailable, the cost may be a small delay.

Other tasks are different. A developer may need to debug a production failure when the assistant lacks the required context. An engineer may need to understand why an automated recommendation conflicts with observed system behavior. A clinician may need to make a judgment when the model is unavailable or its recommendation does not fit the patient.

Capabilities required under those conditions cannot exist only as historical knowledge inside the organization.

They need people who still know how to use them.

This is why the FITT discussion repeatedly returned to practice. One suggestion was to rotate professionals through periods of work without AI and assess their understanding of the system itself, not simply the amount of work they shipped.

That idea deserves serious consideration.

Organizations already maintain redundancy in systems they consider important. They test backups, rehearse incident procedures and maintain fallback processes because reliability cannot depend on every component working perfectly.

Human capability deserves similar attention when it is part of the fallback.

A practical test for AI skill retention

Organizations do not need to preserve every manual process. That would sacrifice useful automation and turn skill retention into ritual.

They do need a way to distinguish disposable effort from capability that supports reliable judgment.

A simple test can begin with three questions:

1. Is this capability necessary to verify AI output? If someone cannot judge the quality of an AI result without the underlying skill, regular practice has operational value.

2. Will this capability be needed when AI fails or lacks context? Skills that support recovery, escalation and unusual cases should remain available inside the organization.

3. Does practicing the task develop judgment used elsewhere? Some work teaches more than the task itself. Debugging can develop systems thinking. Writing can sharpen reasoning. Direct analysis can expose assumptions that disappear when the first interpretation arrives preassembled.

A capability that meets one of these conditions deserves examination. A capability that meets all three deserves deliberate protection.

The policy can then become concrete.

Teams might identify a small set of capabilities that must remain practiced, define how often professionals exercise them without AI assistance and include human understanding in performance reviews or technical assessments.

The exact mechanism will differ by profession. The principle is stable: do not let frequency of AI use become the accidental mechanism that decides which human skills survive.

Productivity metrics need a longer horizon

AI adoption is often evaluated through measures such as output, cycle time, cost and adoption.

Those numbers matter. They can also miss capability erosion until the organization needs the capability again.

A team can become faster this quarter while becoming less capable of diagnosing an unusual failure next year. A junior professional can produce stronger work immediately while receiving fewer opportunities to build the judgment expected of a senior professional later.

That creates a delayed cost.

The difficult part is that capability retention can look inefficient in the short term. Asking an engineer to solve some problems without assistance takes longer. Requiring professionals to explain the reasoning behind an AI generated result adds effort. Preserving direct practice means accepting some friction.

The FITT discussion surfaced a useful idea here: some friction may be part of how expertise is maintained.

Organizations therefore need to decide which friction is waste and which friction is training.

AI makes that distinction increasingly important because the easiest workflow will usually favor more delegation. Without an explicit policy, the organization may discover its skill boundary only after crossing it.

The abilities worth protecting

The purpose of a capability retention policy is not to preserve old ways of working for sentimental reasons.

Its purpose is to protect the human capacity that makes AI useful and safe.

A professional who can verify a result can use AI with greater confidence. A professional who understands the underlying system can intervene when automation behaves unexpectedly. A professional who continues exercising judgment can recognize situations that fall outside the assumptions built into the tool.

That leads to a useful principle for AI adoption:

Delegate the work when it improves the system. Preserve the capability when the organization still depends on the human knowing how to judge, challenge or recover that work.

The difficult decision is identifying where that line sits in each profession.

Which part of your work would you continue practicing even if AI could perform it faster?

The Future Intelligence Think Tank brings together executives, specialists and researchers examining questions like this. Join the discussion on LinkedIn.

Join our LinkedIn group

The harder task is deciding which developments deserve more than a passing glance.

Some questions reveal much more about the direction of artificial intelligence than the biggest headline of the week. They concern what happens to human judgment when AI becomes part of everyday work. They ask whether enormous investment in AI infrastructure reflects durable economic change. They examine what happens when intelligent systems begin making decisions in the physical world.

These are the kinds of questions appearing inside the Future Intelligence Think Tank (FITT), an open professional community hosted on LinkedIn.

And recently, three of them have been particularly difficult to ignore.

Join our LinkedIn group

What happens when AI makes us faster, but thinking becomes harder?

Productivity remains one of the strongest arguments for adopting AI. Developers can generate code more quickly. Knowledge workers can research, summarise and draft with less effort. In medicine, AI supported analysis can make relevant information easier to access.

Yet measuring output tells us only part of what is changing.

In a recent discussion in Future Intelligence Think Tank, Vlad-Vasile Dediță explored the idea of cognitive debt and the possibility that AI can increase immediate productivity while gradually affecting judgment, attention or professional skill.

It raises an uncomfortable question. If a person completes a task faster with AI, what exactly improved?

The answer could be efficiency. It could also mean that part of the reasoning previously required to complete the task has shifted somewhere else. That distinction matters in professions where understanding how a conclusion was reached is as important as reaching it.

Software engineering offers an obvious example. Generating functioning code quickly can be valuable, but understanding its architecture, dependencies, security implications and failure modes remains essential. The same question becomes more consequential in medicine, finance or any environment where decisions carry significant responsibility.

AI adoption therefore creates a second challenge alongside implementation: understanding how human expertise changes once intelligent assistance becomes routine.

What skills become more important when AI handles more of the execution? Which ones weaken? And how do organizations recognize the difference?

Those questions are currently part of the conversation inside FITT.

How much of the AI boom belongs to the technology, and how much belongs to the market around it?

Billions are moving into chips, data centers, energy infrastructure and computing capacity, companies are reorganizing strategies around AI, and expectations for future demand are influencing investment decisions today.

Naturally, comparisons with previous technology bubbles have followed.

But “Is AI a bubble?” is almost too simple a question.

A more useful one is what exactly the market is pricing in.

AI can simultaneously represent a genuine technological transition and attract expectations that outrun current commercial reality.

Infrastructure complicates the picture further. AI growth depends on physical constraints that receive far less attention than models themselves: available computing capacity, electricity, construction, networking and the economics of running increasingly demanding systems.

In a recent FITT discussion, Tara Canham examined the AI investment boom through this wider economic and infrastructure lens. The discussion matters because decisions are already being made on the assumption that AI demand will continue expanding.

Executives have to decide where to invest. Founders have to decide which markets will still exist once the current enthusiasm settles.

Investors have to distinguish durable infrastructure demand from temporary exuberance. Engineers, meanwhile, eventually have to build the systems everyone else is forecasting.

Those perspectives do not always produce the same answer and that is precisely why the question becomes interesting.

What changes when AI can act in the physical world?

Some of the most consequential AI developments may eventually happen far away from a chatbot window. Consider a drone that loses access to GPS and radio communication. Traditional remote operation becomes extremely difficult. An autonomous system, however, may be able to continue navigating using onboard intelligence, local sensors and a previously defined mission.

That possibility formed the basis of another recent conversation initiated by Alexandru Parasca, focused on autonomous drones and the growing capabilities of Physical AI.

Industrial robots, autonomous vehicles, inspection systems and other intelligent machines increasingly need to interpret their surroundings and respond without constant human input.

A poor chatbot response can be corrected. A poor decision made by a machine moving through a warehouse, factory or public environment has physical consequences.

Autonomy therefore forces questions about reliability, edge computing, perception, safety and human oversight into the same discussion as artificial intelligence, and it makes the relationship between human and machine intelligence considerably more tangible.

How much autonomy should a system receive? When should a human remain in control? What level of uncertainty is acceptable before a machine must stop?

The useful part starts when somebody disagrees

There is a connection between cognitive debt, AI investment and autonomous machines: none of them can be understood especially well through one professional lens.

An engineer may see a technical constraint that an investor misses. A researcher may recognize that the evidence behind a popular claim is weaker than it appears. An executive may understand the organizational consequence of a technology that works perfectly well in a laboratory. Someone working directly in an industry may identify a practical problem that changes the entire premise.

That interaction is what Future Intelligence Think Tank is designed to encourage.

The community now includes more than 900 professionals, with conversations spanning human and artificial intelligence, responsible AI, Physical AI, emerging technology and the future of work.

The goal is to keep asking questions that remain interesting after the headline disappears.

The Future Intelligence Think Tank website brings together the community’s purpose, selected conversations and current insights. It offers a public introduction for readers who want to understand the initiative before joining the discussion.

Future Intelligence Think Tank is open to professionals who want to examine questions like these with people working in research, technology and business. Join the LinkedIn group to read the discussions, challenge an argument or introduce a question of your own.

Join the discussion on LinkedIn