AI safety isn’t only about preventing systems from failing. It is also about understanding what happens when AI systems work exactly as designed, but optimise for the wrong things.

Businesses increasingly use artificial intelligence to personalise recommendations, content, prices, advertising and customer experiences.

Usually, the objective sounds reasonable: make the experience more relevant.

But hyper-personalisation creates a less obvious AI safety risk.

What happens when every customer begins experiencing a slightly different version of reality?

The problem with hyper-personalisation

Most AI systems are given measurable objectives.

Increase engagement. Improve conversions. Keep people on a platform longer. Recommend something they are likely to buy.

These metrics make sense individually.

The problem begins when optimisation becomes too aggressive.

Imagine two customers interacting with the same company.

One receives one price, another receives a different price.

One receives certain information, another receives something completely different.

One is encouraged towards a decision based on what an algorithm has learned about their behaviour or vulnerabilities.

Personalisation has now become something else.

It has become fragmentation.

And that creates important questions about AI safety, fairness and trust.

AI safety is about more than technical failures

AI safety conversations often focus on whether an AI system will make a serious mistake.

That matters.

But businesses should also consider the consequences of systems that are functioning correctly according to their objectives.

An algorithm optimised for engagement may discover that controversial content keeps people watching.

An AI sales system optimised for conversion may discover increasingly effective ways of influencing individual customers.

A pricing algorithm may learn exactly how much different customers appear willing to pay.

None of these systems necessarily needs to “break” to create risk.

Sometimes the risk comes from the objective itself.

When one metric becomes too powerful

The philosopher Gilles Deleuze offers an interesting way to think about this problem.

Society is made up of interconnected systems: markets, institutions, communities, workplaces and relationships.

These systems depend on many different human values, including fairness, choice, trust and accountability.

Algorithms, however, often operate around something much simpler:

a metric.

Clicks.

Screen time.

Conversion.

Revenue.

Imagine designing an entire city around one metric: traffic speed.

You could make every decision based on getting cars from A to B faster.

You might succeed.

But would you create a better city?

Probably not.

You may lose green spaces, pedestrian areas, local communities and other things that are difficult to express through a single metric.

AI creates a similar challenge. What is optimal for an algorithm isn’t necessarily optimal for the wider system in which it operates.

That is an AI governance problem.

But it is also an AI safety problem.

Why businesses should care

1. Trust can disappear quickly

Personalisation depends on data and inference.

Customers may accept this when they believe they are receiving something useful.

But there is a line between relevance and manipulation.

When people discover that an algorithm is altering prices, information or choices according to what it knows about them, the relationship can change.

The customer starts asking:

“Is this AI helping me, or steering me?”

Responsible AI therefore requires businesses to think about human agency, not simply performance.

2. AI governance becomes essential

Organisations need to understand what their AI systems are actually being rewarded for.

A surprisingly useful governance question is:

What behaviour are we encouraging this AI system to optimise?

If the answer is only conversion, engagement or cost reduction, something may be missing.

A responsible AI programme should also consider factors such as:

• fairness
• transparency
• accountability
• customer choice
• data protection
• safety
• long-term trust

This is why AI governance needs to develop alongside AI deployment.

Brink Think Labs’ existing AI-readiness work similarly points towards governance-first adoption, including a practical Discover → Govern → Pilot sequence rather than uncontrolled deployment.

The same problem can happen inside organisations

This isn’t only a customer-facing risk.

Generative AI can fragment reality inside businesses too.

Employees may use different AI assistants.

Departments may connect AI to different data sources.

Unofficial or “shadow AI” tools may operate outside normal governance.

One employee receives one answer while another receives something completely different.

Eventually organisations can face a basic problem:

Which version of the information should people trust?

Good AI safety therefore depends on more than choosing a safe model.

It requires organisations to think about:

• AI governance
• data governance
• AI literacy
• workforce readiness
• accountability
• information quality

What should organisations do?

Responsible AI doesn’t require abandoning personalisation.

It requires boundaries.

1. Don’t optimise for one metric

Measure commercial performance, but consider what happens around it.

Ask:

What could the system sacrifice in order to improve this metric?

That one question can reveal surprisingly important risks.

2. Make important AI decisions auditable

Businesses should be able to understand:

• What did the AI do?
• What information influenced the decision?
• What was the system trying to optimise?
• Who is accountable?
• Can a human review the outcome?

The more significant the decision, the more important those questions become.

3. Keep humans and society in the loop

Decisions about acceptable AI behaviour shouldn’t belong only to engineers.

Product teams, data leaders, risk teams, employees, legal professionals and affected users can see different consequences.

This leads towards a broader idea: Society-in-the-Loop.

Instead of asking only whether an AI system performs technically, we should ask whether its behaviour remains acceptable to the society, organisation or community in which it operates.

The bigger question for AI safety

The next generation of successful AI products won’t simply be judged by how effectively they capture attention or maximise conversion.

They will increasingly be judged by whether people trust them.

That means businesses adopting AI should move beyond one question:

Can we optimise this?

And start asking another:

Should we optimise it this way, and what happens to people when we do?

Because AI safety is ultimately not just about keeping artificial intelligence under control. It is about ensuring that the systems we build continue to work for the people and societies around them.