Hallucinations, bias and black boxes: the AI risks leaders need to understand

Artificial intelligence is finding its way into an ever-growing number of business processes. Employees use AI to write, analyse, summarise, search for information and generate decision support. Many organisations are also experimenting with AI solutions that integrate with internal systems, documents and workflows.

For many leaders, AI feels primarily like a new type of software. But this is precisely where much of the challenge lies. AI works in a fundamentally different way from most of the systems organisations are accustomed to managing and controlling.

Traditional software follows rules. AI works with probabilities. Traditional systems are built to be predictable. AI is built to handle uncertainty. This gives rise to risks that many leaders have not previously had to contend with.

Most IT systems are deterministic. This means that the same input normally produces the same output.

If a customer orders a product, the order is recorded. If an invoice is overdue, a reminder is sent. If a user lacks access, they cannot log in. The system does what it is programmed to do.

Modern AI works differently. A language model is not built around fixed rules for every possible situation. It is trained on enormous volumes of text and calculates which responses are statistically most likely based on what it has learned.

The result is that AI can solve tasks it has never explicitly been programmed to handle. At the same time, its behaviour becomes less predictable. Two people can ask roughly the same question and receive different answers. Small changes in phrasing or context can significantly affect the outcome.

This is not necessarily a flaw in the technology. It is a characteristic of how modern AI works. But it means that AI must be assessed differently from traditional systems built around fixed rules and predictable processes.

One of the most widely discussed risks of generative AI is hallucinations.

The term describes situations where an AI model produces information that appears plausible and convincing but is actually incorrect. The model can fabricate sources, references, events, quotations or facts without understanding that the information is false.

For many leaders, this is an unfamiliar type of error. When an accounting system calculates the wrong VAT, it is perceived as a system fault. When an AI model presents incorrect information, it can do so in a way that appears both logical, well-structured and professionally credible.

The problem is therefore not simply that AI can be wrong. The problem is that the errors are often difficult to detect.

This is because language models are not built to assess what is true. They are built to generate the most probable answer based on the patterns they learned during training. In most cases, this produces impressive results. But when the model lacks information or misunderstands the context, it can produce answers that appear correct without being so.

AI models are a product of the data they learn from.

If the underlying data is incomplete, outdated or contains systematic biases, this will often affect the results the model produces. This applies both to general language models and to organisation-specific AI solutions that are trained or fine-tuned on internal data.

Bias is one of the most commonly used terms in AI risk. It describes situations where a model systematically favours or discriminates against certain groups, perspectives or outcomes.

In practice, the challenge is often more complex than the model simply being “biased”. The model largely reflects the patterns it finds in the data it has been exposed to. If historical data contains biases, AI can perpetuate or amplify them.

Data quality is therefore not only about having correct data. It is also about representativeness, relevance and context. Two models using the same technology can produce very different results depending on the data they are built on.

In many organisations, it is important to be able to explain why a decision was made. This is particularly true in quality, audit, compliance, information security and other areas where traceability is central. AI challenges this principle.

Many modern models function as what is often referred to as a “black box”. You can observe which data goes in, and you can see the result that comes out. But it is not always possible to explain exactly how the model arrived at its conclusion.

This does not mean that AI is necessarily uncontrollable. But it does mean that explainability is often weaker than in traditional rule-based systems.

The more advanced the model becomes, the more difficult it can be to understand which factors influenced the result. For organisations that depend on documentation, audit trails and verifiability, this represents a new type of challenge that many have not previously encountered.

Perhaps the most underestimated AI risk is not about algorithms or technology. It is about people.

Automation bias is an established term in decision support research, describing people’s tendency to place excessive trust in recommendations from technological systems.

This phenomenon is not new. It has been observed in everything from aviation to medical diagnostics. But generative AI makes the issue more pressing than ever.

When an AI model delivers fast, well-formulated and apparently intelligent responses, it is natural to attribute high value to its recommendations. Over time, people can become less critical and spend less effort on their own assessments.

The risk is therefore not necessarily that AI makes decisions independently. The risk is that people gradually begin to trust the technology without carrying out the professional quality assurance the situation demands.

The better AI becomes at communicating, the more important it becomes to remember that a persuasive answer is not necessarily a correct one.

Artificial intelligence represents a significant technological leap, but it also introduces a risk landscape that differs from that of traditional software.

Hallucinations, bias, lack of explainability and automation bias are not necessarily signs that the technology is performing poorly. On the contrary, they are often a consequence of how modern AI is built and why it is so powerful.

For leaders, the task is therefore not to become experts in machine learning or neural networks. It is to understand that AI has different characteristics from the systems organisations have traditionally worked with.

The better one understands these characteristics, the better placed one is to assess both the opportunities and the limitations the technology brings.

We help organisations with advisory services, establishment and further development of AI management systems. Get in touch to find out how we can help you!