All Models Are Wrong – Limits, Risks and Practical Use

Models as Simplifications 

Models are essential tools in modern decision-making. They help simplify complex systems, turning uncertainty into structured analysis. In finance, risk management, policy and strategy, models support decisions that would otherwise be difficult to quantify. 

A useful way to understand models is through the idea of a “map versus the territory”. A map does not capture every detail of the real world. It highlights what is relevant for navigation. In the same way, a model represents selected aspects of reality to make them usable. 

This simplification is both a strength and a limitation. Models make complexity manageable, but they do so by excluding elements of reality. The key is to recognise that models are designed to support decisions, not to replicate the real world. 

The central message is clear: models are useful, but inherently limited. Understanding this distinction is critical for effective risk management. 

What Is a Model? 

Definition and Purpose 

A model is a representation of reality built on data, assumptions and mathematical relationships. It translates real-world processes into a structured framework that can be analysed. 

Models are used for multiple purposes: 

  • predicting future outcomes  
  • valuing assets or liabilities  
  • optimising decisions  
  • supporting risk assessment  

They provide a consistent way to interpret information and compare scenarios. 

Why Models Are Necessary 

Real-world systems are complex. Markets, organisations and economies involve multiple variables interacting in uncertain ways. Analysing this complexity without structure is not practical. 

Models provide that structure. They allow decision-makers to isolate key drivers, test scenarios and quantify potential outcomes. They also support consistency and comparability. Using defined methodologies ensures that decisions are based on a common framework rather than ad hoc judgement. 

Without models, decision-making would rely mostly on intuition, which is often insufficient in complex environments. 

Models as “Maps, Not the Territory” 

The Core Analogy 

The analogy of a map is useful. A map simplifies geography to help navigation. It removes unnecessary detail while preserving what is relevant. 

A model does the same. It simplifies reality to make it actionable. It focuses on selected variables and relationships to support analysis and decision-making. This simplification is intentional. A perfect representation of reality would be too complex to use. 

What Models Leave Out 

All models exclude elements of reality. 

They often struggle to capture non-linear behaviours, where small changes lead to disproportionate effects. They also simplify human behaviour, which is difficult to predict and influenced by perception and incentives. 

Feedback loops are frequently underrepresented. These can amplify or dampen effects over time. In addition, rare and extreme events are often underestimated or excluded due to limited data. 

These omissions are not errors; they are inherent to modelling. However, they create blind spots that must be recognised. 

Implications for Risk Management 

For risk management, the implications are significant. 

Models provide guidance, not truth. They offer a structured view of risk, but not a complete one. Treating model outputs as definitive increases exposure to unexpected outcomes. 

Over-reliance on models can create a false sense of certainty. Effective risk management requires combining models with judgement, challenge and alternative perspectives. 

The Role of Assumptions 

Assumptions Drive Outcomes 

Every model is built on assumptions. These include inputs, probability distributions, correlations and behavioural relationships. 

Results are highly sensitive to these assumptions. Changing a single parameter can materially alter outputs. This makes understanding assumptions as important as understanding results. 

Assumptions define the boundaries of the model. 

Hidden Assumptions 

Not all assumptions are explicit. Many are embedded in model design, data selection or methodology. 

These hidden assumptions can introduce bias. For example, historical data may reflect specific conditions that do not hold in the future. Simplifications may exclude relevant variables. 

Poor documentation increases the risk. When assumptions are not transparent, users cannot properly interpret results. 

Model Fragility 

Models can be fragile. Small changes in inputs or assumptions may lead to large differences in outputs. This fragility becomes more evident under stress conditions. Models calibrated on normal environments may not perform well during periods of disruption. 

Understanding where models breakdown is as important as understanding how they perform under standard conditions. 

Model Risk in Practice 

Definition of Model Risk 

Model risk is the risk of making incorrect decisions due to errors, limitations or misuse of models. It arises when model outputs are inaccurate, misunderstood or applied outside their intended scope. In risk management, this can lead to underestimation of exposure or inappropriate strategies. 

Sources of Model Risk 

Model risk can originate from several sources: 

Data limitations are a common issue. Incomplete, biased or outdated data affects model accuracy. 
Methodological errors can arise from incorrect assumptions, inappropriate techniques or flawed design. 
Misinterpretation occurs when users do not fully understand model outputs or limitations. 

These factors often interact, increasing overall risk. 

Real-World Examples 

During the global financial crisis, models underestimated correlations and extreme events. This led to mispricing of risk and insufficient capital buffers. 

Similarly, over-reliance on measures such as Value-at-Risk (VaR) created a narrow view of risk. Tail events and systemic interactions were not fully captured. These examples reinforce a key point: models are powerful tools, but they must be used with caution and critical judgement. 

Transparency Challenges 

Transparency is a core requirement for effective model use. Without it, model outputs cannot be properly understood, challenged or trusted. 

Complexity vs Understandability 

Models are becoming more complex. Techniques such as machine learning and artificial intelligence improve predictive power but reduce interpretability. 

This creates black-box risk. Outputs are generated, but the underlying logic is difficult to explain. Users may rely on results without understanding how they are produced. 

Complexity must be balanced with usability. A model that cannot be explained is difficult to manage. 

Communication Gaps 

Model outputs are often technical. Translating them into business terms is not straightforward. This creates a gap between modellers and decision-makers. Technical teams focus on methodology, while management focuses on outcomes and implications. 

Misalignment leads to misuse. Results may be over-simplified, misinterpreted or applied incorrectly. Clear communication is essential to ensure that outputs are understood and actionable. 

Governance Implications 

Lack of transparency affects governance. Additionally, validation becomes more difficult when models are complex. Oversight functions may struggle to assess assumptions, methodologies and limitations. 

This increases reliance on trust rather than control. Strong governance requires explainability, clear documentation and independent review. 

Robustness Challenges 

Robustness determines whether a model remains reliable under different conditions. Weak robustness increases the risk of failure when it matters most. 

Sensitivity and Stability 

Model outputs often depend heavily on assumptions. Small changes in inputs can produce large variations in results. This sensitivity creates instability, particularly in uncertain environments. 

Under stress conditions, models calibrated on historical data may no longer perform as expected. Robustness requires understanding how models behave beyond normal scenarios. 

Overfitting and False Precision 

Models can fit historical data too closely. This is known as overfitting. 

While results may appear accurate, they reflect past patterns rather than future uncertainty. This creates an illusion of precision. Decision-makers may place undue confidence in outputs that are inherently uncertain. Recognising the limits of accuracy is essential. 

Stress Testing and Scenario Analysis 

Robust models are tested beyond standard conditions. 

Stress testing explores extreme but plausible scenarios. Scenario analysis examines how outcomes change under different assumptions. 

These techniques reveal weaknesses and highlight potential vulnerabilities. They shift the focus from prediction to preparedness. 

From Models to Decision-Making 

The value of a model lies in how it supports decisions. Models should inform judgement, not replace it. 

Decision Support 

Models provide inputs to decision-making. They structure analysis, quantify uncertainty and compare scenarios. However, they do not determine outcomes on their own.

Final decisions require judgement, context and experience. Treating model outputs as definitive removes critical thinking from the process. 

Triangulation of Approaches 

No single model captures all dimensions of risk. 

A more robust approach combines: 

  • multiple models with different assumptions  
  • expert judgement  
  • qualitative insights  

This triangulation reduces reliance on any single perspective. It improves resilience and supports more balanced decisions. 

Building Model-Aware Organisations 

Organisations must understand how models work and where they fail. 

This requires training non-technical stakeholders to interpret outputs and challenge assumptions. Awareness of limitations should be embedded in governance and culture. 

Model-aware organisations use models effectively without becoming dependent on them. 

Best Practices for Managing Model Risk 

Managing model risk requires structured practices and strong governance. 

Clear documentation of assumptions ensures transparency. Users must understand how results are generated and what limitations apply. 

Independent validation provides challenge. Separate teams should review methodology, data and outputs to identify weaknesses. 

Regular review and recalibration ensure that models remain relevant as conditions change. Static models become outdated quickly. 

Stress testing should be integrated into model use. Testing extreme scenarios highlights vulnerabilities that standard analysis may miss. 

Finally, strong governance frameworks are essential. Defined roles, responsibilities and oversight mechanisms ensure that models are used appropriately and consistently. 

Useful, Not Perfect 

Models are essential tools for managing complexity. They support analysis, improve consistency and inform decisions. However, they are not perfect representations of reality. Their limitations must be recognised and managed. 

Transparency and robustness are critical. Without them, models create false confidence and increase risk. Judgement remains central. The best decisions combine models, data and critical thinking. 

Shopping Basket
WordPress Cookie Notice by Real Cookie Banner