Choosing a method for sensitivity analysis

Sensitivity analysis is essential part of developing a model and generating useful insights. Every model makes assumptions and uses estimates of key input variables. Sensitivity analysis shows you much each assumption matters to the results and why. It offers a guide to where it is most worthwhile to expand the model or obtain more data. Analytica offers a rich variety of methods for sensitivity analysis. This page gives an overview of these methods and a guide to help you choose the methods most appropriate to your project.

Expert modelers know that sensitivity analysis is an essential step to test a model, assess its accuracy, and understanding its behavior and implications. A common mistake by novice modelers is to spend so much time building a model that they have little time left for performing systematic sensitivity analysis. The result is that novices -- and their clients -- miss out on the most useful insights from the modeling process. They often end up with a model that has too much detail in some areas, but not enough on other areas that matter most to the decision maker. Experts usually start by building a simple prototype of the model. They then use sensitivity analysis to discover which estimated inputs and parts of the model have the largest effect on the results. This lets them discover where it is most worthwhile to spend more effort to obtain better estimates or expand the model.

Sensitivity analysis is also a valuable way to test whether your model is working they way you expect. You may find that an input assumption that you thought unimportant has a large effect on the results -- or conversely, that something you expected to be important has little effect -- or even that it changes the result in the "wrong" direction. Examining why a sensitivity isn't what you expected will often help you identify and fix a bug in the model. Sometimes, you'll find the model is correct, but there's a bug in your thinking. That's often a potent source of new insight to better understand the model and the system you are modeling.

There are a wide variety of methods sensitivity analysis. They vary in the effort to apply them, and the number of times you need to run the model. Some are designed for models that encode uncertainties with probability distributions. And some require models with explicit decisions and objectives. This page gives an overview of each method with guidelines for when it is most suitable, plus links to pages for more details where needed. See the bottom for a summary table for comparisons.

Parametric analysis for one variable

Perhaps the simplest approach to sensitivity analysis in Analytica is what we call parametric analysis: You can replace the definition of any scalar input variable, whether a decision, chance, or normal variable, by a list of numbers. In its parent diagram, select the node of a variable, X, and show its definition in the Attribute pane at the bottom of the diagram window. From the x+y menu, select List of Numbers. You can then enter three or more alternative values, typically starting from the lowest plausible value up to its highest plausible value. Now look at the result, say the NPV in this case. Assuming the result actuall depends on the variable you have varied parametrically, the result will display as a table or graph showing the effect of the parametric variations. By looking at the graph, you can immediately see the nature of the relationship – whether an increase in X causes an increase or decrease in Y, whether the relationship is linear. If it is nonlinear, is the relationship monotonic, i.e. the slope is always positive or always negative, but with an increasing or decreasing slope -- -i.e. does the sensitivity of Y to X increase or decrease with the value of X. If it is not monotonic, is there a minimum or maximum of Y over X. If the relationship is complex, it maybe helpful to visualize it with a larger number of values for X, say 5 or even 10.

Parametric analysis with multiple inputs

You can extend parametric analysis with two or more variables. Simple select a second scalar input variable, say Z, and replace that by a list of values in increasing order. Now the graph of Y will show multiple lines, with different colors for each value of Z. Or you can pivot the graph to show lines of Y as a function of Z for each value of X. If there are interactions between X and Z, the lines will not be parallel. You can see if there are synergistic effects, where the lines diverge, or counter-synergistic effects where the lines converge. You can extend the parametric analysis with more than two variables, but it gets more challenging to interpret the results with additional dimensions. To compare the results

Tornado chart or Range Sensitivity Analysis

For range sensitivity analysis, also known as a tornado chart, shows the effect on a result of interest of varying each uncertain input over their range, from low to high, while keeping the other variables at their mid or base value. If you order the variables by width of the range on the result from high to low, you get a graph like this. Hence, the name “Tornado chart”:

you start by identifying all the assumptions on input numbers which are uncertain. For each quantity, you provide a base or mid value, usually the most likely (mode), and a low and high value that may enclose it. The low and high values should be plausible extremes. If you think probabilistically, you might estimate them as the approximate 10th and 90th percentiles. For a Boolean value that may be true or false (Yes or No), there are only two values, the mid value, and the other value.

In the Sensitivity analysis library, you first create the list of uncertain inputs or assumptions as the definition of “Sensitivity_inputs. One way to do this is to draw influence arrows form each input node into the Sensitivity_inputs index. After selecting the list of sensitivity inputs, click “Set inputs sensitivity table” button. It will show in the Sensitivity inputs table for each input, its parent module, units, and current value as its Base or Mid value. If the current value is an array, the Base will be a reference button linking to the array. The next step is to

Spider chart

A spider chart is like a Tornado chart in that it shows the effect on a result of varying each uncertain input over its range, while keeping the other inputs at their base value. But it plots the result on Y axis and variations of each input across the x-axis. You can also look at it as a way to superimpose one-way parametric analyses for many inputs on a single chart. The trick is to show percentage (or factor) changes from the mid value of each variable across the x axis. The parametric lines for each input all cross at their mid values, so the result may look like a spider, the number and direction of its legs depending on the number of variables.

Typically, you can use 3, 5, or more variations for each input, say 50%, 75%, 100%, 125%, 150%. You can use different values for each input, for example if some inputs can plausibly have larger percentage changes than others. As in a parametric analysis, you can easily see whether the effect of each variable on the result is increasing or decreasing; linear, concave, or convex; and monotonic or with one (or more) maxima or minima. It does not show the effects of interactions between inputs, since it varies only one input at a time.

Importance analysis

Importance analysis shows how much of the uncertainty in a result is due to uncertainty in each input. It requires the uncertainty in each input to be expressed as a probability distribution. You can use it as a guide to set priorities on which uncertainties are worth most effort to try to reduce by obtaining more information. Or if that’s not possible, which are worth most effort in refining the estimated probability distribution, perhaps by doing more background research and/or obtaining estimates from more experts. It uses the rank-order correlation of the result with each uncertain input as a metric of “importance”.

Analytica has a built-in feature to perform importance analysis. You select the result variable of interest, and then select Make importance from the Object menu. It automatically finds as inputs all the chance variables defined by a probability distribution and generates an Importance variable that contains the rank correlation of the result with each input.

Decision switchover graph

Switchover analysis finds the value of an uncertain parameter at which the preferred decision switches from one option to another. You must specify a discrete decision variable with two or more options, an objective to minimize or maximize, and an uncertain input parameter, X. The chart graphs the value of the objective (or expected value if you have probability distributions) for each decision option as X varies from its low to high value. If the lines for each decision cross, the X value is where the preferred decision with highest value of the objective changes from one to another.

Decision heat map

A decision heat map is a way to visualize which decision to choose as a function of two input parameters. The color in each rectangle identifies which option has the highest value for a discrete decision with two or more options. The vertical and horizontal axes plot values of the two parameters. For example, this graph shows which type of car, standard gasoline engine, hybrid, or battery electric vehicle, has the lowest total cost of ownership (NPV) for each combination of annual miles driven, and price of gasoline.

Expected value of information (EVI, EVPI, and EVSI)

The expected value of information (EVI) measures the increase in expected value due to obtaining information about an uncertain quantity. The expected value of perfect information (EVPI) is the increase in expected value if we learn the exact value of a quantity and so eliminate our uncertainty. To use these measures, the model must include explicit decision variable(s), and a numerical objective that defines the value or utility that you are trying to maximize (or loss to minimize). The uncertainties must be expressed as probability distributions.

It may sound paradoxical that you could estimate the value of information before you know the actual information -- e.g. the value of an uncertain quantity X. The key is that it's the expected value of information on -- the expectation of the change in value due to selecting a better decision after learning the value of X, taking the expectation over the probability distribution expressing the uncertainty about X.

See Expected value of information -- EVI, EVPI, and ESVI for details.

Expected value of including uncertainty (EVIU)

Comparison table of methods

Method	Values per variable	Needs Decisions and Objective	Type	Computational complexity
#Parametric analysis for one variable	Two or more		One at a time	v
#Multi-way parametric analysis	Two or more for each parameter			n x v
#Tornado or range sensitivity	Range with low, mid, and high value			1 + 2n
#Spider chart	Three or more			1 + 2n
#Decision switchover	Two or more for a parameter	Yes		v
#Decision heat map	Two or more			v*n
#Importance analysis	Probability distribution			m*n
#Regression	Probability distribution			m*n
#Expected value of information (EVI, EVPI, and EVSI)	Probability distribution		Yes	vnd^nd
#Expected value of including uncertainty (EVIU)	Probability distribution		Yes	vnd^nd

Computational complexity

Each method must evaluate the model to generate the result of interest multiple times. The rightmost column shows the number of times the model must be evaluated, where

v is the number of values for each input value, typically 2 to 5, although you may use more -- or probabilistic if the method requires the inputs to be specified as probability distributions, in which case it needs a Monte Carlo sample size of m.
n is the number of input or chance variables,
m the Monte Carlo sample size.
d is the number of decision variables
nd is the number of values for each decision variable