View source for Choosing a method for sensitivity analysis

Doing sensitivity analysis is essential for testing a model and getting insights into what assumptions matter to the results and why.  Analytica offers a rich variety of methods for sensitivity analysis. This page gives an introduction to these methods and their pros and cons. It offers a guide in choosing methods to meet your needs.

 <div style="column-count: 2;-moz-column-count:2;-webkit-column-count:2">
__TOC__
</div>

Expert modelers know that sensitivity analysis is an essential step to test a model, assess its accuracy, and understanding its behavior and implications.  A common mistake by novices is to spend so much time building a model that they have little time left for performing systematic sensitivity analysis. So they -- and their clients -- miss out on the most useful insights from the modeling process.  Experts usually start by building a simple prototype of the model. They then use sensitivity analysis to discover which input values and parts of the model make most difference to the results, and so are most worth expanding. This tells them  development to focus on the assumptions and model elements that make the most difference to the results.

There are several methods of sensitivity analysis, which vary in their degree of sophistication and the effort needed to perform them.  This page describes and compares these methods, and explains how to use them in Analytica. Some are built in to Analytica. Others you can add in from libraries, including the Sensitivity analysis lib.ana and Value of Information lib.ana.

== Parametric analysis with a single input  ==

Perhaps the simplest approach to sensitivity analysis in Analytica is what we call parametric analysis: You can replace the definition of any scalar input variable, whether a decision, chance, or normal variable, by a list of numbers. In its parent diagram, select the node of a variable, X, and show its definition in the Attribute pane at the bottom of the diagram window.  From the x+y menu, select List of Numbers. You can then enter three or more alternative values, typically starting from the lowest plausible value up to its highest plausible value.   
Now look at the result, say the NPV in this case. Assuming the result actuall depends on the variable you have varied parametrically, the result will display as a table or graph showing the effect of the parametric variations.  By looking at the graph, you can immediately see the nature of the relationship – whether an increase in X causes an increase or decrease in Y, whether the relationship is linear. If it is nonlinear, is the relationship monotonic, i.e. the slope is always positive or always negative, but with an increasing or decreasing slope  -- -i.e. does the sensitivity of Y to X increase or decrease with the value of X. If it is not monotonic, is there a minimum or maximum of Y over X.    If the relationship is complex, it maybe helpful to visualize it with a larger number of values for X, say 5 or even 10.

== Parametric analysis with multiple inputs ==

You can extend parametric analysis with two or more variables. Simple select a second scalar input variable, say Z, and replace that by a list of values in increasing order.  Now the graph of Y will show multiple lines, with different colors for each value of Z. Or you can pivot the graph to show lines of Y as a function of Z for each value of X.  If there are interactions between X and Z, the lines will not be parallel. You can see if there are synergistic effects, where the lines diverge, or counter-synergistic effects where the lines converge.
You can extend the parametric analysis with more than two variables, but it gets more challenging to interpret the results with additional dimensions.  To compare the results

== Tornado chart or Range Sensitivity Analysis  ==

For range sensitivity analysis, also known as a tornado chart, shows the effect on a result of interest of varying each uncertain input over their range, from low to high, while keeping the other variables at their mid or base value.  If you order the variables by width of the range on the result from high to low, you get a graph like this. Hence, the name “Tornado chart”: 

you start by identifying all the assumptions on input numbers which are uncertain. For each quantity, you provide a base or mid value, usually the most likely (mode), and a low and high value that may enclose it.  The low and high values should be plausible extremes. If you think probabilistically, you might estimate them as the approximate 10th and 90th percentiles.  For a Boolean value that may be true or false (Yes or No), there are only two values, the mid value, and the other value.  

In the Sensitivity analysis library, you first create the list of uncertain inputs or assumptions as the definition of “Sensitivity_inputs. One way to do this is to draw influence arrows form each input node  into the Sensitivity_inputs index.
After selecting the list of sensitivity inputs,  click “Set inputs sensitivity table” button. It will show in the Sensitivity inputs table for each input, its parent module, units, and current value as its Base or Mid value. If the current value is an array, the Base will be a reference button linking to the array.
The next step is to

==Spider chart ==

A spider chart is like a Tornado chart in that it shows the effect on a result of varying each uncertain input over its range, while keeping the other inputs at their base value. But it plots the result on Y axis and variations of each input across the x-axis. You can also look at it as a way to superimpose one-way parametric analyses for many inputs on a single chart. The trick is to show percentage (or factor) changes from the mid value of each variable across the x axis.  The parametric lines for each input all cross at their mid values, so the result may look like a spider, the number and direction of its legs depending on the number of variables.

Typically, you can use 3, 5, or more variations for each input, say 50%, 75%, 100%, 125%, 150%.  You can use different values for each input, for example if some inputs can plausibly have larger percentage changes than others.
As in a parametric analysis, you can easily see whether the effect of each variable on the result is increasing or decreasing; linear, concave, or convex; and monotonic or with one (or more) maxima or minima.  It does not show the effects of interactions between inputs, since it varies only one input at a time.

== Importance analysis ==

Importance analysis shows how much of the uncertainty in a result is due to uncertainty in each input. It requires the uncertainty in each input to be expressed as a probability distribution. You can use it as a guide to set priorities on which uncertainties are worth most effort to try to reduce by obtaining more information. Or if that’s not possible, which are worth most effort in refining the estimated probability distribution, perhaps by doing more background research and/or obtaining estimates from more experts. It uses the rank-order correlation of the result with each uncertain input as a metric of “importance”. 

Analytica has a built-in feature to perform importance analysis. You select the result variable of interest, and then select Make importance from the Object menu. It automatically finds as inputs all the chance variables defined by a probability distribution and generates an Importance variable that contains the rank correlation of the result with each input.

==Decision switchover graph ==

Switchover analysis finds the value of an uncertain parameter at which the preferred decision switches from one option to another.  You must specify a discrete decision variable with two or more options, an objective to minimize or maximize, and an uncertain input parameter, X. The chart graphs the value of the objective (or expected value if you have probability distributions) for each decision option as X varies from its low to high value.   If the lines for each decision cross, the X value is where the preferred decision with highest value of the objective changes from one to another.

==Decision heat map ==

A decision heat map is a way to visualize which decision to choose as a function of two input parameters. The color in each rectangle identifies which option has the highest value for a discrete decision with two or more options. The vertical and horizontal axes plot values of the two parameters. For example, this graph shows which type of car, standard gasoline engine, hybrid, or battery electric vehicle, has the lowest total cost of ownership (NPV) for each combination of annual miles driven, and price of gasoline.

==== Expected value of information (EVI, EVPI, and EVSI) ====

==== Expected value of including uncertainty (EVIU) ====
{| class="wikitable"
!Method
!Values per variable
!Needs Decisions
and Objective
!Type
!Number of 
computations
|-
|One-way parametric analysis
|Two or more
|
|One at a ime
|v
|-
|Multi-way parametric analysis
|Two or more for each parameter
|
|
|n x v
|-
|Tornado or range sensitivity
|Range with low, mid, and high value
|
|
|1 + 2n
|-
|Spider chart
|Three or more
|
|
|1 + 2n
|-
|Decision switchover
|
|Yes
|
|
|-
|
|
|
|
|
|-
|
|
|
|
|
|-
|
|
|
|
|
|-
|
|
|
|
|
|}
Computational effort

Each method requires multiple evaluations of the result variable of interest, and so of some or all of the model.  The rightmost column shows the number of computations, where 
* v is the number of values for each input value, typically 2 to 5, although you may use more.
* n is the number of input or chance variables, 
* m the Monte Carlo sample size.