View source for Writing Array-Abstractable Definitions

[[Category:Concepts]]

== The Benefits of Array Abstraction ==

One of the most powerful and useful features of Analytica is ''Intelligent Array Abstraction''™.
When we give the variable <code>Profit</code> the definition
:<code>Revenue - Expenses</code>

Analytica automatically carries through any dimensions of <code>Revenue</code> and <code>Expenses</code>, seamlessly making the correspondence if the same dimension occurs in each. If the <code>Revenue</code> and <code>Expenses</code> variables are broken down by (i.e., indexed by) ''department'' and ''project'', the resulting <code>Profit</code> will also be broken down across these two dimensions. A conventional programming language would force the modeler to explicitly loop over these dimensions when computing <code>Profit</code>. A spreadsheet would require the modeler to explicitly copy the same formula to every cell in the two-dimensional grid. In each case, handling these dimensions requires the attention and effort of the modeler that is distracting, and fundamentally unnecessary, since the relationship between <code>Profit</code>, <code>Revenue</code> and <code>Expenses</code> is quite separate from whatever the dimensions happen to be in a particular model. [[Array Abstraction]] refers to the principle of separating the fundamental relationships between quantities (variables) and their dimensions.  ''([[Intelligent Arrays]] is a trademark of Lumina Decision Systems.)''

Once you have mastered the basics of array abstraction, it is hard to imagine going back to a more primitive modeling environment (such as a spreadsheet or standard programming language) without such a capability for any non-trivial modeling application. Array abstraction provides a flexibility that goes hand-in-hand with the modeling process. When we begin prototyping a model, the optimum dimensions to include are seldom clear. The modeling process usually involves the elucidation of relevant variables, specification of fundamental relationships, and experimentation with tradeoffs between richness of knowledge to build off of, computational complexity, and levels of detail. The determination of the appropriate dimensionality for key variables is a task that usually falls most naturally after elucidation of relationships between variables. Array abstraction allows the modeler to do just this – once an abstractable model is in place, dimensions can be added and removed with little effort. This can be contrasted with spreadsheets and conventional programming languages where one must commit to the dimensionality of variables early in the modeling process, and because every computational step involves explicit defined iteration over the specified dimensions, altering the dimensionality of variables tends to be extremely tedious.

Array abstraction also promotes correctness. Opportunities to make errors when copying cell references in a spreadsheet abound, are quite common in practice and are hard to verify; likewise, it is quite easy to make mistakes when writing explicit looping constructs in
conventional programming languages, such as accidentally confusing the correspondence between dimensions. Separated from the clutter of all these looping constructs, the clarity of variable definitions in Analytica is far more apparent. Furthermore, array abstraction leaves one with a sense of confidence that solid portions of the model are and will remain correct, regardless of other modifications or dimensional adjustments that may be made outside of that sub-model.

In Analytica, array abstraction enables a number of other important capabilities. A major one of those is the uncertainty analysis that is built into Analytica. Uncertainty analysis in Analytica operates by adding a dimension to the uncertain variables in your model, named the [[Run]] dimension. Since your (presumably abstractable) model describes the relationships between variables, what is true without that dimension should also be true with that dimension, and thus Analytica can seamlessly propagate that dimension through the computations, even though as a modeler, you might not have conceptualized your variables up front as being dimensioned by [[Run]].

Another important capability arising from array abstraction is ''What If'' analysis. Adding new dimensions to key input variables enables one to simultaneously explore and compare multiple scenarios, without having the logic of your model. “What-if” analyses are also often the bases decision making by exploring spaces of possible decisions.

== Non-abstractable Definitions ==
Most Analytica expressions (relationships) that one would write in a variable’s definition field will be “abstractable”, that is, it will be possible to arbitrarily add and adjust dimensions to input variables without altering the correctness of the result. Analytica will propagate any of these dimensions through the variables without any extra thought on the part of the modeler. 

However, there are some notable exceptions, where it is possible to write a definition that is not abstractable. When such a definition is written, the flexibility and benefits of array abstraction may be lost. A key area of the more advanced Analytica modeling skill set is being able to recognize such cases and to know how to “correctly” express the desired relationships in alternative, abstractable ways.

Expressions that are limited in their ability to be array abstracted often fall into one of three groups: Expressions that introduce new dimensions, dimension-reducing expressions that that don’t name the relevant dimension, and convergence/iterative/recursive algorithms where the total number of iterations depends on the computation itself and cannot be known in advance.
Relationships falling in these categories can almost always be written in an array abstractable fashion if the modeler pays appropriate attention to how the expression is written.

An example of a non-abstractable expression in the first group of functions that introduce a new dimension is the Sequence function, e.g., <code>1..N</code> or <code>Sequence(1, N)</code>. When N is “atomic” (i.e., not an array, containing no dimensions), this expression evaluates just fine, but ''N'' is changed to a list of numbers, e.g., <code>[10, 20, 30]</code>, the result would be a ”non-rectangular” array, which is not allowed
as an Analytica value. The presence of a [[Sequence]] function embedded in a more complex expression could make the entire expression non-abstractable if appropriate modeling style is not exercised, as in the following example of an expression to compute the factorial of a number ''N'':
:<code>Product(1..N)</code>

Although there is no inherent reason why the factorial of a number should not be abstractable, this particular way of writing the factorial would not be abstractable. Other functions also falling into the first group of expressions that (may) introduce dimensions include [[Subset]], [[SplitText]], [[SortIndex]] (if the second parameter is omitted), [[Concat]], [[CopyIndex]], [[IndexNames]], and [[Unique]].

An example from the second group of expressions, dimension-reducing expressions not naming the relevant dimension, is the expression: [[Sum]](A), where the second index parameter of [[Sum]] is omitted. This form sums over the “outer” dimension of <code>A</code>. As a modeler, there are generally only a couple of instances where you know what the outer dimension is – the case where <code>A</code> is guaranteed to be only one-dimensional, and the case where <code>A</code> contains an implicit dimension (which will always be the outer dimension). If you assume any other dimension is the outer dimension, you may be unpleasantly surprised when new dimensions are introduced. Thus, as a matter of style, it is best to always include the relevant dimension as a second parameter to the [[:Category:Array-reducing functions|array reducing functions]] in all other cases, and you’ll steer clear with abstractability problems from this class of expressions. The array reducing functions with optional index parameters include
[[Sum]], [[Product]], [[Max]], [[Min]], [[Average]], [[Rank]], [[ArgMax]], [[ArgMin]], [[SubIndex]], [[ChanceDist]], [[CumDist]], and
[[ProbDist]]. Also, the function [[Size]] is essentially also in this category, although there is no concept of relevant index there – basically, [[Size]] should only be used on parameters restricted to be one-dimensional, index parameters being a very important special case.

Finally, the third class of expressions are iterative or recursive convergence algorithms with a dynamically determined number of iterations. This group almost always involves [[While]] or [[Iterate]] functions. For example, another non-abstractable expression for factorial of an integer ''N'' is:

:<code>Var a := 1;</code>
:<code>Var]] fact := 1;</code>
:<code>While]] a < N  Do (</code>
:<code>a := a + 1;</code>
:<code>fact := fact * a</code>
:<code>)</code>

As written, this expression assumes ''N'' to be atomic, and will not evaluate if ''N'' is an array. This expression will be discussed further below when discussing Horizontal and Vertical Array Abstraction. The fundamental problem with abstracting over a [[While]] is that the number of iterations may differ from cell-to-cell, so a single evaluation of the [[While]] loop does not account for the cell-by-cell differences in evaluation flow. It is important to realize that if you are iterating over a dimension, either implicitly or via [[For]] or [[Using..Do|Using..In..Do]], because the number of iterations required is fixed in advanced, abstraction limitations generally are not introduced.

== Horizontal and Vertical Array Abstraction ==
There are two conceptual models for how Analytica actually carries out array abstraction when evaluating expressions, which we will (somewhat arbitrarily) call the horizontal and vertical conceptualizations. For illustrative purposes, suppose you have a model, call it <code>F(X)</code>, that computes a result based on the input variable <code>X</code>, and suppose we add a new dimension <code>I := 1..100</code> to <code>X</code> and compute our result. The horizontal conceptualization is that Analytica evaluates <code>F(x)</code> one hundred separate times, once for each value of <code>X</code>, and collects the result for display at the end. The vertical conceptualization is that the array <code>X</code> is passed through each level of evaluation as an array, vertically through the call stack, until the array-valued value bottoms out at a built-in operator or function. In most instances, either conceptualization is perfectly fine, and knowledge of which method Analytica actually implements is unnecessary. Indeed, the two conceptualizations are functionally equivalent (i.e., produce the same net result) for any fully abstractable expression. However, the expert Analytica modeler can make use of how Analytica abstracts to leverage horizontal or vertical evaluation as appropriate in the problem cases highlighted in the previous section in order to write abstractable variations of their expressions.

In the default case, Analytica uses vertical abstraction to evaluate an expression. Thus, when the user-defined function

:<code>MyFact(N) := Product(1..N)</code>

is called with <code>N:= [10, 20, 30]</code>, the array of values is passed into the definition, then into the parameter of Product, and finally into the second parameter of [[Sequence]], resulting in the evaluation of <code>Sequence(1, [10, 20, 30])</code>, which as discussed in the previous section, presents a problem. However, using the “atomic” qualifier on the parameter, e.g.,

:<code>MyFact(N : atomic]) := Product(1..N)</code>

instructs Analytica to abstract over the parameter of MyFact horizontally, so instead the expression <code>Product(1..10)</code> is evaluated, followed by the expression <code>Product(1..20)</code>, and finally the expression <code>Product(1..30)</code>. Each of these returns a scalar, which is then collected up and returned as <code>[3.6e6, 2.4e18, 2.7e32]</code>. So, although the expression [[Product]](1..N) is not vertically abstractable, it is horizontally abstractable across N. In fact, exerting control over when Analytica abstracts horizontally or vertically is a quite general tool for eliminating limitations on array abstractability.

Leveraging [[User-Defined Functions]] (UDFs) is usually the easiest and most efficient way to horizontally abstract. The qualifier atomic is generalized to any number of dimensions using the parameter syntax, e.g.

:<code>(x: Array[I, J, Time]; I, J: Index ; y: atomic)</code>

The list of indexes in square brackets indicates the “allowed” dimensions – any other dimension will be horizontally abstracted over before calling your function, so that inside your function body, you can assume that only these dimensions appear. The dimensions may be either global indexes (as [[Time]] is above), or other index parameters. The qualifier <code>atomic</code> is equivalent to <code>Array[ ]</code> – i.e., an array with zero dimensions.

If a parameter that is not indexed by <code>I</code> were passed to a parameter with the qualifier <code>Array[I]</code>, the value will be passed in without the index. A value not indexed by <code>I</code> is generally treated as a value that is constant across dimension I by Analytica, so there are few instances where this will matter. One way this is often leveraged is by declaring a parameter as <code>Array[Run]</code>, which allows the [[Run]] index, but doesn’t require it, for example, when the parameter is evaluated in Mid mode. In the case where you must have the dimension, the all qualifier can be added:

:<code>(x: Array All[I, J]</code>

The atomic and Array qualifiers on user-defined functions demonstrated above is a very convenient method for specifying horizontal array abstraction when a [[UDF]] is used. Within the body of definition (whether a variable, user-defined function, or other object), horizontal array abstraction controlled by specifying the allowed dimensions in a variable declaration ([[Var..Do]] or [[Using..Do]]). Two variations on syntax are

:<code>Var X[I, J,…] := expr;</code>
:<code>body1;</code>
:<code>body2;</code>
:<code>…</code>
:<code>bodyL</code>
or
:<code>Using X := expr In Each I, J,… Do body</code>

In the first syntax, the bracketed list of indexes specifies that array abstraction on the local variable <code>X</code> is to be done horizontally for any dimensions not listed in brackets. The local variable <code>X</code> has a lexical scope of <code>body1</code> thru <code>bodyL</code>, and each of those expressions to the end of the current lexical scope can assume that <code>X</code> is only indexed by those dimensions named. The [[Using..Do|Using..In Each..Do]] syntax accomplishes the same with a slightly different syntax, the first syntax having a more procedural flavor, the second a more functional-programming flavor and more consistent with the syntactic style prevalent in earlier versions of Analytica. In either case, the list of dimensions can be empty, which states that <ccode>X</code> should be horizontally abstracted all the way down to a scalar (equivalent to the atomic qualifier on a [[UDF]] parameter). Thus, another way of writing the factorial expression in a fully abstractable way is:

:<code>Using A := N In Each Do Product(1..A)</code>
or synonymously
:<code>Var A[ ] := N;</code>
:<code>Product(1..A)</code>

As another variation, consider the non-abstractable expression

:<code>Sum(0.5^(Min(B, I)..Max(B, I)))</code>

Here the assumption is that <code>B</code> is not indexed by anything other than <code>I</code>, so an abstractable version
of this expression would be

:<code>Var A[I] := B Do Sum(0.5^(Min(A, I)..Max(A, I)))</code>

Since each declaration loops independently, multiple dimensionality declarations can sometimes be inefficient. For example, in:

:<code>Var A[] := X;</code>
:<code>Var B[] := Y;</code>
:<code>F(A, B)</code>

if <code>X<code> and <code>Y</code> are both indexed by <code>I</code>, the first declaration will loop over <code>I</code>, and the second will loop over <code>I</code> again. The final result will utilize only the diagonal if this [[Size]](I)^2 computation. There are two ways to avoid this inefficiency. The easiest is to use a [[User-Defined function]] with a declaration <code>(A, B : atomic)</code>, which would loop over <code>I</code> only once. A more complex method, not requiring a [[UDF]], is to bundle the declaration using references, as in the following:

:<code>Index ABindex := [‘A’, ’B’];</code>
:<code>Var AB[] := Array(ABIndex,[\[]X, \[]Y]);</code>
:<code>Var A := #AB[ABIndex = ’A’];</code>
:<code>Var B := #AB[ABIndex = ’B’];</code>
:<code>F(A, B)</code>

Here the reference syntax, [[Using References|\[]X]], indicates that no indexes are to be “swallowed” by the reference. You would place any desired dimensions for <code>A</code> in the square brackets of the reference operator if <code>A</code> is not just atomic. The single declaration, <code>Var AB[]</code>, ensures that the variables all have their desired dimensionality in the expression body.

== The While Loop ==
When a [[While]] loop is evaluated with vertical array abstraction, the while-loop expression is evaluated only once (the body of that [[While]] is iterated 0 or more times, of course). This is generally problematic when abstractable dimensions are present, since each different column of the extra dimension may need to iterate the while-loop’s body a different number of times. Thus, forcing horizontal abstraction to atomic values for any variables appearing in the condition of the while loop is often the appropriate method for obtaining an abstractable while-loop expression. This method is illustrated by the following [[UDF]]:

:<code>Function MyFact(N: atomic) :=</code>
::<code>Var a := 1;</code>
::<code>Var fact := 1;</code>
::<code>While a < N Do (</code>
:::<code>a := a + 1;</code>
:::<code>fact := fact * a</code>
::<code>);</code>
::<code>fact</code>

The reader should note that the only change between this and the originally expression in Section 2 is the addition of the atomic qualifier on N in the [[UDF]]’s parameter list.

If the same snippet were to appear within a larger expression, and not necessarily within a [[UDF]], the horizontal array abstraction method would result in

:<code>Var m[ ] := N;</code>
:<code>Var a := 1;</code>
:<code>Var fact := 1;</code>
:<code>While ( a < m ) Do (</code>
::<code>a := a + 1;</code>
::<code>fact := fact * a</code>
:<code>);</code>
:<code>fact</code>

Another approach to dealing with the abstractability of [[While]] is to continue using vertical array abstraction, but to adjust the body expression to ensure that the extra iterations do not functionally affect the outcome. This is demonstrated with this abstractable version of the same example

:<code>Var a := 1;</code>
:<code>Var fact := 1;</code>
:<code>While Any ( a < N ) Do (</code>
::<code>a := a + 1;</code>
::<code>fact := If a <= N Then fact * a Else fact</code>
:<code>);</code>
:<code>fact</code>

The Any keyword following while allows the condition to be array-valued, but the while will continue until all elements are 0. The conditional update rule inside the [[While]] ensures that the extra iterations don’t impact the final result. The computational ramifications of the two methods may impact the modeler’s choice of which method to use. The core update operation (in this case, the multiplication) ends up being evaluated many extra times with vertical array abstraction, particularly if the number of iterations required differs wildly from element to element. However, this needs to be weighed against the greater efficiencies of vertical array abstraction and the fewer number of times the complete while loop expression needs to be evaluated. When the internal update operation is very expensive, horizontal array abstraction will usually be more efficient. There are also some cases where only one of these two techniques can be applied.

We should emphasize that the [[While]] loop is a very poor choice for computing a factorial, since the number of iterations is known in advance. It was chosen as a starting example solely for its ease in understanding. The more appropriate applications for [[While]] are iterative convergence or optimization algorithms where the number of iterations is determined dynamically.

Nevertheless, the two methods outlined above are the general techniques as the modeler’s disposal for obtaining array abstractable iterative expressions based on while. The following is an example of an advanced iterative convergence algorithm that has been made fully abstractable using the horizontal abstraction method. The example appears in the example models and libraries shipped with Analytica 3.x. It uses an iteration known as the power method (from an elementary Linear Algebra textbook) to compute the dominant Eigen vector of a square matrix <code>A</code>, dimensioned by equal-sized dimensions <code>I</code> & <code>J</code>. If there are additional dimensions (i.e., <code>A</code> is a set of matrices, or <code>A</code> contains a [[Run]] dimension), the expression abstracts over them. The reader should not worry about the specifics of this particular algorithm, but should take note of the first line, declaring <code>B</code> to horizontally array abstracted should ''A'' contain extra dimensions. The take away message being that a very sophisticated iterative convergence algorithm such as this, which on the surface appears quite incompatible with (vertical) array abstraction, requires only a single line (with a few variable name substitutions within) to convert it into a fully abstractable expression. Thus is the benefit of having a mastery of horizontal array abstraction control. The reader might also examine the [[While]] criteria, which is typical of a convergence algorithm. It is a case where we must guarantee that <code>B</code> is 2-D (so that <code>EigenVal</code> is scalar, and thus the condition to While is scalar), which is accomplished by horizontal array abstraction. Some auxiliary [[UDF]]s called within are not included here, but can be found in the example models shipped with Analytica.

:<code>/* Ensure that it array abstracts if A has other dimensions */</code>
:<code>var B[I, J] := A;</code>
:<code>/* First, generate a random first eigenvector */</code>
:<code>var EigenVec := Random_vector(J);</code>
:<code>var EigenVal := Rayleigh_quotient(B, eigenVec, I, J);</code>
:<code>var prevEigenVal := EigenVal - 1;</code>
:<code>var maxIter := 50;</code>
:<code>while (maxIter>0 and abs(prevEigenVal-EigenVal)>1e-10) Do (</code>
::<code>var xi := subscript(sum(B*EigenVec, J), I, J);</code>
::<code>EigenVec := xi/max(xi, J);</code>
::<code>prevEigenVal := EigenVal;</code>
::<code>EigenVal := Rayleigh_quotient(B, EigenVec, I, J);</code>
::<code>maxIter := maxIter - 1</code>
:<code>);</code>
:<code>EigenVec</code>

''Note: If you really want to compute the Eigen value/vector, use the built-in [[EigenDecomp]] function.''

As one final and very advanced example of a very sophisticated [[UDF]], I present the [[GoalSeek]] function. [[GoalSeek]] is essentially an equation solver based on a Newton-Raphson gradient descent method. It finds the (atomic) value for a variable <code>X</code> that makes the (atomic) variable <code>Y</code> equal to goal, where <code>Y</code> depends on <code>X</code> in the model. Extra dimensions may end up being introduced via array abstraction in a variety of ways, making a particularly challenging exercise to make this generalized library function fully array abstractable. For example, dimensions can be introduced via <code>X</code> (i.e., interpreted as different starting points for the search), into <code>Y</code> by other variables in the model (interpreted as different cases that must be solved), or into goal (interpreted as different cases that must be solved). Before examining the actual code for Goalseek, I will demonstrate an example of its usage.

Problem statement: <code>b</code> and <code>c</code> are uncertain quantities, and are related to <code>R</code> by the transcendental equation:

:<code>R2 = c e-bR</code>

The application requires us to treat the solution for <code>R</code> as an uncertain random variable. We would like to view the [[Mid]]-value of <code>R</code>, as well as the uncertainty graphs for <code>R</code> (e.g., [[PDF]], [[CDF]], [[Sample]], [[ProbBands]], etc). <code>C</code> and <code>b</code> may or may not have the [[Run]] dimension or other dimensions.

We model this with [[GoalSeek]] by first dividing by ''e-bR'' so all instances of <code>R</code> are on one side of the equation

:<code>R2 ebR = c</code>

which is now in a form appropriate for the [[GoalSeek]] function. An example of this model is as
follows:

:<code>b := Uniform(0, 1)</code>
:<code>c := Uniform(0, 5)</code>
:<code>r_guess := 5</code>
:<code>tst := r_guess^2 * exp(b*r_guess)</code>
:<code>R := GoalSeek(r_guess, tst, c) xs</code>

The following graph is the [[CDF]] and [[PDF]] of <code>R</code> computed from an uncertainty sample size of 1,000 points. This took 4½ minutes on a 1.9GHz Pentium 4 processor with 512M RAM.

:[[image:array-abs-defs-goal-seek-cdf.png]]

:[[image:array-abs-defs-goal-seek-pdf.png]]

When <code>R</code> is evaluated in prob-mode (e.g., when its [[CDF]] is viewed), [[GoalSeek]] must deal with the [[Run]] dimension in ''b'' and ''c''. For dealing with extra dimensions that might be introduced into <code>Y</code> from other components of the model that GoalSeek has no way of knowing or reasoning about, [[GoalSeek]] employs the vertical array abstraction method. Any dimensions of <code>b</code> are handled via this method. This is an example where the horizontal method could not be adapted, but the vertical method could. Extra iterations beyond that necessary for minimal convergence in this case are not harmful in terms of the functional result, so we just let them update. (Strictly speaking, this is not perfect array abstraction, since the value in the abstracted case may be a little more precise then the same value computed as a scalar, but since both will be within the convergence threshold, this is not particularly important). For iteration over dimensions introduced thru the goal (''c'' in this case) or <code>X</code> (different initial guesses), [[GoalSeek]] leverages horizontal array abstraction.

== Conclusion ==
One of the key skills of the advanced Analytica modeler is ability to write array abstractable functions and models, leading to maximum flexibility for later dimensional adjustments, what-if analyses, uncertainty computations, and decision making. This skill is developed by perfecting key elements in modeling style, learning to recognize the cases that may be subject to array abstraction limitations, and developing the skill to adjust expressions in ways that will abstract. 

The later requires developing deep understandings of features such as the [[While]] loop, horizontal array abstraction, etc. Many modelers using Analytica may find they seldom run into a situation that does not array abstract… these situations tend to arise in the more complex and algorithmically-oriented applications. But for those in the latter camp, the flexibility that comes from array abstractable models further increases the power of Analytica for exploring and prototyping new models.

==History==
This article was written April 2003 by Lonnie Chrisman, Ph.D., of Lumina Decision Systems, updated Nov 2005, and originally in a PDF format.  It was transferred to the Analytica Wiki Feb 2010.

==See Also==
* [[Array Abstraction]]
* [[Definition]]
* [[Creating or editing a definition]]