What's new in Analytica 4.3?

Analytica 4.3 was released on 14 March 2011. This page describes on feature enhancements added to release 4.3 since release 4.2. We have not attempted to list the large number of bugs that were also fixed.

Two major focuses dominate the feature list for this release:

• Domain attribute refinements
• Structured Optimization.

Licenses and Installation

We've tried to make it easier for you to keep up-to-date with the newest release, and to remove the pain of having to obtain keys or licenses when updating.

Automatic acquisition of license

• When you are eligible for a free upgrade, the installer automatically obtains and activations the upgrade license for you. No need to send an email or go to a special web page to obtain a new activation key. This feature requires:
  • An internet connection.
  • An Analytica 4.2 license previously activated to your computer & account.
  • Active support.

• The same feature is used to acquire a beta-testing license for 4.3 during the beta-test period. Same requirements.

Automatic downloads of new releases

• During beta testing, the Analytica Update Checker will automatically extend the beta testing period while the beta testing period is open. This should happen automatically so you don't need to deal with expiring beta licenses, etc.

• When a new patch releases, beta builds, or new minor releases become available in the future, the Analytica Update Checker will automatically download the patch installer in the background. Once ready, you'll be notified that the patch is ready and prompted to run the installer with a button click. Should make it much easier to keep up to date with the current release.

Domain Attribute

The Domain Attribute has been enhanced in several significant ways, and should play a much more prominent role promoting clarity of thought during the model design, as well as a more prominent role when teaching the skills of quantitative modeling to students. Several fundamental and deep design changes have been made to the Domain as it existed in previous releases, with the purpose of giving this potentially important attribute a more consistent semantics than was previously the case, and opening up the possibility for several new uses of this attribute.

If you are an experienced Analytica model builder in earlier Analytica releases, you probably know of the existance of the Domain attribute and its use in a few particular cases, but you likely view it as a minor side-line feature. Now that it has a more uniform semantics, you may now want to start viewing it as an important partner with the Definition, Description, and Units attributes. The Domain attribute plays a variety of roles for various functionalities, but can also be instrumental in reducing modeling errors. Although you may find the changes subtle at first, the domain enhancements are probably the most significant enhancements in this 4.3 release, and with time may have the greatest impact on your thought processes when building a model.

The domain attribute has a simple unified purpose (semantics):

The domain defines the space of possible values that each cell of the computed result is allowed to have.

Compare this to the Definition attribute, which (implicitly) specifies the actual values of the result. The key differences are that the domain is less precise (the space of possible values vs. the precise value), and the domain's value is less likely to change when exogenous inputs are altered. Like the definition, the domain can be defined by a computed expression, so in less common cases the space of possible values can be a function of other inputs, but more more commonly you will simply select from the more common simple domain types using the domain type pulldown with a static set of values.

Domain Specification

• The domain-type pulldown has a new set of choices:

Integer, Grouped Integer, Boolean, and Expression are entirely new to 4.3.

The Discrete option takes the place of the former Discrete numeric and Categorical options.

The Explicit values option takes the place of the former List and List of labels options.

Grouped integer is used only in optimization problems. This option is seen only in Analytica Optimizer on decision nodes.

Options above the first dividing line (Continuous thru Discrete) implicitly define the set of possible values.

Options between the dividing lines (Explicit values and Copy from index) explicitly define the set of possible values -- i.e., the possible values are explicitly enumerated.

Expression, for advanced users, allows an arbitrary expression to compute the domain.

Domain Expressions

Although intended only for advanced users, a domain expression computes the set of possible values using an arbitrary expression similar to what you would include in a definition. Four special new functions are introduced (Continuous, Integer, GroupedInteger and Discrete), intended only for use within a domain expression, but which can also be intermixed with arbitrary Analytica expression logic, and can even result in domain specifications that vary along an index.

An example of a simple domain expression would be:

Integer(lb: typicalX-variationX, ub: typicalX + variationX)

An example using arbitrary logic would be:

If Use_int_analysis Then Integer() Else Continuous()

When the domain expression is a single identifier, the expression is interpreted as Copy from index, where the identifier is the source of the explicit values. This is not the same as placing the arbitrary expression that computes the domain in a separate variable and then using its result (only because a single identifier is interpreted differently) -- for that, use Mid(computedDomain).

Domain Indexes

In previous releases, domain values became the Self-index values for the variable. This was problematic for several reasons and has now changed. In a model built in Analytica 4.3, the domain values do not become the Self-index values. When a domain is specified as an explicit list of values, the variable will have a special domain index, which is now a full-fledged Analytica index object, connected to but distinct from the variable object itself.

The promotion of domain values to Self-index values in earlier releases of Analytica muddied the semantics of the domain attribute. That promotion has virtually nothing to do with the idea that the domain defines the space of possible values, making it hard to pin down exactly what the domain attribute means. It also created a conflict with other basic principles of self-indexes, such as the idea that a Self-index results when a computed result contains an implicit dimension, which henceforth gets promoted to a Self-index. There were situations where the promotion of domain and promotion of an implicit dimension to the Self-index were in conflict. And there were functionalities that were not possible as a result of that conflict.

The need for an index corresponding to the domain values arises primarily in the context of DetermTables and ProbTables, where these indexes are needed for defining the resulting values that correspond to each possible value. It also arises to some extent when using a self-Choice pulldown. Now the new local domain index for the variable, rather than the self-index of the variable, serves that purpose.

This change introduces a potential backward compatibility issue. The vast majority of models will be unimpacted by this change, but since some legacy models may rely on a self-index that was specified in the domain, a preference setting is introduced, automatically set when loading a legacy model into 4.3, but unset by default when starting a model fresh in 4.3:

Distinct Domain and Self indexes

It can be quite useful to have separate domain and self indexes, something that is only possible if you have the domain acts as self index preference turned off. For example, rather than performing a Parametric Analysis over all possible values, you might opt to perform it over a subset of possible values:

 Index Policy
 Definition: ['None', 'Carbon tax', 'Carbon limit', 'Carbon cap+trade']
  
 Variable Carbon_policy
 Definition: Subset(Selected_policy_types)
 Domain: Policy
  
 Variable Selected_policy_types
 Definition: Table(Policy)(0,1,1,0)
 Domain: Boolean()

Domain of «X»

The expression domain of «x» provides access to the explicit set of values specified for the domain of a variable. This is applicable when the domain of the variable, «x», is set to either Explicit values or Copy from index.

You can use the expression domain of «x» anywhere in an expression, including where an index is expected. When it is used where an index is expected, the index operated over is the local domain index that is connected to «x». This example shows its use in both a value context and an index context:

JoinText(Domain of Carbon_policy,Domain of Carbon_policy)

Clarity of Thought

Often the most difficult and most critical step in model building is variable identification -- i.e., the process of identifying what your variables are, and defining them in a clear and unambiguous fashion. Variables should pass the Clarity test (aka Clairvoyant test). Creating a clean description, identifying the units of measurements, and identifying the space of possible values is all instrumental to the process of achieving clarity.

Incorporating the specification of the domain attribute into the early stages of variable identification can help ensure that you have achieved clarity. Because this requires far less precision and complexity than defining how the variable is computed, it is appropriate to do before you reach the more detailed thinking of creating definitions.

If you are teaching students quantitative modeling skills, it should be valuable to work this step into the curriculum. It forces the students to think about what the variable means at a level slightly more precise than the verbal description, but still more abstract than the definition. It takes practice for students to master skills of thinking in the abstract -- such as about the overall structure and identification of variables -- without diving into the details prematurely. By specifying the domain, a student is forced to visualize the result as a value of a particular form, which can free them from the urge to spend time spelling out the detailed definition before they've completed the higher level thinking about whether their choice of variable captures the necessary distinctions, and of how the variables interrelate with each other.

Where Analytica makes use of Domain information

• Bounds checking on computed results: Computed results that are not consistent with the domain trigger a bounds check warning. (There is a preference setting to turn this off).
• Bounds checking on edit table cells: When a literal is entered into an edit table, the cell will be flagged to the user if the literal value is not consistent with the domain specification.
• Detection of possible modeling errors.
• Structured optimization: Domain determines the variable's bounds and integer type.
• Uncertainty analysis: Determines whether Monte Carlo samples are treated as samples from discrete or from continuous distributions. Discrete distributions result in probability (mass) distributions, continuous distributions result in probability density functions (PDFs).
• DetermTables and ProbTables: These map from the set of possible values to a result. The set of possible values is taken from the domain.
• Choice(Self,n): A self-choice takes its values from the domain
• Label order in graphs: An Explicit values domain or Copy from index domain determines the sort-order for axes labels in a graph. This applies to cases where the result being graphed consists of textual cells rather than numbers, so that the labels appear on the Y-axis.
• Graph axis type: The domain is used in the graphing heuristics when determining the default for whether a given axis should be scaled as numeric or categorical (until you've overridden it in Graph Setup).
• The discrete-type specification (for Discrete and Explicit values domains) is used by various GUI controls to determine how the value should be displayed and edited (e.g., left or right justified, with or without quotes).
• An input node will suppress showing quotation marks and accept only textual input when the domain is discrete textual, i.e., Discrete(type:'text').

Model Validation

New extensions based on the Domain Attribute help detect likely mistakes in your model, that previously may have gone undetected.

The Check Attribute exited previously, and has not changed in 4.3. You can place validation tests in the Check Attribute to help catch mistakes, and you can control the message that results by calling the Error function. Computed results are validated against the expression in the Check Attribute.

Analytica 4.3 also includes a validation against the Domain Attribute. This validation may be more convenient to configure than using the Check Attribute for catching out-of-bounds conditions, plus you may get it for free when you have already configured the Domain Attribute for other purposes.

When result values don't agree with the domain values, a dialog such as the following appears:

Before you hastily ignore this (your model has always worked, right?), take a closer look at the domain of the indicated variable. We've found that usually uncovers a real error in the model! In the example shown here (taken from an actual and very influential model), the domain was [0,1]. At some point, something was changed, but not everything had been updated to be consistent.

Finally, a third type of validation has been added to detect cases where expressions (particularly, comparisons) appear to be inconsistent with a domain, as illustrated here:

There is not a checkbox on the preference dialog to turn off this setting globally. We've seen this warning frequently during our testing of legacy models, and its false-positive rate is so low, we don't think you'd ever want to turn it off globally.

Structured Optimization

Structured optimization features require Analytica Optimizer

Structured optimization makes the encoding of complex optimization models much easier, providing features similar to those found in algebraic optimization languages, but also designed to leverage on the strengths of Analytica's influence diagrams and intelligent array abstraction.

Highlights of structured optimization:

• The specification of your optimization problem utilizes the influence diagram structure of your model.
• Designed to co-exist with parametric analysis and other non-optimized uses of the model.
• Linear, quadratic and non-linear problems are encoded in the same way.
• Objectives and constraints are encoded using Analytica's usual expression syntax (i.e., variable definitions). Constraints are encoded as comparisons (<=, >= or =).
• You can have any number of separate decision nodes and constraint nodes. Each decision or constraint can be multidimensional, and the dimensionality of each decision or constraint can be different. No need to flatten, unflatten, or conconcatenate multidimensional decisions and constraints. It takes care of all that for you.
• Automatically infers whether the problem is linear, non-linear or quadratic. In linear and quadratic cases, automatically determines the coefficients by analyzing your model's structure. There is no need for you to separate out the coefficients. You just use standard Analytica expressions.
• Infers whether dimensions are intrinsic to the problem or should be array abstracted. Provides the ability to explicitly specify if you don't want to rely on this inference.
• Obtains information about decision variables, such as integer type, bounds, etc., from the Domain attribute. Substantial richness and flexibility is possible using computed and array-valued domains (where integer type, bounds, etc. vary along one or more indices).
• Support for piecewise-linear relationships (i.e., models that use LinearInterp). Will automatically produce an integer-LP when a model contains one or more LinearInterp calls with everything else linear.
• Automatic selection of the most appropriate solver engine.
• Support for convext quadratically constrained programs using Frontline's SOCP Barrier engine. Automatically switches to a non-linear engine and converts the problem type when a non-convex constraint set is detected.
• Supports decision variables with explicit domains (e.g., lists of labels). Formulates these as integer-valued NLPs, but automatically manages the mapping from domain value to the integer used internally by the solver.

Data Types Enhancements

Date/Time Values

A date, date-time, or duration has always been a numeric quantity in units of days, a date being the number of days since a fixed date origin, and a time being a fraction of a day.

Analytica 4.3 now distinguishes between normal numbers and date-time numeric quantities. Entries that parsed as dates or times, the results of date or time functions (MakeDate, MakeTime, ParseDate, Sequence(.., dateUnit:..), DateAdd, Round(..,dateUnit:..), Floor(..,dateUnit:..), Ceil(dateUnit:..), DbQuery(..,datesAsText:false) on date columns, SpreadsheetCell and SpreadsheetRange when cells contain dates, etc.) are distinguished as date-time quantities, such that when they are displayed, they display using the Date format, rather than the number format. You find that dates and numbers can now co-exist in the same tables much more readily than was previously the case.

You can specify the date format and non-date formats from the number format dialog. Click on Date to change the format used to display date-times. When you are throw, click back on the number format type you want to use for numeric values.

If you really want to view the number value of the date (the number of days elapsed since the date origin), then check Display dates as numbers on the number format dialog.

Identifiers preserve Camel Case

Analytica 4.3 now preserves the capitalization that you enter for your identifiers. So if you enter a multi-word identifier using CamelCase, the capitalization will be preserved in the object window, on the diagram, and whenever Analytica 4.3 prints a message that includes that identifier.

Other

• Title can be much longer (>255 chars)

Evaluation

Latin Hypercube and Shuffle algorithms

A small undesirable bias was discovered in the random shuffling algorithm used in Latin Hypercube sampling and by the Shuffle function (present early Demos through Analytica 4.2). The bias introduces a small skew in computed results. The bias is extremely small at large sample size, but can become significant at sample sizes under 10.

Analytica 4.3 eliminates this bias, thus increasing the fidelity of Latin-Hypercube simulations. However, a side-effect of this is that you will find that the random samples generated by distribution functions are shuffled differently than they were in Analytica 4.2, so the precise numbers in your Sample view will be different.

If you need to preserve the same sampling order used by 4.2, then you can do so by changing the system variable SampleType from Typescript. The following values are recognized:

• 0 = Median Latin Hypercube
• 1 = Random Latin Hypercube
• 2 = Monte Carlo
• 3 = Biased Median Latin Hypercube (i.e., 4.2 equivalent)
• 4 = Biased Random Latin Hypercube (i.e., 4.2 equivalent)

To set to use BMLH, press [F12] to open the Typescript Window, then type:

SampleType : 3

If you are worried about the impact of this bias on the correctness of results computed by earlier Analytica releases, don't lose sleep over it, especially if you used a sample size greater than 10. The bias occurs because the older algorithm never shuffles the n^th smallest value to the n^th position. It is possible to create some highly contrived examples where this does produce a noticible impact on the uncertainty statistics when the sample size is under 10, but in typical models it is very difficult to find a case where the result statistics are noticibly impacted.

AND/OR precendence

A or B and C now parses as (A or (B and C)).

Formerly AND and OR were parsed with equal left-to-right precedence, so this would have parsed as ((A or B) and C). Analytica 4.3 will automatically add parens to legacy models so that existing logic isn't changed.

Control over Proactive Evaluation

When a model is loaded into Analytica, Analytica attempts to parse all definitions and certain objects are proactively evaluated. In pre-4.3, all Indexes are proactively evaluated, and the only means available to control whether an object gets proactively evaluated is to change its class, either to be Index or something else such as Variable.

In Analytica 4.3 it is now possible to control which objects are proactively evaluated independent of their class. In addition, you can control the default for the entire object class, so for example you can disable proactive evaluation for Index objects, or enable proactive evaluation for Constants. You can also turn proactive evaluation on or off for individual objects, overriding the class default.

These changes mean that you can use the object class to reflect the logical nature of the variable. If it is an index, you can use the parallogram shape, even if it does take a long time to compute that you'd rather not pay at model load time.

Proactive evaluation is controlled by the attribute ProactivelyEvaluate that must be changed from Typescript. To enter Typescript commands, enter the typescript window by pressing [F12]. Then the following examples illustrate its use (the > denotes the typescript prompt - not something you type):

 > ProactivelyEvaluate Index : 0           { Turns off proactive evaluation for all indexes }
 > ProactivelyEvaluate Constant: 1         { Turns on proactive evaluation for all Constant objects }
 > ProactivelyEvaluate Va1 : 1             { Turns on proactive evaluation for Variable Va1 }
 > ProactivelyEvaluate Rows : 0            { Turns off proactive evaluation for the index Rows }

Built-in Functions

New Functions

New Array Functions

• Dispatch: Computes a capacity allocaton onto a set of resources.
• CompressMemoryUsedBy: Attempts to compresses the internal representation of an array, to reduce memory usage.
• ListOfHandles: Takes a list of identifiers and returns a list of Handles.

Set Functions

These new functions perform operations on Sets, i.e., on collections of elements. A set is represented as a reference to a list containing no duplicate elements, and with no implied ordering between elements.

• SetContains
• SetsAreEqual
• SetIntersection
• SetUnion
• SetDifference

Domain Definition Functions

These functions are used within a domain attribute expression to define and store the type of domain. They don't serve a useful purpose outside of the domain attribute.

• Continuous
• Integer
• Boolean
• GroupedInteger
• Discrete

Domain Access Functions

If, for some reason, you want to use information about the space of possible values of a variable in a computation, these functions provide convenient access to the details of how the domain is defined.

• DomainType
• DomainLowerBound
• DomainUpperBound
• DomainIntegerGroup
• DomainAllowed
• DomainNullOk

Parsed Expression Access Functions

Some Analytica attributes hold parsed expressions -- also known as parse trees. For example, the FixedDef attribute holds the parsed Definition. Previously it was not possible for an Analytica model to look inside these parsed expressions. There are meta-level applications that do arise from time to time where it would be nice to do so, and these new functions provide that access.

• ParsedExprParameters
• ParsedExprFunction

Optimization Functions

New names have been introduced for all previously existing optimization functions, and many of these have been extended. A single function, DefineOptimization, now subsumes the previous LpDefine, QpDefine, and NlpDefine functions, with LPs, QPs, and NLPs now being formulated in exactly the same way, unlike in previous releases where the formalation of an NLP was quite different from that of an LP or QP. While previous most optimization functions began with Lp..., such as LpSolution, these names have been changed to start with Opt..., such as OptSolution. (This is because it was a bit wierd to be using LpSolution to access the solution of a QP or NLP).

• DefineOptimization
• OptSolution
• OptStatusText
• OptObjective

• OptGuess

New Math Functions

• ArcCosH
• ArcSinH
• ArcTanH

Changes in Existing Functions

• SubScript and Slice don't issue out-of-range warnings when shifting. E.g.:

A[@I=@I-1]

• Subscript and Slice allow specification of default value (for out-of-range values), which also suppresses the warning when specified. E.g.

A[@I=@I-1,defVal:0]

• Unique sped up substantially when applied to giant arrays.
• MemoryInUseBy Improved accuracy by avoiding double-counting of shared subarrays.
• MdTable:
  • Creates sparse cubes internally when possible. In some cases, this can result in a dramatic reduction in memory used.
  • When a value column in the relational table contains «null», 4.2 treated that cell as unspecified (and thus used the «defaultValue», while 4.3 treats this as specified as uses the value Null for the cell value. (in such cases, backward compatibility issues might result).

• Information about number of Analytica objects via GetProcessInfo("User Objects"), GetProcessInfo("Total Objects") and GetProcessInfo("Free Objects")

• Functions that write to or read from Excel spreadsheets transfer date-time values as date-time numbers, so that cells with dates are imported as dates into Analytica, and Analytica dates are written as dates to Excel.

• An optional parameter, «what», has been added to SpreadsheetCell and SpreadsheetRange to enable reading of the cell formula instead of the cell value. It also allows you to read in dates as numbers rather than as dates, or read formulas in a relative offset format.

Clipboard / OLE Linking

• When you Copy/Paste data from an external application (e.g., Excel) into an Analytica edit table, textual values are treated as textual values. The paste operation no longer attempts to interpret the textual values as Analytica expressions. Previously when textual values that happened to match an Analytica identifier or reserved word were pasted, you'd end up with a parsed expression that could produce very different results than expected.

• When OLE-linking data into Analytica from Excel, the same applies -- textual values are imported directly. No attempt to parse them occurs.
  • Note: If you really want to store Analytica expressions, such as distribution function calls, in your Excel spreadsheet and then link these into Analytica, you should follow your linked table with a second node that calls Evaluate on the result of the linked table.

New Attributes

• the AWP_Attrib attribute
• OptInitialGuess
• OptDimensions

Typescript

• Full precision of scalar result when printed in typescript
• The List command extended to use regular expressions. For example:

List "text" { Lists all identifiers containing the letters "text" within the identifier }

List SysFunction "(Get)|(Set)" { List all built-in functions containing either Get or Set in the identifier }

• Several archaic, deprecated, and non-functional commands and keywords were removed (freeing up these symbols in the global namespace for your own identifiers). These include: Send, To, DdeExecute, DdeInitiate, DdeTerminate, GetResource, Linear, None, d2, d3, keyOff, mapVar, laser, slide, path, project, ValueVar, GraphVar, View, ChanceVar, Field, Location, IndexT, DetermT, Yes, No
• Typescript extension: The command:

Delete all Mo1

deletes everything recursively inside the module Mo1. Or you can use Delete Recursive Mo1.

• New Verbosity flag helps with debugging cycles in Dynamic loops. Set Verbosity:134 (which is 2+4+128) and you get an indented listing.

Preferences

• The Use Safe Intermediates preference has been removed. Obsolete.
• The Check value bounds has two sub-options: against Check attribute and against domain bounds. See Model Validation above.
• New Domain acts as self index preference.

Other GUI

• "Compare results?" places the comparison node more intelligently on the diagram.

• Number format dialog allows you to Set Default to change the global default number format.

• In edit tables containing Checkboxes in the cells, you can select a range of cells and check or uncheck all boxes in the selected range with a single click.

• Hyperlinks to the Analytica Wiki for warning messages.

• Nice look to Message Box dialogs in XP. Plus they adapt better to changes in appearance settings, including system colors and font sizes.

• In result tables, cells containing double-clickable items (i.e., that hyperlink somewhere) now appear in blue, giving a visual depiction that they are clickable. This includes optimization problem instances, references, and handles.

Analytica 64-bit

• Maximum number of Analytica user objects has been increased to over 64,000 when using Analytica 64-bit. (For 32-bit it is still at about 31,000).

ADE

The following new methods have been added to CAEngine:

• CAEngine::DefaultEvaluationTimeLimit
• CAEngine::MonitorProcess
• CAEngine::SetCallbackObject

This new (experimental) interface has been added:

• IAMsgBoxCallback