Distributed Computation using ADE

Uses the Enterprise edition and features new Analytica 4.6

In this application, we use the COM Integration feature (introduced in Analytica 4.6) to call multiple instances of ADE (Analytica Decision Engine), evaluating the same model on both the same multi-core computer and a different computer. By doing this, we are able to take advantage of both multiple cores and multiple computers to speed up the computation on a Monte Carlo simulation. We applied this to a large model developed for and used by the North West Power Council for planning future electric grid capacity. By running 16 ADE instances on 2 computers with 8 cores each, a 10-fold speed-up was measured.

The above graph shows the elapsed time required to complete the full Monte Carlo simulation as a function of the number of parallel ADE instances utilized to carry out the simulation.

This page describes how this is done. It provides an real-life example of how to call COM objects from Analytica. You may find this useful even if you are using COM to do something much different than this, or you might be motivated to use ADE in the same manner described here.

Hardware and Licenses needed

This project uses:

• Two high-end computers running Windows Server 2012. Each server has 8 cores and more than enough RAM to hold 8 instances of the model. (The model itself evaluates in less than 1GB of RAM, each server has 16GB).
• A high-speed network connecting the two computers.

  Note: Any edition of Windows except for Windows 7+ Home edition should work here.

• One edition of Analytica Enterprise or better, release 4.6 or later.
• Two ADE licenses, one for each computer.

Distributing the computation

The master model runs in Analytica. The primary output of the model is computed by Monte Carlo simulation, usually with a sample size of 750. When run sequentially (in a single instance of Analytica), the computation takes about 50 seconds to complete. The model repeats this calculation repeatedly inside an optimization, which typically runs for 8 hours.

When the computation is distributed, the model uses COMCreateObject to instantiate up to 8 instances of ADE on each computer. Each instance loads the model, the inputs are synchronized, and each instance runs the Monte Carlo simulation using Ceil( 750/N ) samples, where N is the number of ADE instances. The ADE instances perform this computation concurrently. The sample for the final result is copied back to the master model and concatenated to get the full 750 samples.

The remainder of this page takes you through the steps and Analytica code in detail.

Specifying how the computation is distributed

To begin, we decide how many computers and how many cores the computation will be distributed over, and how that will be done, and encode that information in the model. This logic is simply deciding how the calculation shall be divided, but using COM yet.

An index contains the names of the computers where ADE can be run:

Index Host_name := ['localhost','NWDistComp2']

The model was arranged so that the user can select how many parallel ADE instances to use. The first 8 instances are "dispatched" to the first host, the next 8 to the second host, etc. The code for deciding how many instances to assign to each computer is as follows:

Variable Cores_to_use := 16    { A user input }
 
Variable Max_cores_per_host := Table(Host_name)(8,8)
 
Variable Num_ADE_Clients := Dispatch( Cores_to_use, Max_cores_per_host, Host_name )

When evaluated, Num_ADE_Clients is an array indexed by Host_name with a number showing how many ADE instances will run on each computer. With Cores_to_use set to 16, this will be 8 and 8.

This next index will index the ADE instances after we instantiate them.

Index ADE_instance := 1..Cores_to_use

And this array specifies which computer each instance of ADE will run on:

Variable Host_of_ADE_instance := StepInterp( Cumulate(Number_of_ADE_client,Distributed_Host_Nam), Distributed_Host_Nam, ADE_Instance, Distributed_Host_Nam )

Host_of_ADE_instance in an array indexed by ADE_instance where the first 8 cells are 'localhost' and the second eight cells are 'NWDistComp2'. So if you want to know where the 7th instance of ADE is running, you can look it up directly in this array.

The next variable specifies how many Monte Carlo samples each instance of ADE should run.

Variable Client_sample_size := Ceil( Size(Run) / Cores_to_use )

Notice that the master model will have the full sample size (e.g., 750), whereas each ADE client will use a smaller sample size. The build-in Run index will be used by the master model to store the full sample, but as we pull the data in from each client, the DC_Run index will correspond to the shorter index used by each client.

Index DC_Run := 1..Client_sample_size

The following mapping from each client sample into the full run index makes it easy to transform between the master and clients:

Variable Run_to_ADE_map := Min([ sampleSize, (ADE_instance-1)*Client_sample_size + DC_Run ])

The result of Run_to_ADE_map shown here tells us, for example, that the 332nd sample in the full result will be the 3rd sample computed by the 8th ADE instance.

Instantiating the ADE clients

Up to this point, none of the code from the previous section has actually made use of COM -- we've simply defined how we intent to break the full computation into pieces, and those pieces map back to the full computation. In this section, we instantiate the ADE clients.

The objective here is to have a variable named ADE that holds an array of the ADE clients, indexed by ADE_instance. Setting this up involves quite a bit more than just a call to COMCreateObject, since we also want to have each instance load the model, set its own sample size and other simulation parameters, and synchronize the values of inputs. There are some important subtleties to pay attention to.

The ADE instances are created like this:

Variable ADE := COMCreateObject( "ADE4.CAEngine", server: Host_of_ADE_instance )

The text "ADE4.CAEngine" identifies which COM object is being instantiated. The «server» parameter specifies where (i.e., on which computer) the ADE instance shall live. Since Host_of_ADE_instance is an array of up to 16 computer names, array-abstraction kicks in and instantiates all 16 instances at the same time.

The variable ADE will be used from various places in our model, and some variables might request its Mid-value, whereas other variables might request its uncertain, or Sample-value. This presents a problem, because Analytica will evaluate its definition in Mid-mode the first time the Mid-value is requested, and then will evaluate it again in Sample-mode the first time the uncertain value is requested. If that happens, we'll end up with twice as many ADEs as desired, and they won't be the same instances. To avoid this problem, we split the instantiation across two variables:

Variable Instantiation_of_ADE := COMCreateObject( "ADE4.CAEngine", server: Host_of_ADE_instance )
 
Variable ADE := Mid(Instantiation_of_ADE)

No other variable in the model uses Instantiation_of_ADE. All code that communicates with the ADE instances uses the variable ADE only. This ensures that Instantiation_of_ADE is only evaluated in mid-mode.

We also load the model into each instance of ADE, so we augment the Definition of Instantiation_of_ADE. As a first cut, we could do this as:

Var a := COMCreateObject( "ADE4.CAEngine", server: Host_of_ADE_instance );
a->OpenModel( Model_file_path );
a

Since the local variable a contains an array of ADE instances, array-abstraction calls the OpenModel method for each of them. These two lines of code perform an impressive amount of work, instantiating 16 ADEs across two different computers via DCOM, and instructing all 16 to load the model. (The model, of course, must be on the disk on each computer already). However, we can do better. Written as shown above, the call to OpenModel occurs sequentially, meaning Analytica tells the first ADE to open the model, and waits for that to complete before telling the second instance to open the model. Analytica's COM Integration allows you to make these 16 calls concurrently when written as follows.

Var a := COMCreateObject( "ADE4.CAEngine", server: Host_of_ADE_instance );
a->COMCallMethod(a, "OpenModel", Model_file_path, concurrent:True );
a

The model contains a flag to indicate whether the computation should be done sequentially (in Analytica only) or via distributed computation:

Variable Use_distributed_computing := Checkbox(1)     { A user input }

It is extremely important that this flag be cleared in each ADE instance; otherwise, each ADE will attempt to spawn its own computation to 16 more ADEs, causing an explosion of ADE instances that will consume all available resources on each computer before you can realize what is happening.