Custom file system providers

Release:	… • 4.6 • 5.0 • 5.1 • 5.2 • 5.3 • 5.4 • 6.0 • 6.1 • 6.2 • 6.3 • 6.4 • 6.5 • • 6.6 • 7.0 • 7.1 • 7.2

New in Analytica 7.2

Requires Analytica Developer edition or better. It is always available in the Analytica Decision Engine, in ACP, and in AMP (the Analytica MCP Platform). In a lesser edition, a «scheme»:// path reports that the feature is not available in this edition.

What a custom file system provider is

A file provider makes a URI-style path

«scheme»://«root»/«path»

for example repo://Sales/Q3.csv, work anywhere the Analytica engine accepts a file name. That includes opening a model and the modules that model links to or includes, ReadTextFile, ReadBinaryFile, FileSystemListing, FileExists, FileFullPath, FilePathPart, and the internal file-exists tests that model loading uses.

The scheme prefix -- repo in that example -- is configuration, not code. Providers are registered when the engine starts up, from a plain text file named FileProviders.config, so one deployment can define several prefixes, each backed by a different storage service. Behind each prefix is an HTTP service that speaks a deliberately tiny protocol. You can write that service yourself: see Custom file system providers/API spec.

Things this is used for:

A cloud object store, so that models and data live in a bucket rather than on a file server or on each analyst's own disk.
A document management system, so that reading a file goes through the same access control and audit trail as every other document in the organization.
A version-controlled model repository, so that a run can name the exact models and data it used and those names keep meaning the same thing.
A corporate content API that already knows where the authoritative copy of a data set lives.

The point of doing it this way is that nothing else in the model changes. The model author writes

ReadTextFile( "repo://Sales/Q3.csv" )

and every other part of the model, and every function that takes a file name, behaves exactly as it does with a local path.

URI paths

The form is «scheme»://«rest». A scheme starts with a letter, continues with letters, digits, +, . or -, and is at least two characters long. The two-character minimum is deliberate: it keeps C://foo a Windows drive path rather than a URI.
Scheme matching is not case-sensitive. repo:// and Repo:// reach the same provider.
A backslash is accepted inside a URI path and treated as /. So a model at repo://models/Main.ana that includes sub\Child.ana resolves to repo://models/sub/Child.ana.
. and .. segments resolve textually when a relative path is joined onto a URI folder, and .. can never pop above «scheme»://«root». A .. segment that would still reach the provider is refused outright.
A URI is treated as an absolute path, so relative-path resolution against a model that was opened from a URI works normally.
The path utilities keep a syntactic URI intact even when no provider is registered for its scheme, so an unknown scheme fails with a clear message instead of being mangled into a Windows path.

FileProviders.config

FileProviders.config is a plain text file with one setting per line. A # starts a comment, blank lines are ignored, and each setting has the form

«key» «scheme» = «value»

Key names are not case-sensitive.

# scheme prefix -> store service.  Several FileStore lines define several schemes.
FileStore repo          = http://mystore.internal:8080
FileStoreToken repo     = «bearer-token»          # literal token
FileStoreTokenFile repo = C:\secrets\repo-token   # OR read the token from this file
FileStoreMaxMB repo     = 256                     # optional fetch size cap (default 256)
FileStoreTimeoutS repo  = 60                      # optional read timeout (default 60)

Setting	Meaning
`FileStore «scheme» = «url»`	Declares «scheme» and gives the base URL of the store service that serves it. Required, and it must be the first line for that scheme.
`FileStoreToken «scheme» = «token»`	A bearer token, written literally, that is sent with every request to that store.
`FileStoreTokenFile «scheme» = «path»`	Reads the bearer token from this file instead. Surrounding white space in the file is trimmed. When both are given, `FileStoreTokenFile` wins.
`FileStoreMaxMB «scheme» = «n»`	The largest file, in megabytes, that will be fetched from this store. Default 256. A fetch that exceeds it is aborted with an error that names this setting.
`FileStoreTimeoutS «scheme» = «n»`	The read timeout, in seconds. Default 60. The connect timeout is fixed at 5 seconds.

The FileStore line must come first for a scheme. Any of the other keys naming a scheme that has not been declared yet is ignored, with a warning on the typescript (see Diagnostics for where that appears in each product).

The URL may include a path prefix, as in http://host:port/prefix. A trailing / is dropped, and the endpoints the engine calls are appended to what remains.

The URL must be http://. An https:// URL is refused when the provider is registered, with a warning on the typescript, because TLS is not compiled into the engine's HTTP client. The deployments this supports are a store service on localhost, or one inside the same private network or cluster. See Security below.

The configuration is read once per process, during engine initialization. Editing FileProviders.config has no effect until Analytica is restarted.

Where Analytica looks for FileProviders.config

Three locations are consulted in this order, and the first one that names a file wins.

The host's own command line option, /stores:«path» -- see Analytica Command Line.
The FileProvidersConfig registry value. Two keys are consulted, and each is looked for under HKEY_CURRENT_USER first and then under HKEY_LOCAL_MACHINE:
- HKCU\Software\Lumina Decision Systems\«product»\«version»\FileProvidersConfig -- per product and per version, for example ...\Analytica\7.2 or ...\ADEW\7.2. Use this when you want one product to see a different set of stores from the others.
- HKCU\SOFTWARE\Lumina Decision Systems\FileProvidersConfig -- product-independent. One value here serves Analytica, ADE and ADEW, ACP and AMP alike. This is the usual choice, and it is the only way to set the location for ADE and ADEW, which have no command line of their own.
A value that names a file which does not exist produces a warning on the typescript, rather than quietly registering nothing.
FileProviders.config in the same folder as the engine binary -- the deployed default.

When none of these finds a file, no providers are registered and nothing is reported.

Using a provider from a model

Once repo has been declared, use it as an ordinary file name.

ReadTextFile( "repo://data/Sales.csv" )

ReadBinaryFile( "repo://data/logo.png" )

FileExists( "repo://models/My Model.ana" )

FileSystemListing returns the names, sizes and modification dates that the store reports:

FileSystemListing( "repo://models/", recurse: true )

FileSystemListing( "repo://data/*.csv", ['Name', 'Size', 'DateLastModified'] )

A wildcard in the last segment of the path filters the listing, as it does for a local folder.

You can open a model whose path is a URI. Modules that the model links to or includes resolve relative to the model's own URI, so a model at repo://models/Main.ana that includes sub\Child.ana loads repo://models/sub/Child.ana from the same store. FileFullPath joins a relative name onto a URI folder the same way:

FileFullPath( "Q3.csv", "repo://Sales" ) → 'repo://Sales/Q3.csv'

An obfuscated model works over a provider scheme exactly as it does from a local file.

A long transfer stays cancellable. Ctrl+Break, or a cancellation from an ACP or AMP client, interrupts a fetch that is in progress instead of waiting for it to finish.

Existence checks

The FileExists function asks the store. An object stored at that exact key is a file. A folder is recognized from the store's listing of its parent folder, since a store normally exposes a folder only as a listing prefix and answers "not found" when asked about it directly. Name matching is case-sensitive, as it is in the store itself. When the store cannot be reached, or rejects the credentials, FileExists reports that error rather than guessing an answer -- the modeler asked a question, and a wrong answer would be worse than an error.

The existence tests the engine makes on its own, while resolving a model and the modules it links to, use a plain HTTP HEAD request and take the opposite tack, deliberately: if the store cannot be reached, or rejects the credentials, the test reports exists -- it cannot prove absence. Control then reaches the code that actually opens the file, and that reports the real transport or authentication problem, instead of a misleading "file not found".

Restrictions in Analytica 7.2

Read-only. Every write path refuses a URI path with a clean error: saving a model to a URI, WriteTextFile, WriteBinaryFile, FileSystemCopy, FileSystemMove, FileSystemDelete and FileSystemNewFolder. Write-back is planned, but is not in 7.2.
Whole-file fetch. A read pulls the entire file, subject to FileStoreMaxMB. It is not a seekable remote stream, so a provider path is not a good way to peek at a few bytes of a very large file.
One short fetch cache. The URI most recently fetched is cached for 10 seconds, because opening a model reads the same file two or three times. Past that, every read goes to the store.
No file browser. A URI path never poses a "locate the file" dialog. A file that is not there is reported as not found.

Diagnostics

Everything the provider layer reports itself is written to the typescript -- the engine's typescript output. Where to look for it depends on which product you are running: in Analytica it appears in the typescript window; ADE and ADEW capture the typescript as their log, and so does ACP; and in AMP the typescript is echoed to stderr, which is the server log, so Amp.exe users find these lines on stderr. AMP is the only product where stderr is the right place to look.

On start-up, each registered provider prints one line:

[FileProviders] repo:// -> http://mystore.internal:8080

Unrecognized lines, unusable scheme names, an https:// URL, a token file that could not be read, and a FileProvidersConfig registry value naming a file that is not there, each produce their own warning there. An absent configuration file is silent: no providers are registered and nothing is written.

Errors that reach the model are reported as The file '...' could not be opened, with one of these explanations:

Situation	What it says
No provider is registered for that scheme	`no file provider is registered for the scheme 'repo'`
The edition does not include the feature	`'repo://' paths require the Developer edition or better`, also naming the edition you do have
The store rejected the credentials (HTTP 401 or 403)	`the repo:// file store rejected the credentials (HTTP 401)`
Any other unexpected HTTP status	`HTTP «n» from the repo:// file store`
The service could not be reached	`could not reach the repo:// file store (...)`
The file is larger than the cap	`the file exceeds the «n» MB fetch limit (FileStoreMaxMB in FileProviders.config)`
A `..` segment in the path	`'..' is not allowed in a repo:// path`
Writing to a URI	`repo:// locations are read-only in this version of Analytica`

HTTP 404 is not an error. It means the file is not there, and the caller then reports a missing file in its usual way.

Security

The connection between Analytica and a store service is plain HTTP, with no TLS. That is a deliberate limitation of this release, and you have to design around it:

The bearer token, and the file contents, travel in the clear over that hop. Run the store service on localhost, or on a private network or cluster network that you already trust with that traffic. Do not put a store service anywhere you would not be willing to send the token in plain text.
FileProviders.config can hold a token, so protect it like any other file holding a secret. Restrict who can read it, and keep it out of version control and out of model archives.
Prefer FileStoreTokenFile to FileStoreToken. It keeps the token out of the file that gets copied from machine to machine, and it lets the token live somewhere with tighter permissions -- or, in a container, be a mounted secret.
A store service should authenticate every request, and should reject path traversal itself. The engine refuses a .. segment before any request is sent, but a service must not rely on that.