View source for Custom file system providers/API spec

[[category:File system functions]]
[[Category:Analytica User Guide]]

{{ReleaseBar}}

''New in [[Analytica 7.2]].'' See also the parent page, [[Custom file system providers]].

__TOC__

== Overview ==

A '''store service''' is an ordinary HTTP service that Analytica calls whenever a model uses a <code>«scheme»://</code> path. It implements three endpoints, and nothing else is required. No Analytica code is involved, so write the service in whatever language and framework you like, and back it with whatever you like: object storage, a database, a version control system, a document management system, or a REST API of its own.

Analytica is told about the service by one line in <code>FileProviders.config</code>:

:<code>FileStore repo = http://mystore.internal:8080</code>

after which every path beginning <code>repo://</code> becomes a request to that base URL. See [[Custom file system providers]] for the rest of the configuration.

== The endpoints ==

All three are relative to the configured base URL, including any path prefix in it. With a base URL of <code>http://mystore.internal:8080/api</code>, the first endpoint is <code>http://mystore.internal:8080/api/files/...</code>.

{| class="wikitable"
! Request !! Purpose !! Success !! Not found
|-
| <code>GET {base}/files/{root}/{path}</code> || Fetch a file || <code>200</code>, with the raw file bytes as the body || <code>404</code>
|-
| <code>HEAD {base}/files/{root}/{path}</code> || Existence check, without transferring the file || <code>200</code> || <code>404</code>
|-
| <code>GET {base}/list/{root}/{path}?recursive=0</code> or <code>?recursive=1</code> || List a folder || <code>200</code>, with a JSON array (see below) || <code>404</code>, taken as an empty listing
|}

<code>{root}/{path}</code> is everything that followed <code>«scheme»://</code> in the model's path. For <code>repo://Sales/2026/Q3.csv</code> the engine requests <code>{base}/files/Sales/2026/Q3.csv</code>. The engine attaches no meaning of its own to <code>{root}</code>: it is simply the first segment. It is there so that one service can host several logical roots -- one per customer, per project or per bucket -- and decide from it which backing store to consult.

The <code>recursive</code> query parameter is sent on every <code>/list</code> request and on no <code>/files</code> request.

A health endpoint is a good idea for your own monitoring, but the engine never calls one and does not require it.

== Path encoding ==

Everything after <code>«scheme»://</code> is the literal path -- unencoded text, as the model author wrote it. Before sending a request, the engine:

# replaces every <code>\</code> with <code>/</code>, and
# percent-encodes every byte that is not in the unreserved set <code>A-Z a-z 0-9 - _ . ~ /</code>.

Your service must percent-decode the path. Because the encoding is applied byte by byte to the UTF-8 form of the name, spaces and non-ASCII characters round-trip correctly: <code>repo://models/Ventas Q3.csv</code> arrives as <code>/files/models/Ventas%20Q3.csv</code>.

The engine refuses a <code>..</code> segment before any request is sent, so a well-behaved client never asks for one. Reject traversal in the service as well; defence in depth is worth the three lines it costs.

== Authentication ==

When a token is configured, with <code>FileStoreToken</code> or <code>FileStoreTokenFile</code>, the engine sends

:<code>Authorization: Bearer «token»</code>

on '''every''' request, to all three endpoints. When no token is configured, no <code>Authorization</code> header is sent at all.

Answer <code>401</code> or <code>403</code> when the token is missing, expired or wrong. Analytica turns either one into an error that names the scheme, so the user is told the credentials were rejected rather than that the file is missing.

== How the engine interprets your status codes ==

{| class="wikitable"
! Status !! What the engine does
|-
| <code>200</code> || Success. The body is the file, or the listing.
|-
| <code>404</code> || Not found. Treated as a missing file, with no error text of its own, so the caller reports a missing file in its usual way. On <code>/list</code>, an empty listing.
|-
| <code>401</code> or <code>403</code> || A credentials error naming the scheme: <code>the repo:// file store rejected the credentials (HTTP 401)</code>.
|-
| anything else || <code>HTTP «n» from the repo:// file store</code>.
|}

A connection that cannot be made at all, or that times out, is reported as <code>could not reach the repo:// file store (...)</code>, with the transport reason in the parentheses. Neither of those is a status code you control, so return a real status for everything you can: the message the user sees is then about your service rather than about the network.

== The listing response ==

<code>GET {base}/list/{root}/{path}</code> returns a JSON '''array''' of objects, one per entry. A body that does not parse, or that is not a JSON array, is an error.

{| class="wikitable"
! Member !! Type !! Meaning
|-
| <code>name</code> || string || The entry name only, with no path. '''Required''' -- an entry with no <code>name</code>, or whose <code>name</code> is not a string, is skipped.
|-
| <code>isFolder</code> || boolean or number || <code>true</code>, or a non-zero number, for a folder. Absent or false means a file.
|-
| <code>size</code> || number || Size in bytes. Use 0 for folders.
|-
| <code>modified</code> || string || The last-modified time in ISO-8601, UTC, for example <code>"2026-08-14T09:12:03Z"</code>. Omit it when you do not know it.
|}

Members other than these are ignored, so you may return extra information for your own clients.

<pre>
[
  {"name": "Q1.csv",  "isFolder": false, "size": 20481, "modified": "2026-04-02T17:45:10Z"},
  {"name": "Q2.csv",  "isFolder": false, "size": 21118, "modified": "2026-07-01T08:03:55Z"},
  {"name": "archive", "isFolder": true,  "size": 0}
]
</pre>

<code>recursive=1</code> asks for the whole subtree below the folder. When you answer one, <code>name</code> should carry the path '''relative to the folder that was listed''' -- <code>"archive/2025/Q4.csv"</code>, not just <code>"Q4.csv"</code>. That is what lets [[FileSystemListing]] report the sub-path the way it does for a local recursive listing. With <code>recursive=0</code>, return only the immediate children.

The engine, not the service, applies any wildcard the model wrote in the last segment of the path, and applies the «files» and «folders» flags of [[FileSystemListing]]. Return the folder's whole contents and let it do that filtering.

== Sizes, timeouts and cancellation ==

* The engine aborts a fetch as soon as it can see that the transfer exceeds <code>FileStoreMaxMB</code> (256 MB by default), whether it learns that from <code>Content-Length</code> or from the bytes received so far. Send a <code>Content-Length</code> where you can, so that an oversized file is refused before it is transferred rather than during.
* The connect timeout is 5 seconds and is not configurable. The read timeout is <code>FileStoreTimeoutS</code>, 60 seconds by default. A service that has slow work to do before it can produce a file should still start responding promptly.
* Streaming or chunking a large body is fine, and is preferable to buffering the whole thing in the service.
* A user can cancel a transfer that is in progress. From the service's point of view that looks like the client closing the connection part-way through a response, which the service should handle gracefully.

== HTTP only ==

The engine's HTTP client has no TLS support, so the base URL must be <code>http://</code>; an <code>https://</code> URL is refused when the provider is registered. Bind the service to <code>127.0.0.1</code>, or to a private network or cluster network only. The bearer token and the file bytes are both in the clear over that hop. See [[Custom file system providers#Security]].

== An illustrative skeleton ==

The smallest thing that works, in Python, serving a couple of directory trees. It is here to show the shape of the three endpoints and of the listing JSON; it is not something to deploy. It uses Python's single-threaded development server, and its traversal checking is only the obvious one.

<pre>
import datetime, json, os, urllib.parse
from http.server import BaseHTTPRequestHandler, HTTPServer

ROOTS = {"Sales": "/srv/data/sales", "models": "/srv/data/models"}
TOKEN = "«bearer-token»"

def resolve(rest):                          # "Sales/2026/Q3.csv" -> a local path
    parts = [p for p in urllib.parse.unquote(rest).split("/") if p]
    if not parts or ".." in parts or parts[0] not in ROOTS:
        return None
    return os.path.join(ROOTS[parts[0]], *parts[1:])

def entry(dirpath, name, rel):
    full = os.path.join(dirpath, name)
    isdir = os.path.isdir(full)
    when = datetime.datetime.utcfromtimestamp(os.path.getmtime(full))
    return {"name": rel + name, "isFolder": isdir,
            "size": 0 if isdir else os.path.getsize(full),
            "modified": when.strftime("%Y-%m-%dT%H:%M:%SZ")}

class Store(BaseHTTPRequestHandler):
    def reply(self, code, body=b"", ctype="application/octet-stream"):
        self.send_response(code)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        if self.command != "HEAD":
            self.wfile.write(body)

    def do_GET(self):                       # HEAD arrives here too
        if self.headers.get("Authorization") != "Bearer " + TOKEN:
            return self.reply(401)
        url = urllib.parse.urlparse(self.path)
        kind, _, rest = url.path.lstrip("/").partition("/")
        target = resolve(rest)
        if target is None:
            return self.reply(404)
        if kind == "files" and os.path.isfile(target):
            with open(target, "rb") as f:
                return self.reply(200, f.read())
        if kind == "list" and os.path.isdir(target):
            recurse = urllib.parse.parse_qs(url.query).get("recursive") == ["1"]
            out = []
            for dirpath, dirs, files in os.walk(target):
                rel = os.path.relpath(dirpath, target).replace(os.sep, "/")
                rel = "" if rel == "." else rel + "/"
                out += [entry(dirpath, n, rel) for n in dirs + files]
                if not recurse:
                    break
            return self.reply(200, json.dumps(out).encode(), "application/json")
        return self.reply(404)

    do_HEAD = do_GET

HTTPServer(("127.0.0.1", 8080), Store).serve_forever()
</pre>

== Testing your provider ==

# Write a configuration file that points at the service and gives it a scheme: a <code>FileStore repo = http://127.0.0.1:8080</code> line, plus <code>FileStoreToken repo = «bearer-token»</code> if the service wants one.
# Make Analytica find that file -- with <code>/stores:«path»</code>, with the <code>FileProvidersConfig</code> registry value, or by putting it beside the engine binary. See [[Custom file system providers#Where Analytica looks for FileProviders.config|Where Analytica looks for FileProviders.config]].
# Start Analytica and look on <code>stderr</code> for the registration line, <code>[FileProviders] repo:// -> http://127.0.0.1:8080</code>. If it is not there, the configuration file was not found, or the line in it was not understood.
# List the root with <code>[[FileSystemListing]]( "repo://" )</code>. That exercises <code>/list</code> and your JSON shape without needing any file to be readable.
# Read something with <code>[[ReadTextFile]]( "repo://Sales/Q3.csv" )</code>. That exercises <code>/files</code>, including the <code>HEAD</code> existence check that runs first.
# Then check the failure paths deliberately: a name that does not exist (expect a plain not-found), a bad token (expect a credentials error naming the scheme), and the service stopped (expect ''could not reach the repo:// file store'').

== See Also ==
* '''[[Custom file system providers]]''' -- the parent page: what providers are, and how to configure them
* [[Analytica Command Line]] -- the <code>/stores:«path»</code> option
* [[FileSystemListing]]
* [[ReadTextFile]]
* [[ReadBinaryFile]]
* [[:Category:File system functions]]