ReadFromUrl

Release:	4.6 • 5.0 • 5.1 • 5.2 • 5.3 • 5.4 • 6.0 • 6.1 • 6.2 • 6.3 • 6.4 • 6.5

Reads a page or text file from a URL and returns the result as a text value.

Requires Analytica Enterprise

ReadFromUrl(url)

This function can read text data from an HTTP or HTTPs page (a standard or secure web page), from FTP, or from GOPHER. You can also submit data as would occur on a web page when a user submits an HTTP form. You can also use it to "call" a web service to obtain data. Its result is a text value. You can use functions like ParseCSV() to parse the text into data.

There are several optional parameters:

«method»: The HTTP method, usually "GET" or "POST". More esoteric methods include "HEAD", "PUT", "DELETE", "OPTIONS". The verbs "CONNECT", "PATCH" and "TRACE" cannot be used.
«formFields», «formValues», «formIndex»: Data for form field values to submit to a web page. Usually, you use «formValues» as an array indexed by «formFields», or if they both have a single common index. If «formValues» and «formFields» have multiple indexes in common, you should specify the common index as «formIndex».
«httpHeaders»: Additional HTTP headers (separate with Chr(13)).
«httpContent»: custom submitted HTTP content{{Release|6.4||
«fieldHeaders»: Used only when sending multi-part requests. In multipart messages, the parts and the content for each part are passed in the «formFields» and «formValues» parameters, and can be text, images or binary blobs. But in general you also can specify separate httpHeaders for each part, which are passed in this parameter. The «httpHeaders» is for the full message, whereas the headers here (each is multi-line text) are the headers for each part. «fieldHeaders» is indexed by «formIndex». By specifying this parameter, the message will be configured as multipart.
«connectTimeoutMs», «sendTimeoutMs», «receiveTimeoutMs»: Timeouts for the key stages of connecting to the server, sending the request, and receiving the response, each expressed in milliseconds.

Return values

ReadFromUrl has multiple return values:

The response. Could be text, image, or binary.
HTTP status code. This is 200 when successful.
HTTP status code textual description.
The HTTP response headers
The content-type (if specified)

For example, to capture the status code, use:

Local ( response, code ) := ReadFromUrl(....);

or to avoid a warning and handle a code<>200 yourself,

Local ( response, code, errMsg, headers, contentType ) := IgnoreWarnings(ReadFromUrl(....));

Reading from a Web page

To read the contents of a web page,usually in HTML format, just supply the URL, e.g.:

ReadFromUrl("http://lumina.com")

If you omit the "http://" part, it defaults to an http query, e.g.:

ReadFromUrl("lumina.com")

Obtaining an image from a web page

ReadFromUrl() can download images with standard formats (i.e., JPG, PNG, GIF, etc) from HTTP pages by supplying the URL, e.g.:

ReadFromUrl("analytica.com/wp-content/uploads/2023/07/analytica-img.png")"

The result is a picture value object. You can assign the result to the Pict attribute of an object on a diagram to view the image in the model (and to cause the image to be saved with the model file).

Submitting Data to a Web Page

To simulate the submission of HTML form data when querying a web page, you can submit the information using either GET or POST methods. With a GET method, you normally see the parameters appear on the URL itself. With a POST method, you normally don't see the parameters -- they are passed in the body of the HTTP request.

To submit form data, you can set up an table of fields values indexed by field names, usally as an index as a list-of-labels. The function call uses these parameters:

ReadFromUrl(url, method, formValues, formFields, formIndex)

You only need to specify the «formIndex» parameter if «formValues» and «formFields» are arrays that may have more than one index in common.

For example, the following queries Google for "Analytica":

INDEX fieldNames := ["hl", "q"];

VARIABLE form := Array(fieldNames, ["en", "Analytica"]);

ReadFromUrl("http://google.com/search", "GET", form, fieldNames)

It returns the result in HTML format.

You don't have to worry about URL-encoding the field names or values. If they containnon-alpha numeric characters in either, it will encode them before they are submitted.

Submitting multi-part data

Requires Analytica 6.4

Data sent in an HTTP method:post can usually be formatted in two ways:

Content-Type:application/x-www-form-urlencoded
Content-Type:multipart/form-data

When a form value contains binary data or an image, you must use the multipart format. By default, it uses Content-Type:application/x-www-form-urlencoded when possible, but uses Content-Type:multipart/form-data when a form value requires a multipart format.

To force a multipart format when it is not required (e.g. if the web-service you requires a multipart format), include the header explicitly, e.g.:

ReadFromURL( url, method:'POST', formValues:vals, formIndex:I, httpHeaders: 'Content-Type:multipart/form-data' )

Or you can specify the «fieldHeaders» parameter (even by passing it an empty string). In either case, it generates a boundary string between parts. If you provide your own boundary, it will use it, but you need to be careful that this exact sequence of bytes does not appear in any of the content. For example

ReadFromURL( url, method:'POST', formValues:vals, formIndex:I, httpHeaders: 'Content-Type:multipart/form-data;boundary=MyBoundary' )

would malfunction if one of your vals contains the byte sequence "MyBoundary".

When sending multi-part data, you may also want to add custom headers to individual items. You can do this using the «fieldHeaders» parameter, which also uses the «formIndex». One example is when you want a field to appear to be the contents of a file and you want to provide the file name.

LocalIndex Field := ['Date','Upload'];

Local fileContents := ReadBinaryFile( filename, typeFlags: 7 {binary data blob} );

Local values:= Array( Field, [ Today(), fileContents ] );

Local specialHeader := f"Content-Disposition: form-data; name=""File""; filename=""{filename}""";

Local fieldHeaders := Array( Field, [ "", specialHeader ] );

ReadFromURL( url, 'Post', formValues: values, formIndex: Field, fieldHeaders: fieldHeaders )

In this code, specialHeader creates a header field for the form item 'Upload' which includes the file name. The server who receives this will think you used an HTML file upload field to select a file named filename. In addition, the server will see the field name as being "File" (from the specialHeader) overriding the name "Upload" in the «formIndex» parameter.

Note that if you are not adding a filename or altering the usual Content-Disposition, you don't have to specify it as a fieldHeader. If you don't specify it, the function automatically creates Content-Disposition for each part.

A second common case where you might need to customize the «fieldHeaders» is when you need to customize the Content-Type. For example, suppose you upload a video file, which you read into memory as a binary data term. If you don't specify the content-type, it will default to Content-Type: application/octet-stream. Thus, you might specify it for the field:

ReadFromURL(url, "Post", vals, fields, I, fieldHeaders: If I="video clip" then "Content-Type: video/mp4" else "" )

If you are sending only a single image or a single binary data term, it is usually simpler to pass it as the «httpContent» rather than using form fields at all. In that case the message is a single part (not multipart) body, assuming you also control what the receiving side expects.

Obtaining content from an FTP site

To obtain content from an FTP site, use:

ReadFromUrl("ftp://site.com/directory/file.txt")

The content must be textual. Binary files will be corrupted as they are read into a text value (specifically, '\0' characters will be converted to spaces).

Authentication

FTP sites usually require user and password authentication, as do web services and some web pages. You can embed the user name and password in the URL like this:

ReadFromUrl("http://user:password@www.site.com/dir/page.htm")

ReadFromUrl("ftp://user:password@www.site.com/dir/page.htm")

If the user or password contains any of the special characters @ : / ? #, these must be percent encoded. Thus an email user name, john.doe@gmail.com, would be

ReadFromURL("https://john.doe%40gmail.com@domain.com")

Here are the % codes for the other special chars that might appear in a username or password:

Char	Code
Space	%20
@	%40
:	%3A
/	%2F
?	%3F
#	%23

Sending data CGI scripts and proprietary web services

If you call a CGI service, the CGI program will accept input in its own format, which could be arbitrary. You can send this data as follows:

ReadFromUrl("http://server.com/cgi-bin/myProgram.cgi", httpContent: data)

In this example, data contains the data that you are submitting to the CGI script. This data will form the body of the HTTP request. Content may only be used with HTTP requests, not with FTP or Gopher requests.

Various web services fall into this category as well, where data being submitted via HTTP may be in a proprietary format.

In some cases, you may also need to include additional HTTP headers in your request. You can insert these using the optional «httpHeaders» parameter. If you have more than one HTTP header, separate them using a CR character, Chr(13). If you enter these into a definition (e.g., with quotes), you can just type a new line, or if you enter them into a single edit table cell, pressing ALT-ENTER to insert a CR (new-line) into the cell. Otherwise, you can use the & operator to concatenate each header line with Chr(13).

ReadFromUrl("http://somehost.com", httpHeaders: "Accept:text/xml"&Chr(13)&"User-Agent:MyModel.ana")

The content value is passed directly with no special encoding of characters.

HTTP Status Codes

During an HTTP request, if the status code returned by the server is greater than or equal to 400, ReadFromUrl issues a warning displaying the status code and status text, unless the Show Result Warnings preference is turned off.

Proxy Servers

ReadFromUrl uses information stored in the system registry to determine whether or a proxy server should be used to access the internet. The configuration can be set up using Internet Explorer or Chrome web browsers (but not through Firefox). The proxy configuration is stored in the system registry in the hive:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings

in the registry values ProxyEnable, ProxyServer and ProxyOverride.

It should work with TIS, SOCKS and CERN style proxy servers as long as Internet Explorer is installed (SOCKS support requires IE).

Limitations

The function waits until the full request from the server has been received. It does not respond to Ctrl+Break. Some requests (e.g. if there is a problem), may take a while to time out before you can return to using Analytica.

On computers that aren't always connected to the internet, or need to dial up a modem or other connection to obtain access, the function does nothing to attempt to establish a connection. It will simply fail with an error.

Several possible error conditions return a cryptic message with only an error code number.

If cannot download binary data. It can only obtain text or pictures from HTTP sources.

It cannot access file system files, e.g., you can't use a url starting with "file://...".

History

ReadFromUrl was introduced in Analytica 4.2.

Analytica 6.4 added support useful for some web service calls that includes

the transfer of binary data
multi-part message payloads in both directions
Explicit control over timeouts
Access to the HTTP status code and text, the HTTP response headers, and the content type (as aux return values).

Analytica 6.5 added asynchronous connection types (WebConnections), and the 'asynchWait' connection type that responds to Ctrl+Break and windows events (like redraw events) while waiting for a lengthy web service call.