ReadFromUrl

Release:	4.6 • 5.0 • 5.1 • 5.2 • 5.3 • 5.4 • 6.0 • 6.1 • 6.2 • 6.3 • 6.4

Requires Analytica Enterprise

ReadFromUrl(url)

Reads a page or text file from the given URL and returns the result as a text value.

Using this function, you can read from an HTTP or HTTPs page (e.g., web page or secure web page), from FTP, or from GOPHER. You can also submit data as would occur on a web page when a user submits an HTTP form. The content retrieved must be text. It can also be used to "call" web services to obtain data. The data is returned as a text value, so in general, you will need to use other Analytica functions to parse the data.

There are several optional parameters that may be useful in various context.

method: The HTTP method, usually "GET" or "POST". (More esoteric methods include "HEAD", "PUT", "DELETE", "TRACE", "OPTIONS", "CONNECT", and potential custom service methods.
formFields, formValues, formIndex: Data for form field values submitted to a web page. Most often, just «formFields» and «formValues» are used, which must be arrays with a common index. When array abstracting such that multiple indexes could be in common, then you should specify the common index using the «formIndex» parameter.
httpHeaders: Additional HTTP headers (separate with Chr(13)).
httpContent: custom submitted HTTP content

Reading from a Web page

To obtain the contents of a web page, which is usually in HTML format, just supply the URL, e.g.:

ReadFromUrl("http://lumina.com")

If you leave off the "http://" part, it defaults to an http query, e.g.:

ReadFromUrl("lumina.com")

Obtaining an image from a web page

Images with recognized image formats (i.e., JPG, PNG, GIF, etc) can be downloaded from HTTP pages but just supplying the URL, e.g.:

ReadFromUrl("lumina.com/images/lum_AnalyticaLogo_Tagline_Snagged.png")

The result is a picture value object. The result can then be assigned to the Pict attribute of an object on a diagram in order to view the image from within the model (and to cause the image to be saved with the model file).

Submitting Data to a Web Page

To simulate the submission of HTML form data when querying a web page, you can either submit the information using GET or POST methods. With a form uses a GET method, you would normally see the parameters appear on the URL itself. With a POST method, you would normally not see the parameters -- they would be passed in the body of the HTTP request.

To submit form data, you need to set up an array of fields and an array of field names. The fields and field names need to share a common index. A common way to do this is to create the field names index as a list-of-labels, and then create a table based on this index for the fields.

The function call uses these parameters:

ReadFromUrl(url, method, formValues, formFields, formIndex)

The «formIndex» parameter is the index that «formValues» and «formFields» have in common. When it is guaranteed that there will be only one index in common, such as when «formFields» is an index, then the «formIndex» parameter is unnecessary.

For example, the following queries Google for "Analytica":

Index fieldNames := ["hl", "q"];

Var form := Array(fieldNames, ["en", "Analytica"]);

ReadFromUrl("http://google.com/search", "GET", form, fieldNames)

The result obtained is in HTML format.

You do not have to worry about URL-encoding the field names or values. If there are non-alpha numeric characters in either, they will be encoded before they are submitted.

Submitting multi-part data

Requires Analytica 6.4

Data sent in an HTTP method:post can, in general, be formatted in two ways:

Content-Type:application/x-www-form-urlencoded
Content-Type:multipart/form-data

When a form value containing binary data or an image, then only the multipart format can be used. By default, Content-Type:application/x-www-form-urlencoded is used if possible, but if there is a form value that requires multipart, then Content-Type:multipart/form-data is used.

To force a multipart format when it is not required (which you might need to do if the web-service you are calling requires a multipart format), include the header explicitly, e.g.:

ReadFromURL( url, method:'POST', formValues:vals, formIndex:I, httpHeaders: 'Content-Type:multipart/form-data' )

A second way to force it to use multipart is to specify the «fieldHeaders» parameter (even by passing it an empty string). In either case, a boundary string between parts will be automatically generated. If you provide your own boundary, it will use it, but you need to be careful that this exact sequence of bytes does not appear in any of the content. For example

ReadFromURL( url, method:'POST', formValues:vals, formIndex:I, httpHeaders: 'Content-Type:multipart/form-data;boundary=MyBoundary' )

would malfunction if one of your vals contains the byte sequence "MyBoundary".

When sending multi-part data, you may also want to add custom headers to individual items. You can do this using the «fieldHeaders» parameter, which also uses the «formIndex». One example is when you want a field to appear to be the contents of a file and you want to provide the file name.

LocalIndex Field := ['Date','Upload'];

Local fileContents := ReadBinaryFile( filename, typeFlags: 7 {binary data blob} );

Local values:= Array( Field, [ Today(), fileContents ] );

Local specialHeader := f"Content-Disposition: form-data; name=""File""; filename=""{filename}""";

Local fieldHeaders := Array( Field, [ "", specialHeader ] );

ReadFromURL( url, 'Post', formValues: values, formIndex: Field, fieldHeaders: fieldHeaders )

In this code, specialHeader creates a header field for the form item 'Upload' which includes the file name. The server who receives this will think you used an HTML file upload field to select a file named filename. In addition, the server will see the field name as being "File" (from the specialHeader) overriding the name "Upload" in the «formIndex» parameter.

Note that if you are not adding a filename or altering the usual Content-Disposition, you don't have to specify it as a fieldHeader. If you don't specify it, the function automatically creates Content-Disposition for each part.

A second common case where you might need to customize the «fieldHeaders» is when you need to customize the Content-Type. For example, suppose you upload a video file, which you read into memory as a binary data term. If you don't specify the content-type, it will default to Content-Type: application/octet-stream. Thus, you might specify it for the field:

ReadFromURL(url, "Post", vals, fields, I, fieldHeaders: If I="video clip" then "Content-Type: video/mp4" else "" )

Note that if you are sending only a single image or a single binary data term, it is usually simpler to pass it as the «httpContent» rather than using form fields at all. In that case the message is a single part (not multipart) body, assuming you also control what the receiving side expects.

Obtaining text content from an FTP site

To obtain content from an FTP site, use:

ReadFromUrl("ftp://site.com/directory/file.txt")

Keep in mind that the content must be textual. Binary files will be corrupted as they are read into a text value (specifically, '\0' characters will be converted to spaces).

Authentication

FTP sites commonly require user and password authentication, as is also common for web services and some web pages. To authenticate, embed the user name and password in the URL as follows:

ReadFromUrl("http://user:password@www.site.com/dir/page.htm")

ReadFromUrl("ftp://user:password@www.site.com/dir/page.htm")

Sending data CGI scripts and proprietary web services

If you call a CGI service, the CGI program will accept input in its own format, which could be arbitrary. You can send this data as follows:

ReadFromUrl("http://server.com/cgi-bin/myProgram.cgi", httpContent: data)

In this example, data contains the data that you are submitting to the CGI script. This data will form the body of the HTTP request. Content may only be used with HTTP requests, not with FTP or Gopher requests.

Various web services fall into this category as well, where data being submitted via HTTP may be in a proprietary format.

In some cases, you may also need to include additional HTTP headers in your request. You can insert these using the optional «httpHeaders» parameter. If you have more than one HTTP header, separate them using a CR character, Chr(13). If you enter these into a definition (e.g., with quotes), you can just type a new line, or if you enter them into a single edit table cell, pressing ALT-ENTER to insert a CR (new-line) into the cell. Otherwise, you can use the & operator to concatenate each header line with Chr(13).

ReadFromUrl("http://somehost.com", httpHeaders: "Accept:text/xml"&Chr(13)&"User-Agent:MyModel.ana")

The content value is passed directly with no special encoding of characters.

HTTP Status Codes

During an HTTP request, if the status code returned by the server is greater than or equal to 400, ReadFromUrl issues a warning displaying the status code and status text, unless the Show Result Warnings preference is turned off.

Proxy Servers

ReadFromUrl uses information stored in the system registry to determine whether or a proxy server should be used to access the internet. The configuration can be set up using Internet Explorer or Chrome web browsers (but not through Firefox). The proxy configuration is stored in the system registry in the hive:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings

in the registry values ProxyEnable, ProxyServer and ProxyOverride.

It should work with TIS, SOCKS and CERN style proxy servers as long as Internet Explorer is installed (SOCKS support requires IE).

Limitations

The function waits until the full request from the server has been received. It does not respond to Ctrl+Break. Some requests (usually if there is a problem), may take a while to time out before you can return to using Analytica.

On computers that aren't always connected to the internet, and need to dial up a modem or other connection to obtain access, the function does nothing to attempt to establish a connection. It will simply fail with an error.

Several possible error conditions return a cryptic message with only an error code number.

Binary content cannot be downloaded. The function can be used to download pictures from HTTP sources, but otherwise content must be text.

The function cannot be used to access file system files, e.g., you can't use a url starting with "file://...".

History

ReadFromUrl was introduced in Analytica 4.2.