ReadFromUrl

Requires Analytica Enterprise

ReadFromUrl(url)

Reads a page or text file from the given URL and returns the result as a text value.

Using this function, you can read from an HTTP or HTTPs page (e.g., web page or secure web page), from FTP, or from GOPHER. You can also submit data as would occur on a web page when a user submits an HTTP form. The content retrieved must be text. It can also be used to "call" web services to obtain data. The data is returned as a text value, so in general, you will need to use other Analytica functions to parse the data.

There are several optional parameters that may be useful in various context.

method: The HTTP method, usually "GET" or "POST". (More esoteric methods include "HEAD", "PUT", "DELETE", "TRACE", "OPTIONS", "CONNECT", and potential custom service methods.
formFields, formValues, formIndex: Data for form field values submitted to a web page. Most often, just «formFields» and «formValues» are used, which must be arrays with a common index. When array abstracting such that multiple indexes could be in common, then you should specify the common index using the «formIndex» parameter.
httpHeaders: Additional HTTP headers (separate with Chr(13)).
httpContent: custom submitted HTTP content

Reading from a Web page

To obtain the contents of a web page, which is usually in HTML format, just supply the URL, e.g.:

ReadFromUrl("http://lumina.com")

If you leave off the "http://" part, it defaults to an http query, e.g.:

ReadFromUrl("lumina.com")

Obtaining an image from a web page

Images with recognized image formats (i.e., JPG, PNG, GIF, etc) can be downloaded from HTTP pages but just supplying the URL, e.g.:

ReadFromUrl("lumina.com/images/lum_AnalyticaLogo_Tagline_Snagged.png")

The result is a picture value object. The result can then be assigned to the Pict attribute of an object on a diagram in order to view the image from within the model (and to cause the image to be saved with the model file).

Submitting Data to a Web Page

To simulate the submission of HTML form data when querying a web page, you can either submit the information using GET or POST methods. With a form uses a GET method, you would normally see the parameters appear on the URL itself. With a POST method, you would normally not see the parameters -- they would be passed in the body of the HTTP request.

To submit form data, you need to set up an array of fields and an array of field names. The fields and field names need to share a common index. A common way to do this is to create the field names index as a list-of-labels, and then create a table based on this index for the fields.

The function call uses these parameters:

ReadFromUrl(url, method, formValues, formFields, formIndex)

The «formIndex» parameter is the index that «formValues» and «formFields» have in common. When it is guaranteed that there will be only one index in common, such as when «formFields» is an index, then the «formIndex» parameter is unnecessary.

For example, the following queries Google for "Analytica":

Index fieldNames := ["hl", "q"];

Var form := Array(fieldNames, ["en", "Analytica"]);

ReadFromUrl("http://google.com/search", "GET", form, fieldNames)

The result obtained is in HTML format.

You do not have to worry about URL-encoding the field names or values. If there are non-alpha numeric characters in either, they will be encoded before they are submitted.

Obtaining text content from an FTP site

To obtain content from an FTP site, use:

ReadFromUrl("ftp://site.com/directory/file.txt")

Keep in mind that the content must be textual. Binary files will be corrupted as they are read into a text value (specifically, '\0' characters will be converted to spaces).

Authentication

FTP sites commonly require user and password authentication, as is also common for web services and some web pages. To authenticate, embed the user name and password in the URL as follows:

ReadFromUrl("http://user:password@www.site.com/dir/page.htm")

ReadFromUrl("ftp://user:password@www.site.com/dir/page.htm")

Sending data CGI scripts and proprietary web services

If you call a CGI service, the CGI program will accept input in its own format, which could be arbitrary. You can send this data as follows:

ReadFromUrl("http://server.com/cgi-bin/myProgram.cgi", httpContent: data)

In this example, data contains the data that you are submitting to the CGI script. This data will form the body of the HTTP request. Content may only be used with HTTP requests, not with FTP or Gopher requests.

Various web services fall into this category as well, where data being submitted via HTTP may be in a proprietary format.

In some cases, you may also need to include additional HTTP headers in your request. You can insert these using the optional «httpHeaders» parameter. If you have more than one HTTP header, separate them using a CR character, Chr(13). If you enter these into a definition (e.g., with quotes), you can just type a new line, or if you enter them into a single edit table cell, pressing ALT-ENTER to insert a CR (new-line) into the cell. Otherwise, you can use the & operator to concatenate each header line with Chr(13).

ReadFromUrl("http://somehost.com", httpHeaders: "Accept:text/xml"&Chr(13)&"User-Agent:MyModel.ana")

The content value is passed directly with no special encoding of characters.

HTTP Status Codes

During an HTTP request, if the status code returned by the server is greater than or equal to 400, ReadFromUrl issues a warning displaying the status code and status text, unless the Show Result Warnings preference is turned off.

Proxy Servers

ReadFromUrl uses information stored in the system registry to determine whether or a proxy server should be used to access the internet. The configuration can be set up using Internet Explorer or Chrome web browsers (but not through Firefox). The proxy configuration is stored in the system registry in the hive:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings

in the registry values ProxyEnable, ProxyServer and ProxyOverride.

It should work with TIS, SOCKS and CERN style proxy servers as long as Internet Explorer is installed (SOCKS support requires IE).

Limitations

The function waits until the full request from the server has been received. It does not respond to Ctrl+Break. Some requests (usually if there is a problem), may take a while to time out before you can return to using Analytica.

On computers that aren't always connected to the internet, and need to dial up a modem or other connection to obtain access, the function does nothing to attempt to establish a connection. It will simply fail with an error.

Several possible error conditions return a cryptic message with only an error code number.

Binary content cannot be downloaded. The function can be used to download pictures from HTTP sources, but otherwise content must be text.

The function cannot be used to access file system files, e.g., you can't use a url starting with "file://...".

History

ReadFromUrl was introduced in Analytica 4.2.