This site will look much better in a browser that supports web standards, but is accessible to any browser or Internet device.

HTTPTest Documentation

NAME

HTTPtest.pl

Purpose

HTTPtest is a generalized HTTP request test program. It allows for efficient regressions testing of any HTTP-based server.

Warning: This program is under construction, so the documentation is subject to change.

SCRIPT CATEGORIES

HTTP - suggested

PREREQUISITES

This script depends on both the strict and vars pragmas. The script also uses the FindBin LWP::UserAgent, HTTP::Cookies, URI::Escape, XML::Parser, Getopt::Std, and File::Spec modules.

COREQUISITES

If the Time::HiRes module is available, it is used to generate higher-resolution timings on the time subtest and the script timing.

OSNAMES

any. Tested on MSWin32 and Linux.

Description

The HTTPtest program uses a battery file to describe a series of tests to perform using the HTTP protocol. In general, a set of tests are aimed at a single server or service. Like HTTP itself, the tests are defined in terms of a request and a response. The battery file describes the tests in an XML-based format, described below.

HTTPtest reads each test in the battery file, performs the specified request, and evaluates the response based on the criteria described in the test. If all criteria are met, the test is successful. Otherwise, the test fails.

Usage

HTTPtest is executed as follows:

 perl HTTPtest.pl [-vwqVl] [-o logfile] [-I incpath] [-f suite] [-jJ cjar] [name=value ...] test.files ...
   Where:  -v   verbose output
           -w   normal output, default
           -q   quiet output
           -V   print version information
           -l   log results to HTTPtest.log
           -o   send results to specified logfile
           -I   semicolon-separated path to check for include files
           -f   supply a test suite filename
           -j   use a temporary cookie jar
           -J   use the cookie jar file 'cjar'
   name=value pairs override parameters in the test files
   test.files is a list of files that contain the tests to run

If run with a bad option or no test files, the above message is printed.

Battery File

The battery file contains a battery root element. The battery may contain include elements to include information from other files, param elements which factor out reused parameters, and test elements which describe the tests to run.

Each test consist of a request, which describes the HTTP request, and a response which describes the criteria for evaluating the HTTP response. A request specifies the URL to call including any query parameters, an HTTP method such as GET or POST, and an optional request body.

A response contains a series of subtests which include the ability to test:

status
The response status code for the response. Reasonable values include 200 and 404.

header
Tests for the existence or value of particular MIME-headers in the response. Useful for verifying redirections and testing returned cookies.

content
Evaluation subtest for the content of response.

time
Compare the response time of the URL request against a minimum and maximum timespan in seconds.

code
Execute arbitrary Perl code on the response to evaluate the response.

msg
This pseudo-subtest displays a message to the user and optionally aborts the current battery or program run.

filter
This pseudo-subtest executes Perl code specifically designed to perform some form of filtering on the response. Always succeeds.

call
This pseudo-subtest executes a list of subtests defined using the subtest-def element.

By default all of the evaluations in the response must succeed in order for the test to succeed. However, five other elements are supported to allow for arbitrary logical combinations of the evaluations.

and
All of the subtests contained in the and element must succeed in order for the and to succeed. Any unsuccessful evaluation causes the and to fail without executing any remaining children.

and-all
All of the subtests contained in the and-all element must succeed in order for the and-all to succeed. Unlike the and subtest, all children are executed regardless of the success or failure of any child.

or
One of the subtests contained in the or element must succeed in order for the or to succeed. Any successful evaluation causes the or to succeed without executing any remaining children.

or-all
One of the subtests contained in the or-all element must succeed in order for the or-all to succeed. Unlike the or subtest, all children are executed regardless of the success or failure of any child.

not
The sense of the subtest contained in the not element is reversed, so that a failure becomes a success.

Element List

and

The and element is used to combine a set of subtests in a logical and relationship. The and element can contain any of the subtest elements or the logical elements. The and element has one optional attribute label which is a displayable description of the set of subtests contained in the and. It may be used in logging and failure reporting.

HTTPtest evaluates the response based on each of the children of the and in turn. When a subtest fails, no more children are tested and the and evaluation fails.

and-all

The and-all element is used to combine a set of subtests in a logical and relationship. The and-all element can contain any of the subtest elements or the logical elements. The and-all element has one optional attribute label which is a displayable description of the set of subtests contained in the and-all. It may be used in logging and failure reporting.

HTTPtest evaluates the response based on each of the children of the and-all in turn. Unlike the and subtest, all of the children are evaluated regardless of the success or failure of any given child.

auth

The auth element supplies an optional user id and password for HTTP authentication of a request. Currently only supports Basic authentication. The auth element has two reuired attributes user and password. These attributes have the obvious values.

battery

The battery is the root element of the battery file. It can contain zero or more param elements, followed by zero or more codelib elements, followed by zero or more test elements. In addition, include elements may occur anywhere in the battery element.

The battery has one required attribute, label which is a short displayable description of the test set.

body

The body element supplies the content for an HTTP request. It is currently only supported for the POST HTTP method. The content of this element is used as the body of the request by HTTPtest. Since this content resides in an XML document, it may be necessary to wrap the content in a CDATA section in order for the file to remain well-formed.

The body element has two optional attributes. The type attribute specifies the MIME type of the content. By default, the content type is application/x-www-form-urlencoded, which is the type used by browsers on HTML forms. The encode attribute specifies whether the content needs to be URL-encoded. The default value is no. If you have already encoded the data, the correct answer is no.

call

The call element executes the list of subtests that have been defined in an earlier subtest_def element. This element allows a group of subtests to be factored out of a response and reused in other responses.

code

The code element specifies a piece of Perl code to execute in order to evaluate the HTTP response. The code element has one optional attribute. The label attribute provides a displayable description of this element that can be used in error reporting and logging.

The content of this element is raw Perl code. Since Perl code can easily violate the well-formedness criteria, it is recommended that the whole content of this element be placed in a CDATA section.

The Perl code is executed with a set of variables that are accessible. These variables can be used to determine how to perform the tests or communicate with other pieces of code. The variables of interest are:

$resp
This is a reference to an HTTP::Response object containing the current response. This is a my variable, and therefore only available directly inside the code element.

$label
This string is the value of the label attribute for the current test. This is a my variable, and therefore only available directly inside the code element.

%Params
This is a global hash matching all of the parameter names to their current values. The Perl code is allowed to modify values in this hash to pass data to other tests. Direct access to this hash is deprecated. Parameters should be accessed through the following methods:
param()
This is the get/set function for parameters. When called with one argument, the value of the parameter with the supplied name is returned. If the function is called with two arguments, the parameter named by the first argument is set to the value of the second argument. For example, 
    my $server = param( 'server' );
    param( 'server' => 'www.example.com' );

param_exists()
This function returns a boolean value that specifies whether the named parameter exists. It has one argument, the name of the parameter to test.

$parser
This is a global reference to an XML::Parser object configured with the Objects style. This object may be used to parse XML from the response to simplify testing. See the XMLdata.pm library file for tools that can manipulate this structure.

response_time()
This function returns the time in seconds for the HTTPresponse to return.

The Perl code should return a list of scalars. The first item in the list is the only required item. It is true if the test was successful, and false otherwise. The second item in the list is a short string describing the result of the test. The third and final item is a long string giving a verbose description of the result of the test. If either the second or third items is missing, a generic answer is used in each case.

codelib

The codelib element declares Perl code that is to be compiled into the test script for use in later tests. This is a good place to use modules or define routines that are needed by code subtest defined later in the file. The content of this element is raw Perl code. Since Perl code can easily violate the well-formedness criteria, it is recommended that the whole content of this element be placed in a CDATA section.

The codelib element has one required attribute and one optional attribute. The label attribute provides a displayable description of this element that can be used in error reporting and logging. The optional id attribute is used to uniquely identify a codelib. This helps HTTPtest to recognize if a codelib is encountered more than once. (For example through the use of include elements.) If a codelib id is encountered a second time, that codelib is not evaluated the second or subsequent times. If a codelib does not contain an id and it is encountered more than once, it is evaluated each time it is encountered.

content

The content element specifies subtest for the body of the HTTP response. The content element has one optional attribute. If the match attribute is missing or has a value of exact, the body of the HTTP response must match the content of the content element exactly. If the match attribute has a value of regexp, the content of the element is treated as a regular expression to be matched against the HTTP response content. If the match attribute has a value of substr, the content of the element is expected to exist somewhere in the response.

WARNING: There is a limit to the size of a regular expression in Perl. It is better to break a complicated test into several smaller tests.

filter

The filter element is a special subset of the functionality of the code element. The filter element has two optional attributes. The label attribute supplies a displayable label for this filter. The class attribute specifies which class of filter we are specifying. If the class attribute is missing or if it has the value content, the filter modifies the content of the response which is available in the variable $_. If the class attribute has the value response, the filter can affect any portion of the response. The response is available in the variable $resp as an HTTP::Response object.

header

The header element specifies a MIME header in either a request or a response. It has three attributes that are treated slightly differently depending on whether the element is contained in a request or a response. In both cases, the name attribute is required and supplies the name of the MIME header of interest.

The value attribute specifies the value of that MIME header. In the case of a request, this attribute is required and specifies the literal text of the value of the header. In the case of a response, the value attribute is interpreted as described by the match attribute.

The match attribute is specifies the interpretation of the value attribute on a header in a response. The default value of the match attribute is exact. The valid values for match are

exact
The value of the MIME header must exactly match the value of the header.

regexp
The value of the header is treated as a regular expression to match the value of the specified MIME header.

exist
The MIME header must exist in the HTTP response. The value attribute is ignored.

nonexist
The MIME header must not exist in the HTTP response. The value attribute is ignored.

include

The include element allows portions of a battery to be factored into separate files. This supports sharing of information between battery files and can simplify maintenance of test sets.

The include element has one required attribute (file) and two optional attributes (label and parse). The file attribute specifies the file to be included. HTTPtest looks for the file as it is specified, relative to the current directory, first. If the file is not found and the filename is not an absolute path, HTTPtest then attempts to find the file by appending the file attribute to each path in the include path. label attribute is a displayable description only used for logging and error reporting.

The parse attribute tells HTTPtest how to treat the data from the included file. If the parse attribute is missing or if its value is xml, the contents of the included file are parsed and all children of the root element replace the include element. If the value of the parse attribute is text, the contents of the file are treated as a text node that replaces the include element. Since this data is not handled directly by the XML parser, there is no need to use character entities such as < and &.

One extra constraint on the include is that the test set file should still be a valid document after the include elements are replaced by the content of their files.

inclusion

The inclusion element is the root element of any included xml-parsed file. Its purpose is to provide a well-formed XML document for the included file.

msg

The msg element is used to report unusual problems to the user and optionally to abort a test run. The msg element has two optional attributes and contains text to display. The print attribute determines when the message is printed. Possible values are:

always
Always print the message when the msg element is encountered.

normal
Print the message when the msg element is encountered and the -w or -v switches are active. This is the default printing behavior.

verbose
Print the message when the msg element is encountered and the -v switch is active.

The action attribute determines the effect of this pseudo-subtest on the test in progress. The possible values are:

success
Treat the msg element as a successful subtest.

fail
Treat the msg element as a failed subtest. This is the default action.

stop
Stop the current battery, but continue processing any battery files remaining on the command line.

exit
Exit the HTTPtest.pl script, altogether.

The msg element is best used as part of an or or and condition. This allows an earlier subtest to determine whether or not the msg element is processed at all.

not

The not element is used to logically invert the outcome of a subtest. The not element can contain any one of the subtest elements or the logical elements. The not element has one optional attribute label which is a displayable description. It may be used in logging and failure reporting.

option

The option element configures default processing options for HTTPtest. All of the options configurable through the option element are also available through command line parameters. The command line parameters override whatever options are in the test set file. All of the option elements must come before any other elements in the battery.

The option element has two required attributes, name and value. The value attributes will be interpreted differently depending on option they are applied to. The allowable options are:

incpath
The value attribute for this option is a semicolon-separated list of paths to use in finding include files. Unlike the other options, the command line switch does not override this option. Instead the paths supplied by the -I switch are prepended to the include path.

logging
The value for this option is either yes (enable logging) or no (disable logging). This is equivalent to the -l switch.

logfile
The value for this option is the name of a file into which the logging information is written. This is equivalent to the -o switch.

msgs
The value for this option is either an empty string (normal output), quiet (quiet mode), or verbose (verbose mode). This modes are equivalent to the -w, -q, and -v switches.

cookiejar
The value for this option is either an empty string (temporary cookie jar) or the name of a file to store the cookie jar. If this option is supplied, HTTPtest retains cookies across all of the tests in the battery. If no name is specified, the cookie jar will be empty when the script begins and all cookies wil be discarded at exit. This usage is equivalent to the -j switch. If a filename is specified, that file is read to initialize the cookie jar if the file exists. When the program prepares to exit, the current state of the cookie jar is saved to the specified file. This usage is equivalent to the -J switch.

or

The or element is used to combine a set of subtests in a logical or relationship. The or element can contain any of the subtest elements or the logical elements. The or element has one optional attribute label which is a displayable description of the set of subtests contained in the or. It may be used in logging and failure reporting.

HTTPtest evaluates the response based on each of the children of the or in turn. When an subtest succeeds, no more children are tested and the or subtest succeeds.

or-all

The or-all element is used to combine a set of subtests in a logical or relationship. The or-all element can contain any of the subtest elements or the logical elements. The or-all element has one optional attribute label which is a displayable description of the set of subtests contained in the or-all. It may be used in logging and failure reporting.

HTTPtest evaluates the response based on each of the children of the or-all in turn. Unlike the or subtest, all of the children are evaluated regardless of the success or failure of any given child.

param

The param element is an empty element which gives a name to a piece of text which is expected to be reused many times in the test set. A good example would be the name of the server undergoing test. For example,

  <param name="server" value="testserver.telescan.com"/>

Using a param in this way simplifies pointing the test to a new server. The param is then referenced in any text or attribute in the test set file using the form {{server}}.

In addition, the value of a param may be changed from the HTTPtest command line using the syntax: name=value. To run this battery on the server live.telescan.com, you would add server=live.telescan.com to the command line before the name of the battery file.

The param element has two required attributes. The name attribute specifies the name by which this parameter is referenced. The value attribute specifies the replacement text.

request

The request element describes a complete HTTP request to be used in the test. The request has a required attribute, url, which specifies the URL to which the request is made. This URL may contain a query string. Because of the requirements of XML, this query string will need to be modified if it contains the ampersand character &. The ampersand must be replaced by ``&amp;'' in the url attribute. This string will be converted back to its proper form before it is requested.

The request element also has an optional method attribute, which defaults to GET. Currently, HTTPtest supports values of GET and POST for this attribute. At some point, the program may be extended to support all of the HTTP request methods.

The request element can contain zero or more header elements followed by zero or more body elements. In addition, include elements may optionally occur anywhere in the request content. The header elements allow you to specify MIME headers to apply to the request. The body element is used to contain the content of a POST request. At this time, only one body element is supported in a request. However, at a future date, this will be extended to multiple body elements for support of the multipart Content types.

response

The response element defines the subtests to execute for the current test. After the HTTP request has been made and an HTTP response is returned, HTTPtest walks the children of the response element and attempts to verify the response.

The response element has one optional attribute. The label attribute defines a displayable description of this response set that may be used in error messages or logging.

The content of the response response element must include one or more of the following elements:

  • status

  • header

  • time

  • content

  • code

  • msg

  • filter

  • and

  • or

  • and-all

  • or-all

  • not

  • call

The response element may also contain include elements. After the include is resolved, the only child elements allowed in a response are the ones listed above.

session

The session element is a deprecated synonym for test-set.

skip

The skip element declares a Perl expression that can determine if the containing test test should not be executed. This expression can be used to test for command line options, system resources, current date or time, or anything else that might cause the running of the test to be a bad idea. The skip element has a required attribute if that specifies the Perl expression that determines the need to skip the test. If the expression evaluates to a true value, the test is skipped, otherwise the test is performed as normal. The skip element also supports an optional label attribute.

If the skip test succeeds and HTTPtest is running in verbose mode, the contents of the skip element are displayed as an explanation for the test being skipped. If the content is empty, the label is used. If there is no label, the actual test expression is displayed.

Skipped tests are reported separately from successful or failed tests. This allows you to count them whichever way makes the most sense.

status

The status element specifies values to match for the HTTP Response status code. The element has one required attribute and one optional attribute. The required code attribute specifies a value to use in the comparison. This value is interpreted based on the value of the optional match attribute. If the value of the match attribute is exact or the attribute is missing, the code must match the response exactly. If the value of the match attribute is regexp, the value of code is treated as a regular expression.

subtest_def

The subtest_def element supports naming a group of subtests so that they may be used from multiple responses.

test

The test element describes a particular test to be run. The test contains only zero or more skip elements, a request, and a response. The test also has one required attribute label, which is a displayable description of the test which is used in failure reporting.

testset

The testset element is a deprecated synonym for battery.

test-set

The test-set element groups a set of test elements together as a single group.

time

The time element specifies the expected time limits for an HTTP response. The time element has two optional attributes, min and max. If the min attribute is specified, the test fails if the HTTP response returns in less than that many seconds. If the max attribute is specified, the test fails if the HTTP response returns in more than that many seconds. If the value of min is greater than the value of max, the test will always fail.

Example Tests

The following is a very simple test. We are just testing to see if the page is returned in a reasonable amount of time. In this case, the test is successful if we recieve a status code of 200 (OK) and if the page returns in no more than 10 seconds.

  <test label="Did it come back">
    <request url="http://www.tscn.com/"; method="GET"/>
        <response>
          <status code="200"/>
          <time max="10"/>
        </response>
  </test>

To extend this test a little, we will check to see if the returned page is an HTML document. In this case, I am taking advantage of the fact that I know the page has the HTML tags in all upper case.

  <test label="Respond in HTML">
    <request url="http://www.tscn.com/"; method="GET"/>
        <response>
          <status code="200"/>
          <time max="10"/>
          <header name="Content-Type" value="text/html"/>
          <content match="regexp"><![CDATA[<HTML>.*</HTML>]]>
        </response>
  </test>

Now we will check to see if we are getting the page (or, at least the title) that we think we are getting back.

  <test label="Old WSC root page">
    <request url="http://www.tscn.com/"; method="GET"/>
        <response>
          <status code="200"/>
          <time max="10"/>
          <header name="Content-Type" value="text/html"/>
          <content match="regexp"><![CDATA[<HTML>.*</HTML>]]>
          <content match="regexp"><![CDATA[
<TITLE>Wall Street City - The Investor's Supersite on the Web</TITLE>
]]>
        </response>
  </test>

Example msg Tests

The msg element is a little more subtle than most of the other subtests. It is almost always used in conjunction with the or or and elements. The msg element serves two purposes: display a message alerting the user to an unusual condition, abort a battery in case an exceptional situation is detected.

In order to report an exceptional condition that has been recognized in the test, use the and element. For example,

  <test label="Part of a long sequence of tests">
    <request url="http://www.test.com/"; method="GET"/>
        <response>
          <status code="200"/>
      <and>
        <content match="regexp">This is exceptional</content>
        <msg action="stop" print="always">
            WARNING: Exceptional condition detected.
            Aborting test set...
        </msg>
      </and>
        </response>
  </test>

If, on the other hand, you want to report the failure of a critical subtest on that would affect any tests that follow, use the or element. For example,

  <test label="First test in a long set">
    <request url="http://www.test.com/"; method="GET"/>
        <response>
          <status code="200"/>
      <or>
        <content match="regexp">This MUST be found</content>
        <msg action="stop" print="always">
            WARNING: Required content missing.
            Aborting test set...
        </msg>
      </or>
        </response>
  </test>

The Include Path

HTTPtest will search the include path starting with the paths specified in the -I command line switch, in the order specified. Next, any paths specified in option elements in the order they are specified in the test set file. Finally, as a last resort, the subdirectory lib under the directory from which HTTPtest was run is checked.

Outstanding Issues

The following are issues still outstanding:

  • The codelib functionality has not been extensively tested.

  • The code functionality has not been extensively tested. In addition, it's functionality may change slightly in the near future.

  • Need to clean up the reporting and logging subsystems.

  • I do not like the name of the inclusion element. It is subject to change.

  • The -f suite command line option which names a test suite file containing the names of battery files. Need to check handling of params in this case.

  • Need to support multiple body tags inside of a request. Also need to check definition of boundaries and multiple levels of nesting.

  • Need to look into other HTTP request methods to verify that the script can support them and add them to the DTD and input function.

  • Once I figure out what a test-set is, I need to implement it. Need to extend the test-sets so they are more than just a dumb container.

  • Need to add functionality similar to param for include elements. This new child of include would declare replaceable parameters that would be replaced in the body of the included file. These templatized includes could simplify creation and maintenance of similar tests.

  • I need to add tutorial docs for the use of code and codelib, especially the new id feature of codelib.