Florent Georges

XTC - XSLT & XQuery Unit Testing

Introduction

XTC provides a unit testing framework for XSLT. It allows you to write test suites related to styelsheet modules. Basically, you can test any XSLT 2.0 sequence constructor like this:

<t:tests>
<t:title>hello-world()</t:title>

<t:test>
  <t:expect select="'Hello, world!'"/>
  <xsl:sequence select="hw:hello-world()"/>

</t:test>
</t:tests>

That will evaluate all the elements after the t:* as a sequence constructor (here the xsl:sequence), and then compare its result to the expected result (the result of an XPath expression, a piece of XML or an expected error). You can even write your own predicates to compare the result or choose one of the available functions (as the eq operator or the standard deep-equal() function).

When the test suite is run, an XML report is generated, containing all the related information (test performed, result of the evaluation...) This report can then be formatted as HTML.

Comments are very welcome at drkm-lib-xslt AT lists.sf.net (a mailing-list that doesn't require to subscribe in order to post).

Processing Model

Here is the processing model for all steps involved in generating and running the test suite:

The processing model of all steps

The two files above on the image are the two files you wrote. *.xsl is your original XSLT script, to test, and *.xtc is the test suite itself.

The three files below on the image are generated files.

Install and Running

Download the xslt-unit.tar.gz tarball, and unpack it somewhere. This tarball includes all needed stylesheets, including for example all the required FXSL stylesheets. To run the samples, go the directory where you unpacked the archive and:

saxon test/simple.xtc src/test-suite-gen.xsl > test/test-simple.xsl
saxon test/hello-world.xsl test/test-simple.xsl > test/test-simple.xml
saxon test/test-simple.xml src/report-to-html.xsl > test/test-simple.html

saxon test/hello-world.xtc src/test-suite-gen.xsl > test/test-hello-world.xsl
saxon test/hello-world.xsl test/test-hello-world.xsl > test/test-hello-world.xml
saxon test/test-hello-world.xml src/report-to-html.xsl > test/test-hello-world.html

To run the sample with the error support, you have to use Saxon 8.8J B and add misc/error-safe.jar to the classpath (at least for the second command below). Then:

saxon test/errors.xtc src/test-suite-gen.xsl xtc-support-errors=true > test/test-errors.xsl
saxon test/hello-world.xsl test/test-errors.xsl > test/test-errors.xml
saxon test/test-errors.xml src/report-to-html.xsl > test/test-errors.html

You that for each test suite, there are three steps. The first step compiles the test suite into a runnable XSLT stylesheet. The second step run the test suite (the stylesheet), producing an XML report. A message is outputed with the result of each test as it is run. Then the XML report is formated to an HTML page (more human-readable).

Pay attention ex:error-safe support is very experimental.

Support for Gexslt is almost done, but there are still a few bugs. Thanks to Colin for having already fix almost all bugs I found.

Sample

All the samples will use this very simple stylesheet. This will be the stylesheet most tested around the world, ever!

<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

             xmlns:xs="http://www.w3.org/2001/XMLSchema"
             xmlns:hw="http://www.fgeorges.org/hello-world"

             version="2.0">

<xsl:function name="hw:hello-world" as="xs:string">

  <xsl:sequence select="'Hello, world!'"/>
</xsl:function>

<xsl:function name="hw:hello-world" as="xs:string">

  <xsl:param name="who" as="xs:string"/>

  <xsl:sequence select="concat('Hello, ', $who, '!')"/>
</xsl:function>

<xsl:template match="elem">
  <elem1/>
  <elem2 id="{@id}"/>

</xsl:template>

</xsl:transform>

Here is a sample of use:

<t:suite xmlns:t="http://www.fgeorges.org/xslt/unit-test"
       xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

       xmlns:hw="http://www.fgeorges.org/hello-world"
       script="hello-world.xsl">

<t:tests id="hello-world">
  <t:title>hello-world()</t:title>

  <t:test>
    <t:expect select="'Hello, world!'" pred="eq"/>

    <xsl:sequence select="hw:hello-world()"/>
  </t:test>

</t:tests>

</t:suite>

It is quite simple to understand. test:suite is the root element, representing a particular test suite. It contains here three namespace declarations: test is the namespace of XTC, xsl is the namespace of XSLT and hw is the namespace of the functions to test in the above stylesheet.

The suite contains a single set of tests, that itslef contains a single test. The sequence constructor in the test (here the xsl:sequence) will be evaluated, and its result will be compared to the string 'Hello, world!' using the operator eq, as stated in the test:expect element.

Writing a test suite

An introduction

You write a test suite to test a particular stylesheet module. A test suite is an XML document containing several tests, grouped in test sets. A test suite is a t:suite element, containing one or more t:tests elements (the test sets), containing one or more t:test element (the tests themselves).

Here is another sample, a little more complete, showing more feature of XTC. I will describe below how to write a test suite, introducing most of the XTC feature, and referencing the below suite as a sample.

<t:suite xmlns:t="http://www.fgeorges.org/xslt/unit-test"

       xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
       xmlns:xs="http://www.w3.org/2001/XMLSchema"

       xmlns:hw="http://www.fgeorges.org/hello-world"
       xmlns:err="http://www.w3.org/2005/xqt-errors"

       script="hello-world.xsl">

<xsl:variable name="elem" as="element()">

  <elem id="id"/>
</xsl:variable>

<xsl:function name="hw:predicate" as="xs:boolean">

  <xsl:param name="lhp"/>
  <xsl:param name="rhp"/>

  <xsl:sequence select="true()"/>
</xsl:function>

<t:tests id="hello-world-0">
  <t:title>hello-world(), arity 0</t:title>

  <t:test>
    <t:expect select="'Hello, world!'" pred="eq"/>

    <xsl:sequence select="hw:hello-world()"/>
  </t:test>

</t:tests>

<t:tests id="hello-world-1">
  <t:title>hello-world(), arity 1</t:title>

  <t:test>
    <t:expect test="$test:result eq 'Hello, Jeni!'"/>

    <xsl:sequence select="hw:hello-world('Jeni')"/>
  </t:test>

</t:tests>

<t:tests id="hello-world-all">
  <t:title>hello-world()</t:title>

  <t:test>
    <t:title>hello-world(), arity 0</t:title>
    <t:expect test="$test:result eq 'Hello, world!'"/>

    <xsl:sequence select="hw:hello-world()"/>
  </t:test>

  <t:test>
    <t:title>hello-world(), arity 1</t:title>
    <t:expect select="'Hello, Jeni!'" pred="eq"/>

    <xsl:sequence select="hw:hello-world('Jeni')"/>
  </t:test>

</t:tests>

<t:tests id="elem">
  <t:title>'elem' template rule</t:title>

  <t:context select="$elem"/>
  <t:test>

    <t:title>'elem' #1</t:title>
    <t:expect as="element()+">

      <elem1/>
      <elem2 id="id"/>
    </t:expect>

    <xsl:apply-templates select="$elem"/>
  </t:test>

  <t:test>
    <t:title>'elem' #2</t:title>
    <t:expect as="element()+" pred="hw:predicate">

      <elem1/>
      <elem2 id="id"/>
    </t:expect>

    <xsl:apply-templates select="$elem"/>
  </t:test>

</t:tests>

<t:tests id="false">
  <t:title>False negative</t:title>

  <t:context select="$elem"/>
  <t:test>

    <t:expect as="element()+">
      <elem2 id="id"/>

    </t:expect>
    <xsl:apply-templates select="$elem"/>

  </t:test>
</t:tests>

<t:tests id="errors">

  <t:title>Catch errors</t:title>
  <t:test>
    <t:title>Div by 0</t:title>

    <t:expect error="err:FOAR0001"/>
    <xsl:sequence select="1 div 0"/>

  </t:test>
  <t:test>
    <t:title>Wrong error NS</t:title>

    <t:expect error="hw:FOAR0001"/>
    <xsl:sequence select="1 div 0"/>

  </t:test>
  <t:test>
    <t:title>Custom error by error()</t:title>

    <t:expect error="hw:FOAR0001"/>
    <xsl:sequence select="error(xs:QName('hw:FOAR0001'), 'Yo!')"/>

  </t:test>
</t:tests>

</t:suite>

Here are the links to the previous test suite, the tested XSLT module, the generated stylesheet (the compiled suite), the XML report and the report as HTML page:

hello-world.xsl
the original XSLT script, that defines two functions and a template rule
hello-world.xtc
the test suite, testing each function and the template rule, as well as a always-false test and several errors
test-hello-world.xsl
the XSLT script generated from the XTC document, that will effectively run the test suite
test-hello-world.xml
the result of the test suite, an XML document telling wich test passed, the actual value of the results, etcetera
test-hello-world.html
the HTML formated version of the above file (the HTML report)

The t:suite and t:tests elements

The t:suite element is the root element of a test suite. It contains one or more test sets. It can also contain first elements in the XSLT namespace (see below).

<t:suite xmlns:t="http://www.fgeorges.org/xslt/unit-test"

       xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
       xmlns:hw="http://www.fgeorges.org/hello-world"

       script="hello-world.xsl">

<t:tests id="tests-1">

  <t:title>Title of this test set</t:title>
  <t:test>
    ...
  </t:test>

  <t:test>
    ...
  </t:test>
</t:tests>

<t:tests id="tests-2">
  <t:title>Second test set</t:title>

  <t:context select="xpath expression"/>
  <t:test>

    ...
  </t:test>
  <t:test>
    ...
  </t:test>

</t:tests>

</t:suite>

The t:tests element is a test set. It can contain an ID (its id attribute), what is strongly recommended to can retrieve a test set withour ambiguity fro the report, a title (its t:title first child), and several test cases (its t:test childs). After the title, it can also contain a t:context element, with the same structure as xsl:variable. If it does, all the expressions appearing in its test cases will be evaluated in this context (the context must evaluate to a single item, and this item will be the current item).

The t:test element represents a single test case. As t:tests, it can contain an ID, a title and a context. Then it contains an XSLT 2.0 sequence constructor, that is, one or more elements in a namespace other than the XTC namespace:

<t:test id="the-test">

<t:title>Test case title</t:title>
<t:expect .../>
<xsl:sequence select="xpath"/>

<!-- Can contain more instructions -->
...
</t:test>

The t:expect element

This is the most important element in XTC. And the most complex. It can have three different format. It can have either the variable, the test or the error format.

The variable format allow you to use a format very close to the structure of the xsl:variable element. Either with an XPath expression (the select attribute) or an XSLT sequence constructor (with content). You can also use the as attribute, as in XSLT. In either case, the result is compared to the result of the sequence constructor. By default, the standard deep-equal() function is used, but you can specify the function or the operator tu use with the pred attribute (the first parameter or the left-hand operand is the expected sequence and the second paramter or the right-hand operand is the actual result of the test case).

The test format uses instead the test attribute to set an XPath expression to evaluate. If the result of this expression is true, the test passes, if it is false the test fails. The expression can use the variable t:result to access the actual result of the test case.

The error format allows you to test if an error is thrown, and if it matches with the expected error code. The error attribute is a QName representing the error code. The test passes if an error is thrown, with the same name as provided (same URI part and same local part). Warning: This feature requires the error-safe extension, and is very experimental!

Extra XSLT elements

XSLT top-level elements can be added at the beginning of the suite, before the test sets. The keys, variables, parameters, functions or templates they define will be availbale in the rest of the suite. You can for example write a comparison predicate or a key to use in t:expect. Be carefull to not introduce bugs this way. Use this feature only when required, and for very simple things.

The initial context item

TODO: ... How to fix the context item? What guarantees? How to access test data (good practices)? ...

Acknowledgement

All the material on this page is based on ideas and code from Jeni Tennison. Thanks a lot, Jeni!

Thanks to Dimitre Novatchev and the FXSL crew for the FXSL stylesheets library.

Thanks to Colin Adams for his free XSLT 2.0 processor, Gexslt, and all his support regarding the use of Gexslt.

Thanks to Michael Kay for Saxon-B, his open-source XSLT 2.0 Basic implementation. Off course, if you're the lucky owner of Saxon-SA, his commercial XSLT 2.0 Schema-Aware implementation, you can use it instead! (TODO: I still have to write how to validate the test suite input data). Thanks also for all the help and the usefull input while implementing a first draft for ex:error-safe.

Thanks also to Norman Walsh for his RNC schema for XSLT 2.0, and to James Clark for his nXML mode for GNU Emacs.

TODO list