Whole document tree


www.fifi.org
    Documentation
        Manpages
        GNU Info
        Debian document tree
        Whole document tree
    Trigance web page
    Public services
    User info
    Mailing lists
    Secure server
    Multilingual usage
 

Copyright (C) 2000-2012
Philippe Troin
<webmaster@fifi.org>.

Validate HTML
Validate CSS

    

Whole document tree

The XSLT C library for Gnome

The XSLT C library for Gnome

libxslt

Libxslt is the XSLT C library developed for the Gnome project. XSLT itself is a an XML language to define transformation for XML. Libxslt is based on libxml2 the XML C library developed for the Gnome project. It also implements most of the EXSLT set of processor-portable extensions functions and some of Saxon's evaluate and expressions extensions.

People can either embed the library in their application or use xsltproc the command line processing tool. This library is free software and can be reused in commercial applications (see the intro)

External documents:

Introduction

This document describes libxslt, the XSLT C library developed for the Gnome project.

Here are some key points about libxslt:

  • Libxslt is a C implementation
  • Libxslt is based on libxml for XML parsing, tree manipulation and XPath support
  • It is written in plain C, making as few assumptions as possible, and sticking closely to ANSI C/POSIX for easy embedding. Should works on Linux/Unix/Windows.
  • This library is released under the MIT Licence
  • Though not designed primarily with performances in mind, libxslt seems to be a relatively fast processor.

Documentation

There are some on-line resources about using libxslt:

  1. Check the API documentation automatically extracted from code comments (using gtk doc).
  2. Look at the mailing-list archive.
  3. Of course since libxslt is based on libxml, it's a good idea to at least read libxml description

Reporting bugs and getting help

If you need help with the XSLT language itself, here are a number of useful resources:

Well, bugs or missing features are always possible, and I will make a point of fixing them in a timely fashion. The best way to report a bug is to use the Gnome bug tracking database (make sure to use the "libxslt" module name). I look at reports there regularly and it's good to have a reminder when a bug is still open. Be sure to specify that the bug is for the package libxslt.

There is also a mailing-list xslt@gnome.org for libxslt, with an on-line archive. To subscribe to this list, please visit the associated Web page and follow the instructions.

Alternatively, you can just send the bug to the xslt@gnome.org list, if it's really libxslt related I will approve it.. Please do not send me mail directly especially for portability problem, it makes things really harder to track and in some cases I'm not the best person to answer a given question, ask the list instead. Do not send code, I won't debug it (but patches are really appreciated!).

Check the following too before posting:

  • make sure you are using a recent version, and that the problem still shows up in those
  • check the list archives to see if the problem was reported already, in this case there is probably a fix available, similarly check the registered open bugs
  • make sure you can reproduce the bug with xsltproc, a very useful thing to do is run the transformation with -v argument and redirect the standard error to a file, then search in this file for the transformation logs just preceding the possible problem
  • Please send the command showing the error as well as the input and stylesheet (as an attachment)

Of course, bugs reports with a suggested patch for fixing them will probably be processed faster.

If you're looking for help, a quick look at the list archive may actually provide the answer, I usually send source samples when answering libxslt usage questions. The auto-generated documentation is not as polished as I would like (I need to learn more about Docbook), but it's a good starting point.

How to help

You can help the project in various ways, the best thing to do first is to subscribe to the mailing-list as explained before, check the archives and the Gnome bug database::

  1. provide patches when you find problems
  2. provide the diffs when you port libxslt to a new platform. They may not be integrated in all cases but help pinpointing portability problems and
  3. provide documentation fixes (either as patches to the code comments or as HTML diffs).
  4. provide new documentations pieces (translations, examples, etc ...)
  5. Check the TODO file and try to close one of the items
  6. take one of the points raised in the archive or the bug database and provide a fix. Get in touch with me before to avoid synchronization problems and check that the suggested fix will fit in nicely :-)

Downloads

The latest versions of libxslt can be found on xmlsoft.org (Seattle, France) or on the Gnome FTP server either as a source archive or RPM packages. (NOTE that you need the libxml2, libxml2-devel, libxslt and libxslt-devel packages installed to compile applications using libxslt.) Igor Zlatkovic is now the maintainer of the Windows port, he provides binaries. Gary Pennington provides Solaris binaries.

Contribs:

I do accept external contributions, especially if compiling on another platform, get in touch with me to upload the package. I will keep them in the contrib directory

Libxslt is also available from CVS:

FAQ

  1. passing parameters on the xsltproc command line doesn't work

    xsltproc --param test alpha foo.xsl foo.xml

    the param does not get passed and ends up as ""

    In a nutshell do a double escaping at the shell prompt:

    xsltproc --param test "'alpha'" foo.xsl foo.xml

    i.e. the string value is surrounded by " and ' then terminated by ' and ". Libxslt interpret the parameter values as XPath expressions, so the string ->alpha<- is intepreted as the node set matching this string. You really want ->'alpha'<- to be passed to the processor. And to allow this you need to escape the quotes at the shell level using ->"'alpha'"<- .

    or use

    xsltproc --stringparam test alpha foo.xsl foo.xml

News

CVS only : check the Changelog file for a really accurate description

1.0.16: April 15 2002

  • Bug fixes: strip-space, URL in HTML output, error when xsltproc can't save
  • portability fixes: OSF/1, IEEE on alphas, Windows, Python bindings

1.0.15: Mar 25 2002

  • Bugfixes: XPath, python Makefile, recursive attribute sets, @foo[..] templates
  • Debug of memory alocation with valgind
  • serious profiling leading to significant improvement for DocBook processing
  • revamp of the Windows build

1.0.14: Mar 18 2002

  • Improvement in the XPath engine (libxml2-2.4.18)
  • Nasty bug fix related to exslt:node-set
  • Fixed the python Makefiles, cleanup of doc comments, Windows portability fixes

1.0.13: Mar 8 2002

  • a number of bug fixes including "namespace node have no parents"
  • Improvement of the Python bindings
  • Charles Bozeman provided fixes and regression tests for exslt date functions.

1.0.12: Feb 11 2002

  • Fixed the makefiles especially the python module ones
  • half a dozen bugs fixes including 2 old ones

1.0.11: Feb 8 2002

  • Change of Licence to the MIT Licence
  • Added a beta version of the Python bindings, including support to extend the engine with functions written in Python
  • A number of bug fixes
  • Charlie Bozeman provided more EXSLT functions
  • Portability fixes

1.0.10: Jan 14 2002

  • Windows fixes for Win32 from Igor
  • Fixed the Solaris compilation trouble (Albert)
  • Documentation changes and updates: John Fleck
  • Added a stringparam option to avoid escaping hell at the shell level
  • A few bug fixes

1.0.9: Dec 7 2001

  • Makefile patches from Peter Williams
  • attempt to fix the compilation problem associated to prelinking
  • obsoleted libxsltbreakpoint now deprecated and frozen to 1.0.8 API
  • xsltproc return codes are now significant, John Fleck updated the documentation
  • patch to allow as much as 40 steps in patterns (Marc Tardif), should be made dynamic really
  • fixed a bug raised by Nik Clayton when using doctypes with HTML output
  • patches from Keith Isdale to interface with xsltdebugger

1.0.8: Nov 26 2001

  • fixed an annoying header problem, removed a few bugs and some code cleanup
  • patches for Windows and update of Windows Makefiles by Igor
  • OpenVMS port instructions from John A Fotheringham
  • fixed some Makefiles annoyance and libraries prelinking informations

1.0.7: Nov 10 2001

  • remove a compilation problem with LIBXSLT_PUBLIC
  • Finishing the integration steps for Keith Isdale debugger
  • fixes the handling of indent="no" on HTML output
  • fixes on the configure script and RPM spec file

1.0.6: Oct 30 2001

  • bug fixes on number formatting (Thomas), date/time functions (Bruce Miller)
  • update of the Windows Makefiles (Igor)
  • fixed DOCTYPE generation rules for HTML output (me)

1.0.5: Oct 10 2001

  • some portability fixes, including Windows makefile updates from Igor
  • fixed a dozen bugs on XSLT and EXSLT (me and Thomas Broyer)
  • support for Saxon's evaluate and expressions extensions added (initial contribution from Darren Graves)
  • better handling of XPath evaluation errors

1.0.4: Sep 12 2001

  • Documentation updates from John fleck
  • bug fixes (DocBook FO generation should be fixed) and portability improvements
  • Thomas Broyer improved the existing EXSLT support and added String, Time and Date core functions support

1.0.3: Aug 23 2001

  • XML Catalog support see the doc
  • New NaN/Infinity floating point code
  • A few bug fixes

1.0.2: Aug 15 2001

  • lot of bug fixes, increased the testsuite
  • a large chunk of EXSLT is implemented
  • improvements on the extension framework
  • documentation improvements
  • Windows MSC projects files should be up-to-date
  • handle attributes inherited from the DTD by default

1.0.1: July 24 2001

  • initial EXSLT framework
  • better error reporting
  • fixed the profiler on Windows
  • bug fixes

1.0.0: July 10 2001

  • a lot of cleanup, a lot of regression tests added or fixed
  • added a documentation for writing extensions
  • fixed some variable evaluation problems (with William)
  • added profiling of stylesheet execution accessible as the xsltproc --profile option
  • fixed element-available() and the implementation of the various chunking methods present, Norm Walsh provided a lot of feedback
  • exclude-result-prefixes and namespaces output should now work as expected
  • added support of embedded stylesheet as described in section 2.7 of the spec

0.14.0: July 5 2001

  • lot of bug fixes, and code cleanup
  • completion of the little XSLT-1.0 features left unimplemented
  • Added and implemented the extension API suggested by Thomas Broyer
  • the Windows MSC environment should be complete
  • tested and optimized with a really large document (DocBook Definitive Guide) libxml/libxslt should really be faster on serious workloads

0.13.0: June 26 2001

  • lots of cleanups
  • fixed a C++ compilation problem
  • couple of fixes to xsltSaveTo()
  • try to fix Docbook-xslt-1.4 and chunking, updated the regression test with them
  • fixed pattern compilation and priorities problems
  • Patches for Windows and MSC project mostly contributed by Yon Derek
  • update to the Tutorial by John Fleck
  • William fixed bugs in templates and for-each functions
  • added a new interface xsltRunStylesheet() for a more flexible output (incomplete), added -o option to xsltproc

0.12.0: June 18 2001

  • fixed a dozen of bugs reported
  • HTML generation should be quite better (requires libxml-2.3.11 upgrade too)
  • William fixed some problems with document()
  • Fix namespace nodes selection and copy (requires libxml-2.3.11 upgrade too)
  • John Fleck added a tutorial
  • Fixes for namespace handling when evaluating variables
  • XInclude global flag added to process XInclude on document() if requested
  • made xsltproc --version more detailed

0.11.0: June 1 2001

Mostly a bug fix release.

  • integration of catalogs from xsltproc
  • added --version to xsltproc for bug reporting
  • fixed errors when handling ID in external parsed entities
  • document() should hopefully work correctly but ...
  • fixed bug with PI and comments processing
  • William fixed the XPath string functions when using unicode

0.10.0: May 19 2001

  • cleanups to make stylesheet read-only (not 100% complete)
  • fixed URI resolution in document()
  • force all XPath expression to be compiled at stylesheet parsing time, even if unused ...
  • Fixed HTML default output detection
  • Fixed double attribute generation #54446
  • Fixed {{ handling in attributes #54451
  • More tests and speedups for DocBook document transformations
  • Fixed a really bad race like bug in xsltCopyTreeList()
  • added a documentation on the libxslt internals
  • William Brack and Bjorn Reese improved format-number()
  • Fixed multiple sort, it should really work now
  • added a --docbook option for SGML DocBook input (hackish)
  • a number of other bug fixes and regression test added as people were submitting them

0.9.0: May 3 2001

  • lot of various bugfixes, extended the regression suite
  • xsltproc should work with multiple params
  • added an option to use xsltproc with HTML input
  • improved the stylesheet compilation, processing of complex stylesheets should be faster
  • using the same stylesheet for concurrent processing on multithreaded programs should work now
  • fixed another batch of namespace handling problems
  • Implemented multiple level of sorting

0.8.0: Apr 22 2001

  • fixed ansidecl.h problem
  • fixed unparsed-entity-uri() and generate-id()
  • sort semantic fixes and priority prob from William M. Brack
  • fixed namespace handling problems in XPath expression computations (requires libxml-2.3.7)
  • fixes to current() and key()
  • other, smaller fixes, lots of testing with N Walsh DocBook HTML stylesheets

0.7.0: Apr 10 2001

  • cleanup using stricter compiler flags
  • command line parameter passing
  • fix to xsltApplyTemplates from William M. Brack
  • added the XSLTMark in the regression tests as well as document()

0.6.0: Mar 22 2001

  • another beta
  • requires 2.3.5, which provide XPath expression compilation support
  • document() extension should function properly
  • fixed a number or reported bugs

0.5.0: Mar 10 2001

  • fifth beta
  • some optimization work, for the moment 2 XSLT transform cannot use the same stylesheet at the same time (to be fixed)
  • fixed problems with handling of tree results
  • fixed a reported strip-spaces problem
  • added more reported/fixed bugs to the test suite
  • incorporated William M. Brack fix for imports and global variables as well as patch for with-param support in apply-templates
  • a bug fix on for-each

0.4.0: Mar 1 2001

  • fourth beta test, released at the same time of libxml2-2.3.3
  • bug fixes
  • some optimization
  • started implement extension support, not finished
  • implemented but not tested multiple file output

0.3.0: Feb 24 2001

  • third beta test, released at the same time of libxml2-2.3.2
  • lot of bug fixes
  • some optimization
  • added DocBook XSL based testsuite

0.2.0: Feb 15 2001

  • second beta version, released at the same time as libxml2-2.3.1
  • getting close to feature completion, lot of bug fixes, some in the HTML and XPath support of libxml
  • start becoming usable for real work. This version can now regenerate the XML 2e HTML from the original XML sources and the associated stylesheets (in section I of the XML REC)
  • Still misses extension element/function/prefixes support. Support of key() and document() is not complete

0.1.0: Feb 8 2001

  • first beta version, released at the same time as libxml2-2.3.0
  • lots of bug fixes, first "testing" version, but incomplete

0.0.1: Jan 25 2001

  • first alpha version released at the same time as libxml2-2.2.12
  • Framework in place, should work on simple examples, but far from being feature complete

The xsltproc tool

This program is the simplest way to use libxslt: from the command line. It is also used for doing the regression tests of the library.

It takes as first argument the path or URL to an XSLT stylesheet, the next arguments are filenames or URIs of the inputs to be processed. The output of the processing is redirected on the standard output. There is actually a few more options available:

orchis:~ -> xsltproc
Usage: xsltproc [options] stylesheet file [file ...]
   Options:
      --version or -V: show the version of libxml and libxslt used
      --verbose or -v: show logs of what's happening
      --output file or -o file: save to a given file
      --timing: display the time used
      --repeat: run the transformation 20 times
      --debug: dump the tree of the result instead
      --novalid: skip the Dtd loading phase
      --noout: do not dump the result
      --maxdepth val : increase the maximum depth
      --html: the input document is(are) an HTML file(s)
      --docbook: the input document is SGML docbook
      --param name value : pass a (parameter,value) pair
      --nonet refuse to fetch DTDs or entities over network
      --warnnet warn against fetching over the network
      --catalogs : use the catalogs from $SGML_CATALOG_FILES
      --xinclude : do XInclude processing on document intput
      --profile or --norman : dump profiling informations 
orchis:~ ->

DocBook

DocBook is an XML/SGML vocabulary particularly well suited to books and papers about computer hardware and software.

xsltproc and libxslt are not specifically dependant on DocBook, but since a lot of people use xsltproc and libxml2 for DocBook formatting, here are a few pointers and informations which may be helpful:

  • The DocBook homepage at Oasis you should find pointers there on all the lastest versions of the DTDs and XSLT stylesheets
  • DocBook: The Definitive Guide is the official reference documentation for DocBook.
  • Here is a shell script to generate XML Catalogs for DocBook 4.1.2 . If it can write to the /etc/xml/ directory, it will set-up /etc/xml/catalog and /etc/xml/docbook based on the resources found on the system. Otherwise it will just create ~/xmlcatalog and ~/dbkxmlcatalog and doing:

    export XMLCATALOG=$HOME/xmlcatalog

    should allow to process DocBook documentations without requiring network accesses for the DTd or stylesheets

  • I have uploaded a small tarball containing XML Catalogs for DocBook 4.1.2 which seems to work fine for me too
  • Informations on installing a Windows DocBook processing setup based on Cygwin (using the binaries from the official Windows port should be possible too)
  • Alexander Kirillov's page on Using DocBook XML 4.1.2 (RPM packages)
  • Tim Waugh's xmlto front-end conversion script
  • Linux Documentation Project DocBook-Install-mini-HOWTO
  • ScrollKeeper the open documentation cataloging project has a DocBook section
  • Dan York presentation on Publishing using DocBook XML

Do not use the --docbook option of xsltproc to process XML DocBook documents, this option is only intended to provide some (limited) support of the SGML version of DocBook.

Points which are not DocBook specific but still worth mentionning again:

  • if you think DocBook processing time is too slow, make sure you have XML Catalogs pointing to a local installation of the DTD of DocBook. Check the XML Catalog page to understand more on this subject.
  • before processing a new document, use the command

    xmllint --valid --noout path_to_document

    to make sure that your input is valid DocBook. And fixes the errors before processing further. Note that XSLT processing may work correctly with some forms of validity errors left, but in general it can give troubles on output.

The programming API

Okay this section is clearly incomplete. But integrating libxslt into your application should be relatively easy. First check the few steps described below, then for more detailed informations, look at the generated pages for the API and the source of libxslt/xsltproc.c and the tutorial.

Basically doing an XSLT transformation can be done in a few steps:

  1. configure the parser for XSLT:

    xmlSubstituteEntitiesDefault(1);

    xmlLoadExtDtdDefaultValue = 1;

  2. parse the stylesheet with xsltParseStylesheetFile()
  3. parse the document with xmlParseFile()
  4. apply the stylesheet using xsltApplyStylesheet()
  5. save the result using xsltSaveResultToFile() if needed set xmlIndentTreeOutput to 1

Steps 2,3, and 5 will probably need to be changed depending on you processing needs and environment for example if reading/saving from/to memory, or if you want to apply XInclude processing to the stylesheet or input documents.

Python and bindings

There is a number of language bindings and wrappers available for libxml2, the list below is not exhaustive. Please contact the xml-bindings@gnome.org (archives) in order to get updates to this list or to discuss the specific topic of libxml2 or libxslt wrappers or bindings:

The libxslt Python module depends on the libxml2 Python module.

The distribution includes a set of Python bindings, which are garanteed to be maintained as part of the library in the future, though the Python interface have not yet reached the maturity of the C API.

To install the Python bindings there are 2 options:

  • If you use an RPM based distribution, simply install the libxml2-python RPM and the libxslt-python RPM.
  • Otherwise use the libxml2-python module distribution corresponding to your installed version of libxml2 and libxslt. Note that to install it you will need both libxml2 and libxslt installed and run "python setup.py build install" in the module tree.

The distribution includes a set of examples and regression tests for the python bindings in the python/tests directory. Here are some excepts from those tests:

basic.py:

This is a basic test of XSLT interfaces: loading a stylesheet and a document, transforming the document and saving the result.

import libxml2
import libxslt

styledoc = libxml2.parseFile("test.xsl")
style = libxslt.parseStylesheetDoc(styledoc)
doc = libxml2.parseFile("test.xml")
result = style.applyStylesheet(doc, None)
style.saveResultToFilename("foo", result, 0)
style.freeStylesheet()
doc.freeDoc()
result.freeDoc()

The Python module is called libxslt, you will also need the libxml2 module for the operations on XML trees. Let's have a look at the objects manipulated in that example and how is the processing done:

  • styledoc : is a libxml2 document tree. It is obtained by parsing the XML file "test.xsl" containing the stylesheet.
  • style : this is a precompiled stylesheet ready to be used by the following transformations (note the plural form, multiple transformations can resuse the same stylesheet).
  • doc : this is the document to apply the transformation to. In this case it is simply generated by parsing it from a file but any other processing is possible as long as one get a libxml2 Doc. Note that HTML tree are suitable for XSLT processing in libxslt. This is actually how this page is generated !
  • result : this is a document generated by applying the stylesheet to the document. Note that some of the stylesheet informations may be related to the serialization of that document and as in this example a specific saveResultToFilename() method of the stylesheet should be used to save it to a file (in that case to "foo").

Also note the need to explicitely deallocate documents with freeDoc() except for the stylesheet document which is freed when its compiled form is garbage collected.

extfunc.py:

This one is a far more complex test. It shows how to modify the behaviour of an XSLT transformation by passing parameters and how to extend the XSLT engine with functions defined in python:

import libxml2
import libxslt
import string

nodeName = None
def f(ctx, str):
    global nodeName

    #
    # Small check to verify the context is correcly accessed
    #
    try:
        pctxt = libxslt.xpathParserContext(_obj=ctx)
        ctxt = pctxt.context()
        tctxt = ctxt.transformContext()
        nodeName = tctxt.insertNode().name
    except:
        pass

    return string.upper(str)

libxslt.registerExtModuleFunction("foo", "http://example.com/foo", f)

This code defines and register an extension function. Note that the function can be bound to any name (foo) and how the binding is also associated to a namespace name "http://example.com/foo". From an XSLT point of view the function just returns an upper case version of the string passed as a parameter. But the first part of the function also read some contextual information from the current XSLT processing environement, in that case it looks for the current insertion node in the resulting output (either the resulting document or the Result Value Tree being generated), and saves it to a global variable for checking that the access actually worked.

For more informations on the xpathParserContext and transformContext objects check the libray internals description. The pctxt is actually an object from a class derived from the libxml2.xpathParserContext() with just a couple more properties including the possibility to look up the XSLT transformation context from the XPath context.

styledoc = libxml2.parseDoc("""
<xsl:stylesheet version='1.0'
  xmlns:xsl='http://www.w3.org/1999/XSL/Transform'
  xmlns:foo='http://example.com/foo'
  xsl:exclude-result-prefixes='foo'>

  <xsl:param name='bar'>failure</xsl:param>
  <xsl:template match='/'>
    <article><xsl:value-of select='foo:foo($bar)'/></article>
  </xsl:template>
</xsl:stylesheet>
""")

Here is a simple example of how to read an XML document from a python string with libxml2. Note how this stylesheet:

  • Uses a global parameter bar
  • Reference the extension function f
  • how the Namespace name "http://example.com/foo" has to be bound to a prefix
  • how that prefix is excluded from the output
  • how the function is called from the select
style = libxslt.parseStylesheetDoc(styledoc)
doc = libxml2.parseDoc("<doc/>")
result = style.applyStylesheet(doc, { "bar": "'success'" })
style.freeStylesheet()
doc.freeDoc()

that part is identical, to the basic example except that the transformation is passed a dictionnary of parameters. Note that the string passed "success" had to be quoted, otherwise it is interpreted as an XPath query for the childs of root named "success".

root = result.children
if root.name != "article":
    print "Unexpected root node name"
    sys.exit(1)
if root.content != "SUCCESS":
    print "Unexpected root node content, extension function failed"
    sys.exit(1)
if nodeName != 'article':
    print "The function callback failed to access its context"
    sys.exit(1)

result.freeDoc()

That part just verifies that the transformation worked, that the parameter got properly passed to the engine, that the function f() got called and that it properly accessed the context to find the name of the insertion node.

pyxsltproc.py:

this module is a bit too long to be described there but it is basically a rewrite of the xsltproc command line interface of libxslt in Python. It provides nearly all the functionalities of xsltproc and can be used as a base module to write Python customized XSLT processors. One of the thing to notice are:

libxml2.lineNumbersDefault(1)
libxml2.substituteEntitiesDefault(1)

those two calls in the main() function are needed to force the libxml2 processor to generate DOM trees compliant with the XPath data model.

Library internals

Table of contents

Introduction

This document describes the processing of libxslt, the XSLT C library developed for the Gnome project.

Note: this documentation is by definition incomplete and I am not good at spelling, grammar, so patches and suggestions are really welcome.

Basics

XSLT is a transformation language. It takes an input document and a stylesheet document and generates an output document:

the XSLT processing model

Libxslt is written in C. It relies on libxml, the XML C library for Gnome, for the following operations:

  • parsing files
  • building the in-memory DOM structure associated with the documents handled
  • the XPath implementation
  • serializing back the result document to XML and HTML. (Text is handled directly.)

Keep it simple stupid

Libxslt is not very specialized. It is built under the assumption that all nodes from the source and output document can fit in the virtual memory of the system. There is a big trade-off there. It is fine for reasonably sized documents but may not be suitable for large sets of data. The gain is that it can be used in a relatively versatile way. The input or output may never be serialized, but the size of documents it can handle are limited by the size of the memory available.

More specialized memory handling approaches are possible, like building the input tree from a serialization progressively as it is consumed, factoring repetitive patterns, or even on-the-fly generation of the output as the input is parsed but it is possible only for a limited subset of the stylesheets. In general the implementation of libxslt follows the following pattern:

  • KISS (keep it simple stupid)
  • when there is a clear bottleneck optimize on top of this simple framework and refine only as much as is needed to reach the expected result

The result is not that bad, clearly one can do a better job but more specialized too. Most optimization like building the tree on-demand would need serious changes to the libxml XPath framework. An easy step would be to serialize the output directly (or call a set of SAX-like output handler to keep this a flexible interface) and hence avoid the memory consumption of the result.

The libxml nodes

DOM-like trees, as used and generated by libxml and libxslt, are relatively complex. Most node types follow the given structure except a few variations depending on the node type:

description of a libxml node

Nodes carry a name and the node type indicates the kind of node it represents, the most common ones are:

  • document nodes
  • element nodes
  • text nodes

For the XSLT processing, entity nodes should not be generated (i.e. they should be replaced by their content). Most nodes also contains the following "navigation" informations:

  • the containing document
  • the parent node
  • the first children node
  • the last children node
  • the previous sibling
  • the following sibling (next)

Elements nodes carries the list of attributes in the properties, an attribute itself holds the navigation pointers and the children list (the attribute value is not represented as a simple string to allow usage of entities references).

The ns points to the namespace declaration for the namespace associated to the node, nsDef is the linked list of namespace declaration present on element nodes.

Most nodes also carry an _private pointer which can be used by the application to hold specific data on this node.

The XSLT processing steps

There are a few steps which are clearly decoupled at the interface level:

  1. parse the stylesheet and generate a DOM tree
  2. take the stylesheet tree and build a compiled version of it (the compilation phase)
  3. take the input and generate a DOM tree
  4. process the stylesheet against the input tree and generate an output tree
  5. serialize the output tree

A few things should be noted here:

  • the steps 1/ 3/ and 5/ are optional
  • the stylesheet obtained at 2/ can be reused by multiple processing 4/ (and this should also work in threaded programs)
  • the tree provided in 2/ should never be freed using xmlFreeDoc, but by freeing the stylesheet.
  • the input tree 4/ is not modified except the _private field which may be used for labelling keys if used by the stylesheet

The XSLT stylesheet compilation

This is the second step described. It takes a stylesheet tree, and "compiles" it. This associates to each node a structure stored in the _private field and containing information computed in the stylesheet:

a compiled XSLT stylesheet

One xsltStylesheet structure is generated per document parsed for the stylesheet. XSLT documents allow includes and imports of other documents, imports are stored in the imports list (hence keeping the tree hierarchy of includes which is very important for a proper XSLT processing model) and includes are stored in the doclist list. An imported stylesheet has a parent link to allow browsing of the tree.

The DOM tree associated to the document is stored in doc. It is preprocessed to remove ignorable empty nodes and all the nodes in the XSLT namespace are subject to precomputing. This usually consist of extracting all the context information from the context tree (attributes, namespaces, XPath expressions), and storing them in an xsltStylePreComp structure associated to the _private field of the node.

A couple of notable exceptions to this are XSLT template nodes (more on this later) and attribute value templates. If they are actually templates, the value cannot be computed at compilation time. (Some preprocessing could be done like isolation and preparsing of the XPath subexpressions but it's not done, yet.)

The xsltStylePreComp structure also allows storing of the precompiled form of an XPath expression that can be associated to an XSLT element (more on this later).

The XSLT template compilation

A proper handling of templates lookup is one of the keys of fast XSLT processing. (Given a node in the source document this is the process of finding which templates should be applied to this node.) Libxslt follows the hint suggested in the 5.2 Patterns section of the XSLT Recommendation, i.e. it doesn't evaluate it as an XPath expression but tokenizes it and compiles it as a set of rules to be evaluated on a candidate node. There usually is an indication of the node name in the last step of this evaluation and this is used as a key check for the match. As a result libxslt builds a relatively more complex set of structures for the templates:

The templates related structure

Let's describe a bit more closely what is built. First the xsltStylesheet structure holds a pointer to the template hash table. All the XSLT patterns compiled in this stylesheet are indexed by the value of the the target element (or attribute, pi ...) name, so when a element or an attribute "foo" needs to be processed the lookup is done using the name as a key.

Each of the patterns is compiled into an xsltCompMatch structure. It holds the set of rules based on the tokenization of the pattern stored in reverse order (matching is easier this way). It also holds some information about the previous matches used to speed up the process when one iterates over a set of siblings. (This optimization may be defeated by trashing when running threaded computation, it's unclear that this is a big deal in practice.) Predicate expressions are not compiled at this stage, they may be at run-time if needed, but in this case they are compiled as full XPath expressions (the use of some fixed predicate can probably be optimized, they are not yet).

The xsltCompMatch are then stored in the hash table, the clash list is itself sorted by priority of the template to implement "naturally" the XSLT priority rules.

Associated to the compiled pattern is the xsltTemplate itself containing the information required for the processing of the pattern including, of course, a pointer to the list of elements used for building the pattern result.

Last but not least a number of patterns do not fit in the hash table because they are not associated to a name, this is the case for patterns applying to the root, any element, any attributes, text nodes, pi nodes, keys etc. Those are stored independently in the stylesheet structure as separate linked lists of xsltCompMatch.

The processing itself

The processing is defined by the XSLT specification (the basis of the algorithm is explained in the Introduction section). Basically it works by taking the root of the input document and applying the following algorithm:

  1. Finding the template applying to it. This is a lookup in the template hash table, walking the hash list until the node satisfies all the steps of the pattern, then checking the appropriate(s) global templates to see if there isn't a higher priority rule to apply
  2. If there is no template, apply the default rule (recurse on the children)
  3. else walk the content list of the selected templates, for each of them:
    • if the node is in the XSLT namespace then the node has a _private field pointing to the preprocessed values, jump to the specific code
    • if the node is in an extension namespace, look up the associated behavior
    • otherwise copy the node.

    The closure is usually done through the XSLT apply-templates construct recursing by applying the adequate template on the input node children or on the result of an associated XPath selection lookup.

Note that large parts of the input tree may not be processed by a given stylesheet and that on the opposite some may be processed multiple times. (This often is the case when a Table of Contents is built).

The module transform.c is the one implementing most of this logic. xsltApplyStylesheet() is the entry point, it allocates an xsltTransformContext containing the following:

  • a pointer to the stylesheet being processed
  • a stack of templates
  • a stack of variables and parameters
  • an XPath context
  • the template mode
  • current document
  • current input node
  • current selected node list
  • the current insertion points in the output document
  • a couple of hash tables for extension elements and functions

Then a new document gets allocated (HTML or XML depending on the type of output), the user parameters and global variables and parameters are evaluated. Then xsltProcessOneNode() which implements the 1-2-3 algorithm is called on the root element of the input. Step 1/ is implemented by calling xsltGetTemplate(), step 2/ is implemented by xsltDefaultProcessOneNode() and step 3/ is implemented by xsltApplyOneTemplate().

XPath expression compilation

The XPath support is actually implemented in the libxml module (where it is reused by the XPointer implementation). XPath is a relatively classic expression language. The only uncommon feature is that it is working on XML trees and hence has specific syntax and types to handle them.

XPath expressions are compiled using xmlXPathCompile(). It will take an expression string in input and generate a structure containing the parsed expression tree, for example the expression:

/doc/chapter[title='Introduction']

will be compiled as

Compiled Expression : 10 elements
  SORT
    COLLECT  'child' 'name' 'node' chapter
      COLLECT  'child' 'name' 'node' doc
        ROOT
      PREDICATE
        SORT
          EQUAL =
            COLLECT  'child' 'name' 'node' title
              NODE
            ELEM Object is a string : Introduction
              COLLECT  'child' 'name' 'node' title
                NODE

This can be tested using the testXPath command (in the libxml codebase) using the --tree option.

Again, the KISS approach is used. No optimization is done. This could be an interesting thing to add. Michael Kay describes a lot of possible and interesting optimizations done in Saxon which would be possible at this level. I'm unsure they would provide much gain since the expressions tends to be relatively simple in general and stylesheets are still hand generated. Optimizations at the interpretation sounds likely to be more efficient.

XPath interpretation

The interpreter is implemented by xmlXPathCompiledEval() which is the front-end to xmlXPathCompOpEval() the function implementing the evaluation of the expression tree. This evaluation follows the KISS approach again. It's recursive and calls xmlXPathNodeCollectAndTest() to collect nodes set when evaluating a COLLECT node.

An evaluation is done within the framework of an XPath context stored in an xmlXPathContext structure, in the framework of a transformation the context is maintained within the XSLT context. Its content follows the requirements from the XPath specification:

  • the current document
  • the current node
  • a hash table of defined variables (but not used by XSLT)
  • a hash table of defined functions
  • the proximity position (the place of the node in the current node list)
  • the context size (the size of the current node list)
  • the array of namespace declarations in scope (there also is a namespace hash table but it is not used in the XSLT transformation).

For the purpose of XSLT an extra pointer has been added allowing to retrieve the XSLT transformation context. When an XPath evaluation is about to be performed, an XPath parser context is allocated containing and XPath object stack (this is actually an XPath evaluation context, this is a remain of the time where there was no separate parsing and evaluation phase in the XPath implementation). Here is an overview of the set of contexts associated to an XPath evaluation within an XSLT transformation:

The set of contexts associated

Clearly this is a bit too complex and confusing and should be refactored at the next set of binary incompatible releases of libxml. For example the xmlXPathCtxt has a lot of unused parts and should probably be merged with xmlXPathParserCtxt.

Description of XPath Objects

An XPath expression manipulates XPath objects. XPath defines the default types boolean, numbers, strings and node sets. XSLT adds the result tree fragment type which is basically an unmodifiable node set.

Implementation-wise, libxml follows again a KISS approach, the xmlXPathObject is a structure containing a type description and the various possibilities. (Using an enum could have gained some bytes.) In the case of node sets (or result tree fragments), it points to a separate xmlNodeSet object which contains the list of pointers to the document nodes:

An Node set object pointing to

The XPath API (and its 'internal' part) includes a number of functions to create, copy, compare, convert or free XPath objects.

XPath functions

All the XPath functions available to the interpreter are registered in the function hash table linked from the XPath context. They all share the same signature:

void xmlXPathFunc (xmlXPathParserContextPtr ctxt, int nargs);

The first argument is the XPath interpretation context, holding the interpretation stack. The second argument defines the number of objects passed on the stack for the function to consume (last argument is on top of the stack).

Basically an XPath function does the following:

  • check nargs for proper handling of errors or functions with variable numbers of parameters
  • pop the parameters from the stack using obj = valuePop(ctxt);
  • do the function specific computation
  • push the result parameter on the stack using valuePush(ctxt, res);
  • free up the input parameters with xmlXPathFreeObject(obj);
  • return

Sometime the work can be done directly by modifying in-situ the top object on the stack ctxt->value.

The XSLT variables stack frame

Not to be confused with XPath object stack, this stack holds the XSLT variables and parameters as they are defined through the recursive calls of call-template, apply-templates and default templates. This is used to define the scope of variables being called.

This part seems to be the most urgent attention right now, first it is done in a very inefficient way since the location of the variables and parameters within the stylesheet tree is still done at run time (it really should be done statically at compile time), and I am still unsure that my understanding of the template variables and parameter scope is actually right.

This part of the documentation is still to be written once this part of the code will be stable. TODO

Extension support

There is a separate document explaining how the extension support works.

Further reading

Michael Kay wrote a really interesting article on Saxon internals and the work he did on performance issues. I wishes I had read it before starting libxslt design (I would probably have avoided a few mistakes and progressed faster). A lot of the ideas in his papers should be implemented or at least tried in libxslt.

The libxml documentation, especially the I/O interfaces and the memory management.

TODOs

redesign the XSLT stack frame handling. Far too much work is done at execution time. Similarly for the attribute value templates handling, at least the embedded subexpressions ought to be precompiled.

Allow output to be saved to a SAX like output (this notion of SAX like API for output should be added directly to libxml).

Implement and test some of the optimization explained by Michael Kay especially:

  • static slot allocation on the stack frame
  • specific boolean interpretation of an XPath expression
  • some of the sorting optimization
  • Lazy evaluation of location path. (this may require more changes but sounds really interesting. XT does this too.)
  • Optimization of an expression tree (This could be done as a completely independent module.)

Error reporting, there is a lot of case where the XSLT specification specify that a given construct is an error are not checked adequately by libxslt. Basically one should do a complete pass on the XSLT spec again and add all tests to the stylesheet compilation. Using the DTD provided in the appendix and making direct checks using the libxml validation API sounds a good idea too (though one should take care of not raising errors for elements/attributes in different namespaces).

Double check all the places where the stylesheet compiled form might be modified at run time (extra removal of blanks nodes, hint on the xsltCompMatch).

Writing extensions

Table of content

Introduction

This document describes the work needed to write extensions to the standard XSLT library for use with libxslt, the XSLT C library developed for the Gnome project.

Before starting reading this document it is highly recommended to get familiar with the libxslt internals.

Note: this documentation is by definition incomplete and I am not good at spelling, grammar, so patches and suggestions are really welcome.

Basics

The XSLT specification provides two ways to extend an XSLT engine:

In both cases the extensions need to be associated to a new namespace, i.e. an URI used as the name for the extension's namespace (there is no need to have a resource there for this to work).

libxslt provides a few extensions itself, either in libxslt namespace "http://xmlsoft.org/XSLT/" or in other namespace for well known extensions provided by other XSLT processors like Saxon, Xalan or XT.

Extension modules

Since extensions are bound to a namespace name, usually sets of extensions coming from a given source are using the same namespace name defining in practice a group of extensions providing elements, functions or both. From libxslt point of view those are considered as an "extension module", and most of the APIs work at a module point of view.

Registration of new functions or elements are bound to the activation of the module, this is currently done by declaring the namespace as an extension by using the attribute extension-element-prefixes on the xsl:stylesheet element.

And extension module is defined by 3 objects:

  • the namespace name associated
  • an initialization function
  • a shutdown function

Registering a module

Currently a libxslt module has to be compiled within the application using libxslt, there is no code to load dynamically shared libraries associated to namespace (this may be added but is likely to become a portability nightmare).

So the current way to register a module is to link the code implementing it with the application and to call a registration function:

int xsltRegisterExtModule(const xmlChar *URI,
                          xsltExtInitFunction initFunc,
                          xsltExtShutdownFunction shutdownFunc);

The associated header is read by:

#include<libxslt/extensions.h>

which also defines the type for the initialization and shutdown functions

Loading a module

Once the module URI has been registered and if the XSLT processor detects that a given stylesheet needs the functionalities of an extended module, this one is initialized.

The xsltExtInitFunction type defines the interface for an initialization function:

/**
 * xsltExtInitFunction:
 * @ctxt:  an XSLT transformation context
 * @URI:  the namespace URI for the extension
 *
 * A function called at initialization time of an XSLT
 * extension module
 *
 * Returns a pointer to the module specific data for this
 * transformation
 */
typedef void *(*xsltExtInitFunction)(xsltTransformContextPtr ctxt,
                                     const xmlChar *URI);

There are 3 things to notice:

  • the function gets passed the namespace name URI as an argument, this allow a single function to provide the initialization for multiple logical modules
  • it also gets passed a transformation context, the initialization is done at run time before any processing occurs on the stylesheet but it will be invoked separately each time for each transformation
  • it returns a pointer, this can be used to store module specific informations which can be retrieved later when a function or an element from the extension are used, an obvious example is a connection to a database which should be kept and reused along the transformation. NULL is a perfectly valid return, there is no way to indicate a failure at this level

What this function is expected to do is:

  • prepare the context for this module (like opening the database connection)
  • register the extensions specific to this module

Registering an extension function

There is a single call to do this registration:

int xsltRegisterExtFunction(xsltTransformContextPtr ctxt,
                            const xmlChar *name,
                            const xmlChar *URI,
                            xmlXPathEvalFunc function);

The registration is bound to a single transformation instance referred by ctxt, name is the UTF8 encoded name for the NCName of the function, and URI is the namespace name for the extension (no checking is done, a module could register functions or elements from a different namespace, but it is not recommended).

Implementing an extension function

The implementation of the function must have the signature of a libxml XPath function:

/**
 * xmlXPathEvalFunc:
 * @ctxt: an XPath parser context
 * @nargs: the number of arguments passed to the function
 *
 * an XPath evaluation function, the parameters are on the
 * XPath context stack
 */

typedef void (*xmlXPathEvalFunc)(xmlXPathParserContextPtr ctxt,
                                 int nargs);

The context passed to an XPath function is not an XSLT context but an XPath context. However it is possible to find one from the other:

  • The function xsltXPathGetTransformContext provide this lookup facility: 
    xsltTransformContextPtr
         xsltXPathGetTransformContext
                          (xmlXPathParserContextPtr ctxt);
  • The xmlXPathContextPtr associated to an xsltTransformContext is stored in the xpathCtxt field.

The first thing an extension function may want to do is to check the arguments passed on the stack, the nargs will precise how many of them were provided on the XPath expression. The macros valuePop will extract them from the XPath stack:

#include <libxml/xpath.h>
#include <libxml/xpathInternals.h>

xmlXPathObjectPtr obj = valuePop(ctxt);

Note that ctxt is the XPath context not the XSLT one. It is then possible to examine the content of the value. Check the description of XPath objects if necessary. The following is a common sequcnce checking whether the argument passed is a string and converting it using the built-in XPath string() function if this is not the case:

if (obj->type != XPATH_STRING) {
    valuePush(ctxt, obj);
    xmlXPathStringFunction(ctxt, 1);
    obj = valuePop(ctxt);
}

Most common XPath functions are available directly at the C level and are exported either in <libxml/xpath.h> or in <libxml/xpathInternals.h>.

The extension function may also need to retrieve the data associated to this module instance (the database connection in the previous example) this can be done using the xsltGetExtData:

void * xsltGetExtData(xsltTransformContextPtr ctxt,
                      const xmlChar *URI);

again the URI to be provided is the one used which was used when registering the module.

Once the function finishes, don't forget to:

  • push the return value on the stack using valuePush(ctxt, obj)
  • deallocate the parameters passed to the function using xmlXPathFreeObject(obj)

Examples for extension functions

The module libxslt/functions.c containsthe sources of the XSLT built-in functions, including document(), key(), generate-id(), etc. as well as a full example module at the end. Here is the test function implementation for the libxslt:test function:

/**
 * xsltExtFunctionTest:
 * @ctxt:  the XPath Parser context
 * @nargs:  the number of arguments
 *
 * function libxslt:test() for testing the extensions support.
 */
static void
xsltExtFunctionTest(xmlXPathParserContextPtr ctxt, int nargs)
{
    xsltTransformContextPtr tctxt;
    void *data;

    tctxt = xsltXPathGetTransformContext(ctxt);
    if (tctxt == NULL) {
        xsltGenericError(xsltGenericErrorContext,
            "xsltExtFunctionTest: failed to get the transformation context\n");
        return;
    }
    data = xsltGetExtData(tctxt, (const xmlChar *) XSLT_DEFAULT_URL);
    if (data == NULL) {
        xsltGenericError(xsltGenericErrorContext,
            "xsltExtFunctionTest: failed to get module data\n");
        return;
    }
#ifdef WITH_XSLT_DEBUG_FUNCTION
    xsltGenericDebug(xsltGenericDebugContext,
                     "libxslt:test() called with %d args\n", nargs);
#endif
}

Registering an extension element

There is a single call to do this registration:

int xsltRegisterExtElement(xsltTransformContextPtr ctxt,
                           const xmlChar *name,
                           const xmlChar *URI,
                           xsltTransformFunction function);

It is similar to the mechanism used to register an extension function, except that the signature of an extension element implementation is different.

The registration is bound to a single transformation instance referred by ctxt, name is the UTF8 encoded name for the NCName of the element, and URI is the namespace name for the extension (no checking is done, a module could register elements for a different namespace, but it is not recommended).

Implementing an extension element

The implementation of the element must have the signature of an XSLT transformation function:

/** 
 * xsltTransformFunction: 
 * @ctxt: the XSLT transformation context
 * @node: the input node
 * @inst: the stylesheet node 
 * @comp: the compiled information from the stylesheet 
 * 
 * signature of the function associated to elements part of the
 * stylesheet language like xsl:if or xsl:apply-templates.
 */ 
typedef void (*xsltTransformFunction)
                          (xsltTransformContextPtr ctxt,
                           xmlNodePtr node,
                           xmlNodePtr inst,
                           xsltStylePreCompPtr comp);

The first argument is the XSLT transformation context. The second and third arguments are xmlNodePtr i.e. internal memory representation of XML nodes. They are respectively node from the the input document being transformed by the stylesheet and inst the extension element in the stylesheet. The last argument is comp a pointer to a precompiled representation of inst but usually for extension function this value is NULL by default (it could be added and associated to the instruction in inst->_private).

The same functions are available from a function implementing an extension element as in an extension function, including xsltGetExtData().

The goal of extension element being usually to enrich the generated output, it is expected that they will grow the currently generated output tree, this can be done by grabbing ctxt->insert which is the current libxml node being generated (Note this can also be the intermediate value tree being built for example to initialize a variable, the processing should be similar). The functions for libxml tree manipulation from <libxml/tree.h> can be employed to extend or modify the tree, but it is required to preserve the insertion node and its ancestors since there is existing pointers to those elements still in use in the XSLT template execution stack.

Example for extension elements

The module libxslt/transform.c containsthe sources of the XSLT built-in elements, including xsl:element, xsl:attribute, xsl:if, etc. There is a small but full example in functions.c providing the implementation for the libxslt:test element, it will output a comment in the result tree:

/**
 * xsltExtElementTest:
 * @ctxt:  an XSLT processing context
 * @node:  The current node
 * @inst:  the instruction in the stylesheet
 * @comp:  precomputed informations
 *
 * Process a libxslt:test node
 */
static void
xsltExtElementTest(xsltTransformContextPtr ctxt, xmlNodePtr node,
                   xmlNodePtr inst,
                   xsltStylePreCompPtr comp)
{
    xmlNodePtr comment;

    if (ctxt == NULL) {
        xsltGenericError(xsltGenericErrorContext,
                         "xsltExtElementTest: no transformation context\n");
        return;
    }
    if (node == NULL) {
        xsltGenericError(xsltGenericErrorContext,
                         "xsltExtElementTest: no current node\n");
        return;
    }
    if (inst == NULL) {
        xsltGenericError(xsltGenericErrorContext,
                         "xsltExtElementTest: no instruction\n");
        return;
    }
    if (ctxt->insert == NULL) {
        xsltGenericError(xsltGenericErrorContext,
                         "xsltExtElementTest: no insertion point\n");
        return;
    }
    comment =
        xmlNewComment((const xmlChar *)
                      "libxslt:test element test worked");
    xmlAddChild(ctxt->insert, comment);
}

The shutdown of a module

When the XSLT processor ends a transformation, the shutdown function (if it exists) of all the modules initialized are called.The xsltExtShutdownFunction type defines the interface for a shutdown function:

/**
 * xsltExtShutdownFunction:
 * @ctxt:  an XSLT transformation context
 * @URI:  the namespace URI for the extension
 * @data:  the data associated to this module
 *
 * A function called at shutdown time of an XSLT extension module
 */
typedef void (*xsltExtShutdownFunction) (xsltTransformContextPtr ctxt,
                                         const xmlChar *URI,
                                         void *data);

this is really similar to a module initialization function except a third argument is passed, it's the value that was returned by the initialization function. This allow to deallocate resources from the module for example close the connection to the database to keep the same example.

Future work

Well some of the pieces missing:

  • a way to load shared libraries to instanciate new modules
  • a better detection of extension function usage and their registration without having to use the extension prefix which ought to be reserved to element extensions.
  • more examples
  • implementations of the EXSLT common extension libraries, Thomas Broyer nearly finished implementing them.

Contributions

I'm still waiting for someone to contribute a simple XSLT processing module for Apache :-)

Daniel Veillard