Test Tools

Overview

Testing Web-based, JavaScript-heavy or just HTML-based applications poses special issues. It is one thing to create a button in a form, fire up the browser and click it to see what happens; or testing a complete and complex Web GUI, achieving sufficient test coverage; doing unit testing of specific parts of an application; or even run automated tests against such a GUI, ensuring old functionality continues to work when development proceeds (regression tests).

Many agile development processes put great emphasis on tests, yet applying such approaches to Web development is not straight forward. The (almost) single deployment platform, the browser, is itself a GUI application that is not immediately accessible to non-interactive treatment. Several approaches exist:

  • "Mouse-automation" - Mouse and keyboard actions are applied to the app as if a human was sitting in front of it - usually produced by recording the actions of a real human beforehand. This approach, well known from native GUI app development, has the least requirements towards the application and its implementation since it relies on the least common denominator - the interactive interface -, but is often fragile and demanding in preparation and maintenance.
  • "HTML-Scripting" - The "assembler of the Web", HTML, is easily parseable (well, sort of) and allows e.g. for automatic execution of links. While good for static HTML pages, this approach does not work for dynamic client-side documents, as is usually the case with JavaScript.
  • "DOM-Scripting" - Looking at the DOM rather the source HTML you capture both static as well as dynamic pages. This is suitable for complex JavaScript applications, and the way to implement the test is again via JavaScript. But to execute the test code in the browser, alongside the application, with enough control and yet not introducing unwanted side effects is the challenge here.

Other concerns have also to be considered:

  • Browsers are not easily controlled automatically. IE can be driven from a COM controller, Firefox offers various JavaScript plugins of the GreaseMonkey type. Some approaches try to replace the traditional browsers with "batch browsers" which are easier to control and are helpful to inspect flow of control and data, but fall short when all the visual and rendering issues are concerned.
  • Tests are ideally developed cross-browser/cross-platform, allowing for testing various browsers on various operating systems with the same code. This is not easily achieved.
  • The network always comes into play. A server may be down or loaded, requests time out or responses are incomplete. Backends are too heavy-weight for routine testing and databases do not allow for messing around with the data. Authentication, authorization, session management, user profiling, a stateless protocol … you name it. There are numerous constraints and side effects that are part of the game. Local servers and backend mock-ups become part of the overall test bed. Offline becomes a big issue, and things like Google Gears might gain the status of an essential part in the test environment. Not that their inventors intended that.
  • Tests are not only run, they're run for a purpose. Their purpose is to detect flaws and errors. But what is an error? If an external linked page cannot be retreived it is not the fault of the core app. An HTTP 200 return code is not a reliable sign of correctness, when the returned page contains the wrong data. Successful display of a new widget in the GUI is not a success in itself when this widget is poorly aligned, obscuring other essential parts of the GUI, or has a hampered display of the contained information or controls. Or the colors or the borders get screwed up. How can a program detect all of this?! Performance is an issue, but when is a program reaction too slow, when human preception (and judgement) is not strictly linear in time?!
  • Reporting - After all the tests are run you want a good report, spotting problems and shortcomings of the app under investigation. Go figure …

This page is intended as a broad collection of tools, techniques and tips that should help to tackle this area.

qooxdoo Specific

Unit Tests

The qooxdoo SDK comes with a build-in test framework, the Test Runner. It is a JSUnit like test application that offers a navigational GUI to run dedicated test classes. These classes in turn invoke methods of the classes under investigaton, check their behaviour and return values, and communicate with the test framework. You can easily create this test environment for your own Skeleton-based development with the ./generate.py test command.

DOM Automation

A suitable approach to testing complete qooxdoo applications seems to take advantage of DOM automation. The tests work on the same level as qooxdoo, can react on changes, or manipulate the DOM themselves. The challenge here is that most of the DOM that qooxdoo generates are multitudes of nested DIV elements, and it is not entirely obvious from the DOM tree which DIV implements which qooxdoo widget.

A quick fix to overcome this issue would be to add HTML id or name attributes (using the setHtmlProperty method on a qooxdoo widget). This would allow the test code to easily identify controls it wants to manipulate (e.g. like pushing a button or selecting from a combo). But this ultimately means to "instrument" the application code, i.e. extend it just for the purpose of testing it later. The ultimate goal would be to efficiently test a "vanilla" application, one which is completely unaware of the testing.

JavaScript Objects

In order to test an application, what you really want is to address qooxdoo runtime objects. You want to code something like this:

var myapp = qx.core.Init.getInstance().getApplication();
// 'push()' being a function of the test framework
push(myapp.button1);
// or just with qooxdoo
myapp.button1.execute();

Yet how to achieve this kind of referencing under a testing framework is still an open question.

Test Tools

An assorted list of test/test automation tools that are out there and we have heard people use.

    • Uses DOM automation. JavaScript runs the application under test (AUT) in a dedicated iframe.
    • Has a server component (Selenium RC) to start/stop and control browsers remotely (using a polling component in the JS code).
    • qooxdoo-specific customisation available - documented here.
    • Why? The offline features of Google Gears might provide a helpful decoupling of a Web frontend from its server/backend, to allow for modular testing.
    • A general job scheduling/process automation suite; offers GUI ("mouse") automation.