Joseph Brenner
                                       November 27, 2007

These notes can be found at:    http://obsidianrook.com/devnotes/talks/test_everything/

Using perl to test everything

An ambitious topic for a lighting talk, but I had a hard time coming up with a more narrow title. (Funny: the Perl Testing Notebook closes with a section titled similarly).

I got interested recently in using perl test frameworks to test external code, including things that aren't necessarily perl

This was inspired by some recent mysterious problems I've had with emacs installation(s), which is why emacs is used in my early examples here, but the techniques aren't emacs-specific.

There are few things I'd like to cover...

analyzing text output

The first technique is the obvious one: shelling out to some other program that generates textual output that the perl code then analyzes.

My immediate goal here was to test an emacs regexp.

The perlnow.el package uses this on *.pm files to pick out the name of the perl module from the package line:

      "^[ \t]*package[ \t\n]+\\(.*?\\)[ \t\n;]"

We can test this emacs regexp using a perl test harness by using an obscure feature of emacs: it can be run non-interactively by with the "--batch" command-line option. Then it's just a lisp interpreter (with extra text-manipulation features): you can execute a chunk of elisp code passed in via the "--eval" option so from within perl I can do things like this:

    $emacs_cmd =
      "emacs -q --batch  -eval '$elisp' -f $func_name >& $log";

This script is a working an example of testing an elisp snippet using a perl framework:
    1-test_emacs_regexps.t

If you run it, you see this exciting output:


perl /home/doom/End/Cave/GuiTest/Wall/code/1-test_emacs_regexps.t
ok 1 - Testing case number 1: Expected failure: no space after 'package'
ok 2 - Testing case number 2: Open bracket same line as semi-colon
ok 3 - Testing case number 3: a string after semi-colon
ok 4 - Testing case number 4: Space before semi-colon
ok 5 - Testing case number 5: linebreak seperators
1..5

capturing a GUI window image: counting colors

The second techniques here are more interesting as far as perl features are concerned... it makes use of fork, exec and kill.

One of the problems I've had of late is emacs syntax coloring intermittently going away for no particular reason.

If I could write an automated test to determine if syntax coloring is working, then I could hypothetically write automated probes of the problem.

The second example here shows a way of doing this:

  2-test_emacs_syntax_coloring.t

Here we start up a new emacs, take a screenshot of it, and analyze the image. Then we get rid of the new emacs after we don't need it any more.

The way we do this is to fork the current process, and save the returned pid. The code then:

  1. examines the pid to determine whether it's the child or the parent copy
  2. if it's the child, it uses exec to transform itself from a process running perl to a process running emacs (and since we're re-using the process, the pid doesn't change).
  3. the parent pauses to give the new emacs time to render, then takes a picture of the child (by shelling out to import, in this case),
  4. the parent then kills the child.
Then we do some image crunching using Image::Magick, cropping the screenshot so that it just shows the emacs window, and (usually) trimming it further to get rid of the window decorations.

There's an Image::Magick method named "colors" that gives us the number of colors in use in the image. If we've got more than a half-dozen or so, we can be reasonably confident syntax coloring is working.

A typical test run with this code might look like:

    perl /home/doom/End/Cave/GuiTest/Wall/code/2-test_emacs_syntax_coloring.t
    I am the child.  About to exec:
    emacs --font=9x15 --no-splash /home/doom/End/Cave/GuiTest/bin/testes-syntax_colors -f font-lock-fontify-buffer --eval ' (progn
    (set-frame-height   (selected-frame)  30 )
    (set-frame-width    (selected-frame)  80 )
    (set-frame-position (selected-frame) 0 0))
   '
    I am the parent: 27899
    The child is: 27901
    Wait a moment... about to take a picture of the child
    Death to the child
    ok 1 - Testing that there are more syntax colors visible than 7
    1..1

This is the way a screen-capture looks after being cropped (but before trimming the decorations):

analysing a GUI image capture: image diffs

The third technique took a bit more programming on my part. I was wondering if there might be a way to keep rolling with this technique, to write tests on code that has a GUI interface.

It's always bugged me that you can easily write tests for the easy stuff, like a string parsing routine, but it's not so easy to check a perl/Tk routine that puts a window up with some buttons and things.

Though actually, if you've got the technique down of capturing a window off of the screen, you can obviously put up a perl/Tk dialog, take a picture of it, and then compare that picture to a known good copy with a binary diff or some thing like that. (And of course, perl/Tk itself runs tests like this during installation).

But that's a pretty brittle test that will "fail" for the tiniest change in the dialog, even minor layout changes that don't impact functionality.

analysing a GUI image capture: identifying rectangles

To take this a little further, I wondered if I could write "fuzzier" tests of functionality, in analogy to the "colors" method used above: I wanted a routine that could count the number of rectangles in a window.

The idea is then you'd have more robust tests that won't panic if a button is moved over 10 pixels, as long as it's still there.

I looked around a little for code that might do this, and didn't turn up anything... so I tried my hand at hacking something out:

   Image::BoxFind

Image::BoxFind is a "proof of concept", which is another way of saying it doesn't work very well.
It would be easy to rig the demo with a simple case such as this:

where Image::Boxfind finds the following rectangles (highlighted in red):

On this one it works okay, but behaves a little strangely:

Note, it sees one square file icon, but misses another, and it doesn't see the blue band there, nor does it see the white areas as rectangles:

And there are other cases where it misses quite a bit:

You might hope that it would see at least some of those tabs across the top, even if the checkboxes are a little small to register:


Joseph Brenner, 06 Oct 2007