Joseph Brenner
November 27, 2007
These notes can be found at: http://obsidianrook.com/devnotes/talks/test_everything/
An ambitious topic for a lighting talk, but I had a hard time coming up with a more narrow title. (Funny: the Perl Testing Notebook closes with a section titled similarly).
I got interested recently in using perl test frameworks to test external code, including things that aren't necessarily perl
This was inspired by some recent mysterious problems I've had with emacs installation(s), which is why emacs is used in my early examples here, but the techniques aren't emacs-specific.
There are few things I'd like to cover...
The first technique is the obvious one: shelling out to some other program that generates textual output that the perl code then analyzes.
My immediate goal here was to test an emacs regexp.
The perlnow.el package uses this on *.pm files to pick out the name of the perl module from the package line:
"^[ \t]*package[ \t\n]+\\(.*?\\)[ \t\n;]"
We can test this emacs regexp using a perl test harness by using an obscure feature of emacs: it can be run non-interactively by with the "--batch" command-line option. Then it's just a lisp interpreter (with extra text-manipulation features): you can execute a chunk of elisp code passed in via the "--eval" option so from within perl I can do things like this:
$emacs_cmd =
"emacs -q --batch -eval '$elisp' -f $func_name >& $log";
This script is a working an example of testing an
elisp snippet using a perl framework:
1-test_emacs_regexps.t
If you run it, you see this exciting output:
perl /home/doom/End/Cave/GuiTest/Wall/code/1-test_emacs_regexps.t ok 1 - Testing case number 1: Expected failure: no space after 'package' ok 2 - Testing case number 2: Open bracket same line as semi-colon ok 3 - Testing case number 3: a string after semi-colon ok 4 - Testing case number 4: Space before semi-colon ok 5 - Testing case number 5: linebreak seperators 1..5
The second techniques here are more interesting as far as perl features are concerned... it makes use of fork, exec and kill.
One of the problems I've had of late is emacs syntax coloring intermittently going away for no particular reason.
If I could write an automated test to determine if syntax coloring is working, then I could hypothetically write automated probes of the problem.
The second example here shows a way of doing this:
2-test_emacs_syntax_coloring.t
Here we start up a new emacs, take a screenshot of it, and analyze the image. Then we get rid of the new emacs after we don't need it any more.
The way we do this is to fork the current process, and save the returned pid. The code then:
There's an Image::Magick method named "colors" that gives us the number of colors in use in the image. If we've got more than a half-dozen or so, we can be reasonably confident syntax coloring is working.
A typical test run with this code might look like:
perl /home/doom/End/Cave/GuiTest/Wall/code/2-test_emacs_syntax_coloring.t
I am the child. About to exec:
emacs --font=9x15 --no-splash /home/doom/End/Cave/GuiTest/bin/testes-syntax_colors -f font-lock-fontify-buffer --eval ' (progn
(set-frame-height (selected-frame) 30 )
(set-frame-width (selected-frame) 80 )
(set-frame-position (selected-frame) 0 0))
'
I am the parent: 27899
The child is: 27901
Wait a moment... about to take a picture of the child
Death to the child
ok 1 - Testing that there are more syntax colors visible than 7
1..1
This is the way a screen-capture looks after being
cropped (but before trimming the decorations):
The third technique took a bit more programming on my part. I was wondering if there might be a way to keep rolling with this technique, to write tests on code that has a GUI interface.
It's always bugged me that you can easily write tests for the easy stuff, like a string parsing routine, but it's not so easy to check a perl/Tk routine that puts a window up with some buttons and things.
Though actually, if you've got the technique down of capturing a window off of the screen, you can obviously put up a perl/Tk dialog, take a picture of it, and then compare that picture to a known good copy with a binary diff or some thing like that. (And of course, perl/Tk itself runs tests like this during installation).
But that's a pretty brittle test that will "fail" for the tiniest change in the dialog, even minor layout changes that don't impact functionality.
To take this a little further, I wondered if I could write "fuzzier" tests of functionality, in analogy to the "colors" method used above: I wanted a routine that could count the number of rectangles in a window.
The idea is then you'd have more robust tests that won't panic if a button is moved over 10 pixels, as long as it's still there.
I looked around a little for code that might do this, and didn't turn up anything... so I tried my hand at hacking something out:
Image::BoxFind
Image::BoxFind is a "proof of concept", which is another
way of saying it doesn't work very well.
It would be easy to rig the demo with a simple case such as this:
where Image::Boxfind finds the following rectangles (highlighted in red):
On this one it works okay, but behaves a little strangely:
Note, it sees one square file icon, but misses another, and it doesn't see the blue band there,
nor does it see the white areas as rectangles:
And there are other cases where it misses quite a bit:
You might hope that it would see at least some of
those tabs across the top, even if the checkboxes are
a little small to register:
