This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
To: modperl@apache.org
From: David Wheeler <david@wheeler.net>
Subject: UI Regression Testing
Date: 25 Jan 2002 09:46:04 -0800
Hi All,
A big debate is raging on the Bricolage development list WRT CVS
configuration and application testing.
http://www.geocrawler.com/mail/thread.php3?subject=%5BBricolage-Devel%5D+More+on+Releases&list=15308
It leads me to a question about testing. Bricolage is a monster
application, and its UI is built entirely in HTML::Mason running on
Apache. Now, while we can and will do a lot more to improve the testing
of our Perl modules, we can't really figure out a way to automate the
testing of the UI. I'm aware of the the performance testing utilities
mentioned in the mod_perl guide --
http://perl.apache.org/guide/performance.html
-- but they don't seem to be suited to testing applications.
Is anyone familiar with how to go about setting up a test suite for a
web UI -- without spending an arm and a leg? (Remember, Bricolage is an
OSS effort!).
Thanks!
David
===
To: modperl@apache.org
From: Rob Nagler <nagler@bivio.biz>
Subject: Re: UI Regression Testing
Date: Fri, 25 Jan 2002 11:03:42 -0700
> Is anyone familiar with how to go about setting up a test suite for a
> web UI -- without spending an arm and a leg? (Remember, Bricolage is an
> OSS effort!).
Yes, it's very easy. We did this using student labor, because it is
an excellent project for students and it's probably cheaper. It's
very important. We run our test suite nightly.
I'm an extreme programming (XP) advocate. Testing is one of the most
important practices in XP.
I'm working on packaging what we did so it is fit for public
consumption. Expect something in a month or so. It'll come with a
rudimentary test suite for our demo petshop app.
There are many web testers out there. To put it bluntly, they don't
let you write maintainable test suites. The key to maintainability is
being able to define your own domain specific language. Just like
writing maintainable code, you have to encapsulate commonality and
behavior. The scripts should be short and only contain the details
pertinent to the particular test. Perl is ideal for this, because you
can easily create domain specific languages.
===
To: "Rob Nagler" <nagler@bivio.biz>, <modperl@apache.org>
From: "Perrin Harkins" <perrin@elem.com>
Subject: Re: UI Regression Testing
Date: Fri, 25 Jan 2002 13:12:07 -0500
> There are many web testers out there. To put it bluntly, they don't
> let you write maintainable test suites. The key to maintainability is
> being able to define your own domain specific language.
Have you tried webchat? You can find webchatpp on CPAN.
===
To: Perrin Harkins <perrin@elem.com>
From: David Wheeler <david@wheeler.net>
Subject: Re: UI Regression Testing
Date: 25 Jan 2002 11:52:22 -0800
On Fri, 2002-01-25 at 10:12, Perrin Harkins wrote:
> Have you tried webchat? You can find webchatpp on CPAN.
Looks interesting, although the documentation is rather sparse. Anyone
know of more examples than come with it?
Thanks,
David
===
To: modperl@apache.org
From: Rob Nagler <nagler@bivio.biz>
Subject: Re: UI Regression Testing
Date: Fri, 25 Jan 2002 14:11:12 -0700
> Have you tried webchat? You can find webchatpp on CPAN.
Just had a look. It appears to be a rehash of chat (expect) for the
web. Great stuff, which is really needed and demonstrates the power
of Perl for test scripting.
But...
This is a bit hard to explain. There are two types of XP testing:
unit and acceptance. Unit testing is pretty clear in Perl circles
(ok, I have a thing or two to say about it, but not now :-).
Acceptance testing (aka functional testing) is traditionally handled
by a third party testing organization. The test group writes scripts.
If they are testing GUIs, they click in scripts via a session
recorder. They don't "program" anything. There's almost no reuse,
and very little abstraction.
XP flips testing on its head. It says that the programmers are
responsible for testing, not some 3rd party org. The problem I have
found is that instead of programming the test suite, XPers script it,
using the same technology that a testing organization would use. With
the advent of the web, this is a real shame.
HTTP and HTML are middleware. You have full programmatic control to
test your application. You can't control the web browser, so you
still need to do some ad hoc "how does it look" testing, but this
isn't the hard part.
The acceptance test suite is testing the system from the user's point
of view. In XP, the user is the customer, and the customer writes
tests. In my opinion, this means the customer writes tests in a pair
with a programmer. The programmer's job is to create a language which
the user understands.
Here's an example from our test suite:
Accounting->setup_investment('AAPL');
The user knows what an investment is. She also knows that AAPL is a
stock ticker. This statement sets up the environment (using LWP to
the app) to execute tests such as entering dividends, buys, sells,
etc.
The test infrastructure must support the ability to create new
language elements with the ability to build elements using the other
elements. This requires modularization, and today this means classes
and instances. There's also a need for state management, just like
the request object in your web application.
Part of the packaging process we're going through is making it even
easier to create domain specific languages. You actually want to
create lots of dialects, e.g. in our case this means investments, cash
accounts, member accounts, and message boards. These dialects use
building blocks such as logging in, creating a club, and so on. At
the bottom you use LWP or webchat. However, the user doesn't care if
the interface is HTTP or Windows. You're job as a test suite
programmer is meeting her domain knowledge, and abstracting away
details like webchat's CLICK and EXPECT OK.
In the end, your test suite is a domain knowledge repository. It
contains hundreds of concise scenarios comprised of statements, or
facts, in knowledge base parlance. The execution of the test suite
asserts all the facts are true about your application. The more
concise the test language. The more easily the user-tester can verify
that she has encoded her expertise correctly.
===
To: modperl@apache.org
From: Gunther Birznieks <gunther@extropia.com>
Subject: Re: UI Regression Testing
Date: Sat, 26 Jan 2002 08:33:33 +0800
I suppose it depends on what you want out of testing.
Frequently, unit testing is OK in simple applications. But in an
application whose job it is to communicate with a mainframe or back-end
databases, frequently the tests you might perform are based on some
previous persistent state of the database.
It takes a lot of effort in other words. For example, 2-3 programmers could
easily get away (through normal testing) with sharing a database. But
generally if you require each programmer to have their own database and
their own system completely where they are constantly wiping the state of
the database to perform a test suite, this can get time consuming and
entails a lot of infrastructural overhead.
I've seen some articles that demonstrate doing something with test XML
backends to emulate database retrieval results, but these seem quite hard
to set up also.
I agree that testing is great, but I think it is quite hard in practice.
Also, I don't think programmers are good to be the main people to write
their own tests. It is "OK" for programmers to write their own tests but
frequently it is the user or a non-technical person who is best at doing
the unexpected things that are really were the bug lies.
The other annoying thing about programmers writing tests is deciding where
to stop. I believe the HTTP level for tests is really good. But I see much
unit testing being done on the basis of writing a test class for every
class you write. Ugh! That means that any time you refactor you throw away
the 2x the coding you did.
To some degree, there should be intelligent rules of thumb as to which
interfaces tests should be written to because the extreme of writing tests
for everything is quite bad.
Finally, unit tests do not guarantee an understanding of the specs because
the business people generally do not read test code. So all the time spent
writing the test AND then writing the program AND ONLY THEN showing it to
the users, then you discover it wasn't what the user actually wanted. So 2x
the coding time has been invalidated when if the user was shown a prototype
BEFORE the testing coding commenced, then the user could have confirmed or
denied the basic logic.
The same frequently and especially is true for UIs.
===
To: modperl@apache.org
From: Rob Nagler <nagler@bivio.biz>
Subject: Re: UI Regression Testing
Date: Fri, 25 Jan 2002 17:59:10 -0700
Gunther Birznieks writes:
> the database to perform a test suite, this can get time consuming and
> entails a lot of infrastructural overhead.
We haven't found this to be the case. All our database operations are
programmed. We install the database software with an RPM, run a
program to build the database, and program all schema upgrades. We've
had 194 schema upgrades in about two years.
> unit testing being done on the basis of writing a test class for every
> class you write. Ugh! That means that any time you refactor you throw away
> the 2x the coding you did.
By definition, refactoring doesn't change observable behavior. You
validate refactorings with unit tests. See http://www.refactoring.com
> To some degree, there should be intelligent rules of thumb as to which
> interfaces tests should be written to because the extreme of writing tests
> for everything is quite bad.
Again, we haven't seen this. Every time I don't have unit tests, I
get nervous. How do I know if I broke something with my change?
> Finally, unit tests do not guarantee an understanding of the specs because
> the business people generally do not read test code. So all the time spent
> writing the test AND then writing the program AND ONLY THEN showing it to
> the users, then you discover it wasn't what the user actually wanted. So 2x
> the coding time has been invalidated when if the user was shown a prototype
> BEFORE the testing coding commenced, then the user could have confirmed or
> denied the basic logic.
Unit tests aren't about specs. They are about APIs. Acceptance tests
need to be written by the user or written so the user can understand
them. You need both kinds of testing.
See http://www.xprogramming.com/xpmag/Reliability.htm
===
To: modperl@apache.org
From: Ed Grimm <ed@tgape.org>
Subject: Re: UI Regression Testing
Date: Fri, 25 Jan 2002 19:20:14 -0500 (EST)
On Sat, 26 Jan 2002, Gunther Birznieks wrote:
> I agree that testing is great, but I think it is quite hard in practice.
> Also, I don't think programmers are good to be the main people to write
> their own tests. It is "OK" for programmers to write their own tests but
> frequently it is the user or a non-technical person who is best at doing
> the unexpected things that are really were the bug lies.
My experience is that the best testers come from technical support,
although this is not to suggest that all technical support individuals
are good at this; even among this group, it's rare. Users or other
non-technical people may find a few more bugs, but frequently, the
non-technical people don't have the ability to correctly convey how to
reproduce the problems, or even what the problem was. "I clicked on the
thingy, and it didn't work."
This being said, users and tech support can't create unit tests; they're
not in a position to.
> Finally, unit tests do not guarantee an understanding of the specs because
> the business people generally do not read test code. So all the time spent
> writing the test AND then writing the program AND ONLY THEN showing it to
> the users, then you discover it wasn't what the user actually wanted. So 2x
> the coding time has been invalidated when if the user was shown a prototype
> BEFORE the testing coding commenced, then the user could have confirmed or
> denied the basic logic.
For your understanding of the spec, you use functional tests. If your
functional test suite uses test rules which the users can understand,
you can get the users to double-check them.
For example, at work, we use a suite which uses a rendered web page as
its test output, and the input can be sent to a web page to populate a
form; this can be read by most people who can use the application.
Unit software is a means of satisfying a spec, but it doesn't satisfy
the spec itself - if it did, you'd be talking about the entire package,
and therefore refering to functional testing. (At least, this is the
way I distinguish between them.)
Admittedly, we are a bit lacking in our rules, last I checked.
Ed
===
To: "Rob Nagler" <nagler@bivio.biz>, <modperl@apache.org>
From: "Perrin Harkins" <perrin@elem.com>
Subject: Re: UI Regression Testing
Date: Sat, 26 Jan 2002 00:23:40 -0500
> Gunther Birznieks writes:
> > the database to perform a test suite, this can get time consuming
and
> > entails a lot of infrastructural overhead.
>
> We haven't found this to be the case. All our database operations are
> programmed. We install the database software with an RPM, run a
> program to build the database, and program all schema upgrades. We've
> had 194 schema upgrades in about two years.
But what about the actual data? In order to test my $product->name()
method, I need to know what the product name is in the database. That's
the hard part: writing the big test data script to run every time you
want to run a test (and probably losing whatever data you had in that
database at the time).
This has been by far the biggest obstacle for me in testing, and from
Gunther's post it sounds like I'm not alone. If you have any ideas
about how to make this less painful, I'd be eager to hear them.
===
To: David Wheeler <david@wheeler.net>, modperl list
From: Stas Bekman <stas@stason.org>
Subject: Re: UI Regression Testing
Date: Sat, 26 Jan 2002 13:42:42 +0800
David Wheeler wrote:
> Hi All,
>
> A big debate is raging on the Bricolage development list WRT CVS
> configuration and application testing.
>
> http://www.geocrawler.com/mail/thread.php3?subject=%5BBricolage-Devel%5D+More+on+Releases&list=15308
>
> It leads me to a question about testing. Bricolage is a monster
> application, and its UI is built entirely in HTML::Mason running on
> Apache. Now, while we can and will do a lot more to improve the testing
> of our Perl modules, we can't really figure out a way to automate the
> testing of the UI. I'm aware of the the performance testing utilities
> mentioned in the mod_perl guide --
>
> http://perl.apache.org/guide/performance.html
>
> -- but they don't seem to be suited to testing applications.
>
> Is anyone familiar with how to go about setting up a test suite for a
> web UI -- without spending an arm and a leg? (Remember, Bricolage is an
> OSS effort!).
You probably also need some good back-end engine for testing since you
mostly likely need to test against a live Apache/mod_perl. If that's the
case you should try to use the new Apache::Test framework that will be
released with mod_perl 2.0. You can get it from here:
http://cvs.apache.org/snapshots/modperl-2.0/ (or use cvs)
Some docs are here:
http://cvs.apache.org/viewcvs.cgi/modperl-docs/src/docs/2.0/devel/testing/testing.pod
===
To: "Perrin Harkins" <perrin@elem.com>
From: Tatsuhiko Miyagawa <miyagawa@edge.co.jp>
Subject: Re: UI Regression Testing
Date: Sat, 26 Jan 2002 15:37:50 +0900
On Sat, 26 Jan 2002 00:23:40 -0500
"Perrin Harkins" <perrin@elem.com> wrote:
> But what about the actual data? In order to test my $product->name()
> method, I need to know what the product name is in the database. That's
> the hard part: writing the big test data script to run every time you
> want to run a test (and probably losing whatever data you had in that
> database at the time).
>
> This has been by far the biggest obstacle for me in testing, and from
> Gunther's post it sounds like I'm not alone. If you have any ideas
> about how to make this less painful, I'd be eager to hear them.
You're not alone ;) here is my solution.
* All datasource are maintained with separate config file
* Generate config file for testing
* Create database and tables for testing (called test_foo)
* Insert dummy data into test_foo
* Test 'em
* Drop dummy data
Then my test script has both client side testing and server side
testing, like this.
use Test::More 'no_plan';
BEGIN { do 'db_setup.pl'; }
END { do 'db_teardown.pl'; }
# server-side
my $product = Product->create({ name => 'foo' });
# client-side
my $ua = LWP::UserAgent->new;
my $res = $ua->request(GET "/foo/bar");
like $res->content, qr/foo/;
my $form = HTML::Form->parse($res->content);
my $req2 = $form->click;
my $res2 = $ua->request($req);
like $res2->content, qr/blah/;
# server-side
my @p = Product->retrieve_all;
is @p, 2;
===
To: modperl@apache.org
From: Gunther Birznieks <gunther@extropia.com>
Subject: Re: UI Regression Testing
Date: Sat, 26 Jan 2002 19:14:39 +0800
At 08:59 AM 1/26/2002, Rob Nagler wrote:
>Gunther Birznieks writes:
> > the database to perform a test suite, this can get time consuming and
> > entails a lot of infrastructural overhead.
>
>We haven't found this to be the case. All our database operations are
>programmed. We install the database software with an RPM, run a
>program to build the database, and program all schema upgrades. We've
>had 194 schema upgrades in about two years.
>
> > unit testing being done on the basis of writing a test class for every
> > class you write. Ugh! That means that any time you refactor you throw away
> > the 2x the coding you did.
>
>By definition, refactoring doesn't change observable behavior. You
>validate refactorings with unit tests. See http://www.refactoring.com
>
> > To some degree, there should be intelligent rules of thumb as to which
> > interfaces tests should be written to because the extreme of writing tests
> > for everything is quite bad.
>
>Again, we haven't seen this. Every time I don't have unit tests, I
>get nervous. How do I know if I broke something with my change?
>
> > Finally, unit tests do not guarantee an understanding of the specs because
> > the business people generally do not read test code. So all the time spent
> > writing the test AND then writing the program AND ONLY THEN showing it to
> > the users, then you discover it wasn't what the user actually wanted.
> So 2x
> > the coding time has been invalidated when if the user was shown a
> prototype
> > BEFORE the testing coding commenced, then the user could have confirmed or
> > denied the basic logic.
>
>Unit tests aren't about specs. They are about APIs. Acceptance tests
>need to be written by the user or written so the user can understand
>them. You need both kinds of testing.
>See http://www.xprogramming.com/xpmag/Reliability.htm
>
>Rob
From the description of your scenario, it sounds like you have a long
product life cycle etc.
I think your testing, especially regression testing and the amount of
effort you put into it makes a lot of sense because your software is a
long-term investment possibly even a product.
I think you can "worry" about an API not having a test for it, and I do a
bit, but I don't lose sleep over it. The reality is many APIs succeed OK
without tests and maybe a few grumbles hear and there if and when things
break.
I've rarely seen a documented API break in such a significant way that
either I programmed or used from someone else that I could not easily
rectify it or go back to a previous version until it was resolved. ie major
API changes usually have a way of being known anyway very quickly.
Would API tests for every API in existance solve that? Could be, probably
yes. But is it worth all the extra coding time for those tests? I don't
think so. Not for all APIs.
To each his own I guess.
I agree with tests for some things, just not for all things including not
all APIs.
===
To: modperl@apache.org
From: Rob Nagler <nagler@bivio.biz>
Subject: Re: UI Regression Testing
Date: Sat, 26 Jan 2002 09:59:57 -0700
Hi Craig,
> Have you ever heard of the hw verification tool Specman Elite by Verisity
> (www.verisity.com)?
No, but it looks interesting. It would be good to have something like
this for unit tests. I haven't had very good experience with
automated acceptance testing, however. The software should be robust
against garbage in, but the main problem we have is making sure the
numbers add up, and that we generate the correct tax forms! It's
pretty tricky stuff.
FWIW, we are very happy with our unit test structure. It has evolved
over many years, and many different languages. I've appended a simple
example, because it is quite different than most of the unit testing
frameworks out there. It uses the XP philosophy of once and only once
as well as test what is likely to break.
Rob
----------------------------------------------------------------------
#!perl -w
# $Id: Integer.t,v 1.7 2001/11/24 04:30:19 nagler Exp $
#
use strict;
use Bivio::Test;
use Bivio::Type::Integer;
use Bivio::TypeError;
Bivio::Test->unit([
'Bivio::Type::Integer' => [
get_min => -999999999,
get_max => 999999999,
get_precision => 9,
get_width => 10,
get_decimals => 0,
can_be_zero => 1,
can_be_positive => 1,
can_be_negative => 1,
from_literal => [
['9'] => [9],
['+00009'] => [9],
['-00009'] => [-9],
['x'] => [undef, Bivio::TypeError->INTEGER],
[undef] => [undef],
[''] => [undef],
[' '] => [undef],
['-99999999999999'] => [undef, Bivio::TypeError->NUMBER_RANGE],
['-00000000000009'] => [-9],
['+00000000000009'] => [9],
['-999999999'] => [-999999999],
['+999999999'] => [999999999],
['+1000000000'] => [undef, Bivio::TypeError->NUMBER_RANGE],
['-1000000000'] => [undef, Bivio::TypeError->NUMBER_RANGE],
],
],
Bivio::Type::Integer->new(1,10) => [
get_min => 1,
get_max => 10,
get_precision => 2,
get_width => 2,
get_decimals => 0,
can_be_zero => 0,
can_be_positive => 1,
can_be_negative => 0,
from_literal => [
['00001'] => [1],
['+00001'] => [1],
['0'] => [undef, Bivio::TypeError->NUMBER_RANGE],
['11'] => [undef, Bivio::TypeError->NUMBER_RANGE],
['-00001'] => [undef, Bivio::TypeError->NUMBER_RANGE],
[undef] => [undef],
['-00000000000009'] => [undef, Bivio::TypeError->NUMBER_RANGE],
['+00000000000009'] => [9],
],
],
]);
===
To: modperl@apache.org
From: Rob Nagler <nagler@bivio.biz>
Subject: Re: UI Regression Testing
Date: Sat, 26 Jan 2002 10:38:02 -0700
Perrin Harkins writes:
> But what about the actual data? In order to test my $product->name()
> method, I need to know what the product name is in the database. That's
> the hard part: writing the big test data script to run every time you
> want to run a test (and probably losing whatever data you had in that
> database at the time).
There are several issues here. I have answers for some but not all.
We don't do complex unit tests. We save those for the acceptance test
suite. The unit tests do simple things. I've attached a basic unit
test for our DBI abstraction layer. It runs on Oracle and Postgres.
Acceptance tests take over an hour to run. We have a program which
sets up some basic users and clubs. This is run once. It could be
run before each test suite run, but we don't. We have tests which
test creating users, entering subscription payments, twiddling files
and email. By far the biggest piece is testing our accounting. As I
said, we used student labor to write the tests. They aren't perfect,
but they catch lots of errors that we miss.
Have a look at:
http://petshop.bivio.biz/src?s=Bivio::PetShop::Util
This program populates the database for our petshop demo. It builds
the entire schema, too. The test suite for the petshop will assume
this data.
The amount of data need not be large. This isn't the point of
acceptance testing imo. What you want is enough data to exercise
features such as paging, form submits, etc. Our production database
is multi-GB.
We do have a particularly nasty problem of our quote database. We
update all of our quote databases nightly using the same software
which talks to our quote provider. This tests the software in
real-time on all systems. We run our acceptance test suite in the
morning after all the nightly stuff is done. It takes hours to
re-import our quote database.
You need a test system distinct from your production and development
systems. It should be as close in configuration to the production
system as possible. It can be very cheap. Our test system consists of
a refurb Compaq Presario and a Dell 1300 with 4 disks. We use
hardware RAID on production and software RAID on test. Differences
like these don't matter.
The database source needs to be configurable. Disk is cheap.
You can have multiple users (schemata) using the same database host.
Our database abstraction allows us to specify the target database
vendor, instance, user, and password. Our command line utility
software allows us to switch instances easily, and the config module
does, too.
I often test against my development database at the same time as I
compare the same results against the test database. I can do this,
e.g.
b-petshop -db test create_db
All utilities have a '-db' argument. Alternatively, I can specify the
user in long hand for the Connection test below:
perl -w Connection.t --Bivio::Ext::DBI.database=test
All config parameters can be specified this way, or in a dynamically
selectable file.
> This has been by far the biggest obstacle for me in testing, and from
> Gunther's post it sounds like I'm not alone. If you have any ideas
> about how to make this less painful, I'd be eager to hear them.
It isn't easy. We don't write a unit test per class. Indeed we're
far from this. OTOH, we reuse heavily. For example, we don't need
to test our product list:
http://petshop.bivio.biz/src?s=Bivio::PetShop::Model::ProductList
It contains no "code", only declarations. All the SQL is generated
by the object-relational mapping layer which handles paging,
column sorting, and so on. The view is as simple:
http://petshop.bivio.biz/src?s=View.products
Neither of these modules is likely to break, so we feel confident
about not writing unit tests for them.
Rob
--------------------------------------------------------------
#!/usr/bin/perl -w
use strict;
use Bivio::Test;
use Bivio::SQL::Connection;
my($_TABLE) = 't_connection_t';
Bivio::Test->unit([
Bivio::SQL::Connection->create => [
execute => [
# Drop the table first, we don't care about the result
["drop table $_TABLE"] => undef,
],
commit => undef,
{
method => 'execute',
result_ok => \&_expect_statement,
} => [
# We expect to get a statement back.
[<<"EOF"] => [],
create table $_TABLE (
f1 numeric(8),
f2 numeric(8),
unique(f1, f2)
)
EOF
["insert into $_TABLE (f1, f2) values (1, 1)"] => [],
],
commit => undef,
execute => [
["insert into $_TABLE (f1, f2) values (1, 1)"]
=> Bivio::DieCode->DB_CONSTRAINT,
],
{
method => 'execute',
result_ok => \&_expect_one_row,
} => [
["update $_TABLE set f2 = 13 where f2 = 1"] => [],
],
execute_one_row => [
["select f2 from $_TABLE where f2 = 13"] => [[13]]
],
{
method => 'execute',
result_ok => \&_expect_one_row,
} => [
["delete from $_TABLE where f1 = 1"] => [],
],
],
]);
# sub _expect_statement(any proto, string method, array_ref params, array_ref expected, array_ref actual) : boolean
#
# Returns true if $expected->[0] is a DBI::st.
#
sub _expect_statement {
my($proto, $method, $params, $expected, $actual) = @_;
return 0 unless ref($actual) eq 'ARRAY';
my($st) = $actual->[0];
return ref($st) && UNIVERSAL::isa($st, 'DBI::st') ? 1 : 0;
}
# sub _expect_one_row(any proto, string method, array_ref params, array_ref expected, array_ref actual) : boolean
#
# Returns true if $actual->[0] is a DBI::st and we processed one row.
#
sub _expect_one_row {
my($proto, $method, $params, $expected, $actual) = @_;
return 0 unless _expect_statement(@_);
return $actual->[0]->rows == 1 ? 1 : 0;
}
===
To: modperl@apache.org
From: Rob Nagler <nagler@bivio.biz>
Subject: Re: UI Regression Testing
Date: Sat, 26 Jan 2002 10:59:05 -0700
Gunther Birznieks writes:
> From the description of your scenario, it sounds like you have a long
> product life cycle etc.
We release weekly. We release to test multiple times a day. We "code
freeze" the test system over the weekend.
We run all weekly jobs on test during the day on Sat, and then release
to production Sat Night. The job testing change was introduced
recently. On production, we have a large job which runs on Tues. It
also ran on Tues on test. We changed something later in the week one
release which broke the job, but it wasn't tested. Now, we get that
extra assurance of having the weeklies run just before the release.
Having an efficient release mechanism is critical. Also, we get paged
when something goes wrong on production. With Perl, we can and do
patch individual files midweek in critical emergencies. For example,
our ec code broke soon after our site went to a pure subscription
model. It was fun, because it broke from too many paying
customers. $-) Needless to say, we patched the system asap!
> I think your testing, especially regression testing and the amount of
> effort you put into it makes a lot of sense because your software is a
> long-term investment possibly even a product.
Yes, that's an important point. We run the accounting for over 8,000
investment clubs. We have a responsibility to make sure the software
is reliable. We released our Petshop demo without any tests. :)
> To each his own I guess.
Agreed.
===
To: Rob Nagler <nagler@bivio.biz>
From: Stas Bekman <stas@stason.org>
Subject: Re: UI Regression Testing
Date: Sun, 27 Jan 2002 12:53:48 +0800
Rob Nagler wrote:
> FWIW, we are very happy with our unit test structure. It has evolved
> over many years, and many different languages. I've appended a simple
> example, because it is quite different than most of the unit testing
> frameworks out there. It uses the XP philosophy of once and only once
> as well as test what is likely to break.
Rob, it seems like you definitely have a lot of experience with these
things.
Have you considered talking about Testing at OSC this summer? Mischael
Schwern's talk was a great success last summer.
Also writing things down as a doc explaining how things work, with some
light examples, to add to our knowledge base would be really cool!
I'm sure other folks with similar experience will help working on
perfecting the document.
===
To: modperl@apache.org
From: Rob Nagler <nagler@bivio.biz>
Subject: Re: UI Regression Testing
Date: Sat, 26 Jan 2002 22:15:01 -0700
> Have you considered talking about Testing at OSC this summer? Mischael
> Schwern's talk was a great success last summer.
Thanks for the suggestion. I'll think about it, and see what I can
do.
> Also writing things down as a doc explaining how things work, with some
> light examples, to add to our knowledge base would be really cool!
Absolutely. If there other Extreme Perl programmers out there, send
me a private email.
===