The word testing means different things to different people. In many cases, testing is just a means to an end. When we are talking about testing in Perl, we are actually talking more about a development philosophy.

There is nothing wrong with ad hoc testing, unless it is the only form of testing you use. This approach (obviously) requires a lot of person-hours to perform. It is also not really suitable for use in verifying code repeatedly. It is very repetitious, which generally means error-prone when executed by humans.

Most of the testing we want to talk about is automated. When the procedure is to run the same test day after day, you really want to hand the test off to a computer. This makes the test more reproducable and, quite frankly, the computer does not get bored and wander off topic.

Each stakeholder has different things she wants from any testing. Good automated testing can actually provide all of it, with less expense that you assume.

Like many things in Perl, thre are many different frameworks that provide different approaches to solving the problem. Since our time is short, I won't focus on these.

We will focus on the most widely used and standardized system in the Perl ecosystem: TAP.

This is a very simple example of the output from a TAP-based test. It shows all of the major parts of a TAP test.

• Marker indicating the number of tests to run
• A series of test assertions.
  • Success indicator: ok or not ok
  • Assertion number
  • Assertion name/label
• Diagnostic information preceded by a #

The standard Test::More module provides a more complete set of assertion functions. These don't really do much more than the straight ok() function (which is also provided) if the assertion succeeds. But, if the assertion fails, they provide more useful diagnostic output.

These are the major test assertion functions provided by Test::More. The module does provide other functionality, but we will not be exploring that in an introduction.

You are not limited to Test::More. There are a large number of specialized modules that provide other assertions and extend functionality. All of these still output TAP, so they are compatible with any tools that would recognize that output.

You still might ask why you should test. Technical solutions are great, but why should you apply them?

Much has been written on the inability of testing to find all of the bugs in software. Likewise, testing cannot prove that the code is correct and bug-free. But, testing does provide a link in a chain of evidence.

And, not testing provides even less evidence of correctness.

I have worked in environments where the documentation group was able to answer a lot of their own questions from well-written tests. That meant that they only needed to interrupt the development team for the questions that really mattered. This is a better use of time for both teams.

Nothing makes you appreciate the awkwardness of your interface like forcing you to use it.

If you add a unit test for each bug that you find, the bug cannot reappear due to later changes in the code without triggering the test.

The old Extreme Programming methodology used the first approach to decide what should be tested.

Most people remember to test the obvious success case. It is equally important to look for places on the edges of good behavior or changes in behavior and test around those areas as well. Edges are a rich source of bugs.

Fewer developers remember to test the failure cases. Do you know how the code acts if it is passed a bad parameter? How about if a file it depends on does not exist? How about permission failures? Testing these areas makes certain code is in place to handle the sorts of surprises that happen in the real world. These corners of code are often never tested except by unit tests. So the first time they may execute is in production.

Almost nobody remembers to test their assumptions. These assertions almost never fail. When they do, they tend to generate very strange behavior that may take days to unravel.

The supplied code is not production. List.pm is an example that is complex enough to have reasonable tests and simple enough that you won't be distracted by the problem being solved. See List::Util for the real solution to this problem.

The code is pretty straight-forward. The first test file shows some minimal testing that someone might apply. The final test is somewhat more complete.

The images in the slides are from the Open Clip Art Library.

The confidence given by your unit tests can only be as good as your coverage. Using Devel::Cover you can measure the degree of confidence you should have in your unit tests.

In a real system, different modules will have different levels of coverage and, therefore, different levels of confidence.

All Test::More-based test files are Perl scripts that can be executed directly.

Using the prove command supports running multiple test scripts at a time. More importantly, prove summarizes the tests in a much nicer fashion.

Sometimes you have test assertions that you do not want to run in some circumstances. For example, if you have a multi-platform module, the MS Windows-specific tests don't need to run under Mac OS X. Test::More provides a nice mechanism for skipping tests, with a message telling why and still running in a way that prove gives a good summary.

TODO tests are great for adding a test for something that you know needs to be done, but that you cannot fix right this moment. You could leave a failing test, but having your tests always successful is a really good habit. The TODO mechanism allows the test to fail, but reports it as a success for the test run. The message reminds you what is happening so you don't forget. Just as importantly, prove gives a special indication when a TODO test starts passing unexpectedly.

On rare occasions, you can find yourself in a state where no further testing can happen. (Hardware to test is missing, no hard drive, etc.) Rather than fail all of the tests one at a time, Test::More provides an ability to stop a full test run by Bailing out.

To answer a great question from the talk. UI testing was not in the scope of the talk. But, WWW::Selenium would be a good place to look tools to solve this problem.

There are many kinds of testing. The list above gives some terms that are worth learning about. All but the last two are automatable.