We have had a long-standing interest in streamlining the software testing process. The simple fact is that you spend more energy and effort on testing than on all other parts of the development effort, yet the resources for most testing efforts are squeezed to minimal levels. This has resulted in many software applications being installed before the bugs have been sufficiently fixed. In fact, many people have come to accept that the first few releases of a new software application will be less than excellent until users have found and reported bugs that then are fixed in later releases. It does not have to be this way, and as software applications become more and more central to the way we run business operations it should become apparent also that failures are not an acceptable option.

Our experience has shown a dismal level of test coverage of perhaps 5% - 10% of what should be tested in a typical business application. It is not a matter of negligence: manual testing typically takes between 2 and 5 minutes per transaction to enter, observe, record results, report errors if necessary, re-test, and so on. Usually applications are developed under severe time constraints. People are under pressure to deliver, and this leads to cutting corners when it comes to testing because any delays can only be compensated for by reducing the time taken for testing. It is a simple exercise in mathematics that even with a team of testers you can only expect to get a limited amount of testing done per day, perhaps 70-80 transactions per tester in a demanding GUI environment. To test the boundary conditions for a single data element takes at least 4 transactions, more if you deal with multiple ranges, so the number of transactions to test all data elements properly quickly exceeds the time available.

To strike a balance between the need to test an application and the need to reduce the time and cost required for testing you have to consider ways to automate the testing effort, and begin a journey of implementing test automation.

You have to be careful that in your quest for improvement you do not overcharge the ability of people to adapt to new ways of doing things. The concept of going from a manual test environment to an automated test environment may be simple enough, but actually doing just that takes a little more time and know-how. People who are able to produce results doing a job in a certain way are naturally going to be concerned that change might affect their success rate, no matter how desirable change may seem to you.

You should always start with a simple goal, such as to provide a tool to support the existing manual testing process, that paves the way for more challenging objectives. Simple goals are not necessarily without any challenge. There is always the appearance of an overnight success that does not account for months of prior preparation to make that success possible. For a large Financial Institution we converted 5,500 Excel-based scripts into 65,000 task steps for the Certify script repository. If those scripts had not been consistent with our basic format then it would have taken considerably longer than a month to convert those scripts, and the client might even have been discouraged from taking the first step towards automation.

Placing scripts in a repository is of itself not test automation of course. When we noted that the effort depends on the right tools we had to consider the initial manual execution as much as the future automated execution, and there are not many systems that can help a client through that transition. For example, the fact that Excel is used by many companies as a pseudo script repository does not of itself provide a gateway to test automation, but it provided a starting point in this case to convert the scripts to the right format.

The conversion of scripts from an Excel based format to a repository format may seem to be a small step, but it is often seen as a major barrier that so many automated testing tools do not support the existing investment in test scripts. Clearly, it takes some time to convert an existing script base to full automation so you have to support manual testing while you work towards the bigger and better goals that may require a rethinking of the scripts.

You will want to make sure that the intermediate stages provide direct benefits to the testing process, so you do not want to move to an intermediate product and then introduce an automated testing product. For one, there are always specific tests that you need to perform manually, and in some cases automation is used for setup tasks and post-testing audit and cleanup tasks while the tester observes the user-friendly aspects of the GUI at the focus of the testing effort. In our case we used Worksoft Certify as the selected automated testing product since it can support manual testing, mixed testing, and automated testing in one product.

Once you have all your scripts in a common repository you will be in a much better position to maintain the test scripts, and there are many scheduling and tracking services included in the test automation software that work well with manual testing. Your database will be better secured and backed up as compared to your Excel based scripts, so this first step is a major step to protect your investment in test scripts. If your existing scripts are proven to be effective at finding potential problems then this is a good foundation for future automation. As a first step the manual scripts repository can take some of the pressure off the effort to develop automated scripts to give you more time to establish automated scripts.

You should not underestimate the effort even of converting your manual scripts if they were written without ever thinking of moving towards automation. Even a manual script needs to spell out the individual steps required to perform sometimes complex testing tasks. If your scripts are written at a high level for use by a tester with more or less in-depth product knowledge then conversion is no small feat. In that case you will have to make an effort to translate a general description into specific task steps that can be performed by testers with no prior product knowledge (typical of a dedicated general purpose testing team).

If you have consistent scripts the conversion effort is a much less difficult task to complete. That is why we emphasize a well-structured script format regardless of the tools used to maintain the scripts. Some conversions are easier than others, depending on what tools you use for your existing script base. A general-purpose Excel format is fairly easy to convert. A custom database system can be a bit of a challenge and be dependent on specialized technical support.

As part of our training curriculum we emphasize the use of structured scripts in Excel with a Word document to explain the nature of the scripts. If you convert these scripts you only convert the processing steps contained in the Excel format: the explanation in the Word document becomes supplementary information to the repository. Ideally you should be able to store the documents with a link in the repository so that any questions about the test script can be answered by the supporting documentation. The processing steps can easily be extracted and converted from the Excel based source to a repository compatible import file.

We have the tools to expedite the test script conversion and to then import the results into a Worksoft Certify test script repository. At least the repository will support the maintenance of test scripts, such as to adapt to new data (test accounts, etc.) as required. It is much easier to find the necessary things to change and to get organized to streamline manual test execution. Those tests that are repeatedly used to verify the proper execution of certain applications (regression tests) should be the first in line for further automation efforts, but for some testing we may always use a mix of automated and manual execution. The goal of automating the regression testing is to lower the cost of that testing and to support testing on demand.

One of the major challenges with manual testing is that you have to provide trained testers on demand when any changes are made to the application. In practice this only discourages testing, because meeting that need will only drive up the cost of maintaining a testing capability. You can go outside to have your testing done, but there is still a need to maintain the inventory of test scripts so that you maintain long-term benefits, and the question is if you believe it to be to your advantage to let outsiders maintain your testing capability.

We recommend an organized approach to building your testing capability, and the conversion from your existing scripts to a formal script repository format is an essential step in that process. It provides you with a benchmark that you can compare future automated scripts against, and it provides consistency in case the target software GUI changes to such an extent that the automation tool cannot recognize the handshaking needed to communicate with the application. Some tools give up more easily than other tools. One of the reasons that we like Worksoft Certify is that it is capable of analyzing what has changed and of adapting the script to the new GUI definitions. Anyone who has experienced the agony of having to sift through hundreds of test commands after a series of cosmetic changes restructured the GUI interface of an application will know what we mean when we call this a real time-saver.

While streamlining the manual testing effort will reduce costs, it does not give you dramatic savings compared to ad-hoc testing. What it will give you is a better organized testing process that should be better able to achieve in-depth test coverage. Most repository tools have ways of relating the scripts to requirements so that you can cross-reference what you are testing with what the application is supposed to do. This provides more value for your testing investment.

In our examples we used the Worksoft Certify software product to demonstrate how you can accomplish your goals.

There are other software products that provide similar services, and we are happy to work with those products, but we have had particular success with Certify because it provides a full range of services to support testing from a completely manual paradigm to a completely automated paradigm. This makes it an ideal tool to get started and to keep growing until you have automated as much as you can, while at the same time you are not forced to look towards other means to maintain those tests that need to be executed manually. Automation is made easy with Certify, which helps you to move forward even after you have converted your existing scripts to a central repository. This was an important consideration for us, so that we would not consolidate and convert all your existing scripts into a new repository that then in turn would become an impediment to further automation.

You cannot get there without a well organized methodology. On the next page we explain part 2 of our front page "infomercial" that deals with the use of a methodology, and how the proper tools help you achieve your objectives. Our methodology is consistent with what Worksoft wants to accomplish with Certify, and it is based on the collective experience of many testing experts that have willingly shared their views on the Internet. This site is in part designed to similarly share our ideas with that collective to help with pushing towards better and more reliable testing techniques. If you visit Worksoft (click on the logo above) you will find many white papers on testing that provide a useful insight into how you can improve your efforts.