Quality Assurance should be an integral part of the development culture. Relegating the Quality Assurance effort to an after-the-fact testing role reduces the value of what it can contribute, and yet many organizations refer to QA only in terms of testing at the end of the project. If you do not put quality into a project as a result of how you go about developing a new product you cannot hope to add that quality by doing some testing at the end of the development life cycle. With that "testing only" focus you often find that many deliverables are poorly documented, which complicates testing even more. If you do not have solid requirements and functional specifications there is very little to base your test scripts on to determine if the product works as expected. At best you can ask yourself if it looks good and if it feels good, perhaps even if it seems to do the sort of thing a user might want to use it for, but unless things are clearly defined that kind of testing can only lead to an individual's opinion based on personal expectations.

The role of testing is to provide a last line of defense against a system that is inadequate with respect to expectations of what it is supposed to do. Clearly, if this is the first time you find out that the system is inadequate, then a lot of development expenditures are in the balance and the costs of changing and fixing things may be a serious blow to the development organization. This is not the sort of condition that provides an incentive for people to have an open mind about what to do next. Even if the problems are irrefutable there will be pushback against introducing the necessary fixes that will delay the release of the product despite the obvious logic that a premature release could lead to a loss of credibility. That is why we recommend the integration of QA work activities with the full development life cycle rather than to associate the role of QA with the final stage of the development effort. This is explained on the previous web page.

The testing effort itself is an important part of the process to prevent the release of inadequate software, but it is only the culmination of a fully integrated effort that allows the testing to be focused and precise. The earlier test scripts are defined, the sooner the testing analysts can identify conflicting specifications or undefined aspects that may well corrupt the way in which the software is written. Even before then, the systems analyst who has to write these specifications sufficiently precise to make testing possible will probably realize what is missing and fix any shortcomings without being prodded to do so. This illustrates what is difficult about determining the value of QA, because it reflects a continuous improvement of the way we work as opposed to a specific series of QA actions that are clearly discernable throughout the development life cycle. Let's look at some examples:

Testing Analysts must have a head for automation...

Not only does it make sense to understand automation of the Testing Process, it makes sense for the QA Test Analyst to be an expert in automating any process in order to devise the appropriate test conditions and scripts. They must have an eye for detail in order to identify all the combinations of conditions that must be tested. They must understand how to work with an application in order to test all aspects of how the system can be used.

Testing Analysts are not the rejects from development. QA Testing can sometimes be a thankless job when people think that testers have little to contribute to a project when the opposite tends to be true. It generally takes a completely different mindset to be able to distance yourself from the obvious and to discover what is possibly wrong with an application. The point is not to be critical of the developer but to make sure that all aspects of the specifications have been rendered as requested. It is difficult for developers to wear two hats. If they could see that there are gaps in the implemented code those gaps would not be there in the first place. QA Testing Analysts reconcile requirements with what they can observe to make sure the application works exactly as specified.

Testing may require development of supporting applications to write files, create database contents, extract results, or even to compute the expected results, depending on the level of expertise in your testing team. You just do not want the developers involved in the application to also be involved in the testing development as any errors of interpretation would then be replicated in the testing tools. This would defeat the purpose of these testing tools as a means to validate the correctness of the application. Sure, the testing tools could be wrong, but the chances of two different teams making the same (or a complementary) mistake are much less than when a single team develops both the application as well as the testing tools. If the independent tools do not properly reconcile you can focus your analysis to find out which one is right and which one is wrong. Testing differences are just that. Testing errors are conditions where the same error is made in testing as in development so that no differences are found when in fact the application is flawed. That is what we want to avoid.

Even if QA testers do not develop custom tools to perform their tests they have to be well versed in what automation does. They have to be comfortable with using development and debugging tools, to dig into the underlying probable cause of any identified failures. That again is mostly an analyst skills set that has to be recognized, especially if the testing is performed without a carefully designed test script. Testers must have sufficient skills in using general development debugging tools and data management tools, so that they are able to at least view the files even if they do not create the files. They have to understand the consequences of what may be happening in the processing environment that could possibly affect the application so that they can know what to look for if things do not quite work as expected.

Testers need to be good diagnosticians...

At the most basic level of exploratory testing you need to have a good feel for what to look for in order that you can identify any flaws in an application. We do not really recommend this mode of testing, because it does not provide any assurances that you have explored all aspects worth testing, nor does it establish explicit test observations that can establish success or failure with absolute certainty.

Exploratory testing is an art. Compared to scripted testing, everything is left to the real-time perception of an application as the tester puts the application through its paces. The tester must keep track of all the possible branches within that process to make sure all parts of the application are being tested, not just a repetition of the same processes. Typically only the more experienced testers are good at exploratory testing as they may have developed the right skills to explore all aspects of an application. They recognize when a "standard" drop down menu contains "non-standard" menu options, for example. They know how to exercise a date field, to test amount fields with limit checks, and so on, without being prompted to do so by following a detailed predefined test script.

One of the challenges is to keep detailed records of where you have been and what has been tested in order to make some assessment of how well the application has been tested. Some people claim that by looking at the relative frequency of errors you can determine how complete the testing is, but we find that difficult to support. It is not as if you have a list of things to be tested, as for scripted testing, so that you can tally what portion of the test scripts has been completed. Even such basic feedback as where you stand with testing is more conjecture than an objective status report. That does not mean that there is no room for exploratory testing, it means that you must have a good reason to engage in testing without first writing the proper test scripts.

Among the more plausable reasons is the testing of potential conflicts with other applications, where you have no design specifications that tell you what specific conflicts the developer should avoid. In fact, the developer is generally unaware of any such conflicts, assuming instead that the application will work as expected regardless of what other applications may be running. Since you cannot prepare specific scripts you must be experienced to notice suspect behaviour and to dig deeper to find possible hidden issues that interfere with the application. For most software development, however, we prefer to develop scripts that itemize all potential test conditions.

Many aspects of testing require manual script execution...

No tool is going to comment on the presentation of information to determine ease of use, only humans can make such observations. Manual testing at least confirms the correct execution of each script before you can even consider automating the testing process, but it can add a lot more value in terms of confirming that several different OS/Browser combinations can faithfully reproduce each application window.

Scripted testing is far more precise than exploratory testing. Test scripts are often equivalent to program statements, except that they look at a process from the outside in rather than from inside the application. Test scripts tend to be based on the same requirements and specifications, in order to validate that the code as implemented exhibits the behaviour that is expected according to the design. It the expected results are not encountered the Testing Analyst may have to dig deeper to get anwers that enables them to record a precise problem report to go back to the developer. Unless the Testing Analyst is familiar with the development process the nature of that communication may be less than what is required to give the developer a heads-up to work on the problem.

One of the major benefits of scripted testing is that you can reconstruct the exact circumstances that caused an error condition in the first place. The conditions are all in the script, right down to instructions on what data must be collected in the case of a script failure. Typically there are formal problem reporting procedures and forms to ensure that problems are tracked after they are reported, and that any fixes will trigger the repeat of appropriate test scripts. Developers prefer to reconstruct the conditions in a development environment to research exactly what caused an application to behave in unsuspected ways, something that is made much easier when each test is fully scripted and where the testers can log the reference to the script accordingly.

Test scripts are a valuable resource. We recommend the use of an automated repository even if you are not yet ready to automate the script execution. It is important that you have easy tools to keep the scripts current, that you can find all scripts that are affected by a common change to anything related to the testing. This allows you to update the test scripts at once, rather than to allow the test scripts to become outdated. Too much testing takes place without formal tools and without a methodology that ensures your testing will cover all the important parts of the application. Any time you need to regression test you should be able to perform a previous set of scripts without having to worry if those scripts are still relevant. Maintenance should be cheap yet effective so that testing does not become the major cost item of the development effort. Mind you, if testing is more expensive all that means is that the scope of the changes probably does not warrant a major effort.

Regression testing is the most obvious application of scripted testing: you should be able to repeat any script at will at any time in the future. Because you should be able to repeat the execution of these scripts at will, they are also an excellent candidate for future automation. The fewer the changes that are required from test run to test run, the better the regression scripts will work in an automated regression execution context. However, we are well aware that not all testing lends itself to automation.

Automating the Test Script Execution...

Ultimately the goal is to have as much of the test script execution automated as possible, and to find useful tools to deal with aspects of testing that you cannot automate. We must improve the quality of testing before we increase the volume of test scripts to exercise all aspects of the application software.

Automated testing takes repetition out of the equation. The premise is that your testing will already be proven correct through manual testing, since automation should replicate the exact results you can get through manual testing. If the underlying conditions are unstable then test automation is only an exercise in frustration and there would be little to gain from automation. Establishing the appropriate discipline to ensure consistency in the testing process can pay off handsomely in the form of you being able to replace manual test execution with automated test execution.

One of the great benefits from test automation is that you can afford to run all test scripts against an application, whereas the resource costs for manual testing tends to encourage people to focus on a subset of the scripts as sufficient to find all but the more obscure errors. With the proper tools the automated scripts will run significantly faster than what manual testing can achieve, but the potential savings can often better be directed towards more complete testing that may flush out additional errors. We know that manual testing is highly labour intensive, but if you use automation to flush out those potential problems that you want to have reviewed more closely then the focused use of QA Testing Analysists to repeat failures with manual tests and associated research into the root causes of confirmed failures will be a much more cost efficient use of testing resources.

We have taken companies from disorganized manual testing to highly organized automated testing in the span of less than 2 years, providing that the staffing in the QA Testing organization is such that people can be trained to use the more advanced tools. This is very aggressive, especially if at the beginning the testers resemble a typing pool more than a development team, but you do not want to dwell too long in the intermediate stage that is characterized as manual scripts in a sophisticated on-line script repository. This is a necessary transition stage to make sure you preserve your old scripts, and to provide a reference base on which you can build your new automated execution scripts. Ultimately the goal for test automation is to let the system run by itself and then to collect and review errors that have been reported, so that the QA Testing Analysts can focus on reported errors rather than to get mired in stacks of valid output.

Test execution is the final verification that the application works (or not) in order to decide on the rollout. Based on independent testing you can protect yourself against a rollout of a flawed software product. The importance of QA Testing is only increasing as companies consider outsourcing software development to offshore companies and the opportunities for miscommunication grow as a result. You need that final guard to ensure that what you get is what you want, and only an independent testing process can give you an unbiased assessment of what you actually received from the development organization.