TESTING TOOLS INTERVIEW QUESTION PART 4

Q: How do you know when to stop testing?


A: This can be difficult to determine. Many modern software applications are so complex and run in such an interdependent environment, that complete testing can never be done. Common factors in deciding when to stop are...

· Deadlines, e.g. release deadlines, testing deadlines;

· Test cases completed with certain percentage passed;

· Test budget has been depleted;

· Coverage of code, functionality, or requirements reaches a specified point;

· Bug rate falls below a certain level; or

· Beta or alpha testing period ends.



Q: What if there isn't enough time for thorough testing?


A: Since it's rarely possible to test every possible aspect of an application, every possible combination of events, every dependency, or everything that could go wrong, risk analysis is appropriate to most software development projects. Use risk analysis to determine where testing should be focused. This requires judgment skills, common sense and experience. The checklist should include answers to the following questions:

· Which functionality is most important to the project's intended purpose?

· Which functionality is most visible to the user?

· Which functionality has the largest safety impact?

· Which functionality has the largest financial impact on users?

· Which aspects of the application are most important to the customer?

· Which aspects of the application can be tested early in the development cycle?

· Which parts of the code are most complex and thus most subject to errors?

· Which parts of the application were developed in rush or panic mode?

· Which aspects of similar/related previous projects caused problems?

· Which aspects of similar/related previous projects had large maintenance expenses?

· Which parts of the requirements and design are unclear or poorly thought out?

· What do the developers think are the highest-risk aspects of the application?

· What kinds of problems would cause the worst publicity?

· What kinds of problems would cause the most customer service complaints?

· What kinds of tests could easily cover multiple functionalities?

· Which tests will have the best high-risk-coverage to time-required ratio?



Q: What if the project isn't big enough to justify extensive testing?


A: Consider the impact of project errors, not the size of the project. However, if extensive testing is still not justified, risk analysis is again needed and the considerations listed under "What if there isn't enough time for thorough testing?" do apply. The test engineer then should do "ad hoc" testing, or write up a limited test plan based on the risk analysis.


Q: What can be done if requirements are changing continuously?


A: Work with management early on to understand how requirements might change, so that alternate test plans and strategies can be worked out in advance. It is helpful if the application's initial design allows for some adaptability, so that later changes do not require redoing the application from scratch. Additionally, try to...

· Ensure the code is well commented and well documented; this makes changes easier for the developers.

· Use rapid prototyping whenever possible; this will help customers feel sure of their requirements and minimize changes.

· In the project's initial schedule, allow for some extra time to commensurate with probable changes.

· Move new requirements to a 'Phase 2' version of an application and use the original requirements for the 'Phase 1' version.

· Negotiate to allow only easily implemented new requirements into the project; move more difficult, new requirements into future versions of the application.

· Ensure customers and management understand scheduling impacts, inherent risks and costs of significant requirements changes. Then let management or the customers decide if the changes are warranted; after all, that's their job.

· Balance the effort put into setting up automated testing with the expected effort required to redo them to deal with changes.

· Design some flexibility into automated test scripts;

· Focus initial automated testing on application aspects that are most likely to remain unchanged;

· Devote appropriate effort to risk analysis of changes, in order to minimize regression-testing needs;

· Design some flexibility into test cases; this is not easily done; the best bet is to minimize the detail in the test cases, or set up only higher-level generic-type test plans;

· Focus less on detailed test plans and test cases and more on ad-hoc testing with an understanding of the added risk this entails.





Q: What if the application has functionality that wasn't in the requirements?


A: It may take serious effort to determine if an application has significant unexpected or hidden functionality, which it would indicate deeper problems in the software development process. If the functionality isn't necessary to the purpose of the application, it should be removed, as it may have unknown impacts or dependencies that were not taken into account by the designer or the customer.
If not removed, design information will be needed to determine added testing needs or regression testing needs. Management should be made aware of any significant added risks as a result of the unexpected functionality. If the functionality only affects areas, such as minor improvements in the user interface, it may not be a significant risk.

Q: How can software QA processes be implemented without stifling productivity?


A: Implement QA processes slowly over time. Use consensus to reach agreement on processes and adjust and experiment as an organization grows and matures. Productivity will be improved instead of stifled. Problem prevention will lessen the need for problem detection. Panics and burnout will decrease and there will be improved focus and less wasted effort. At the same time, attempts should be made to keep processes simple and efficient, minimize paperwork, promote computer-based processes and automated tracking and reporting, minimize time required in meetings and promote training as part of the QA process. However, no one, especially talented technical types, like bureaucracy and in the short run things may slow down a bit. A typical scenario would be that more days of planning and development will be needed, but less time will be required for late-night bug fixing and calming of irate customers.