An interesting article written by Yegor Bugayenko, founder and CEO of Zerocracy, published in the last number of the Communications of the ACM, enters the neverending debate on how much testing is enough to ensure that a product can be safely commercialized.
There has to be a certain point where testers stop looking for bugs. […] Finding bugs motivates testers, and they’ll keep looking for them. At some point, you have to launch the product. But what happens, though, if you launch a product before you find all the bugs? If you do that, then won’t you launch it with bugs? Yes!
Despite all that work, all that money, all that effort, you still launched a program riddled with bugs. […]
But you have to ask yourself, How many critical bugs remain? You could have provided your team more time to complete the impossible task of finding all the bugs, but would they have found more critical bugs?
It is better to launch a product that you have confidence in than waste time and resources trying to make it perfect. […] Instead of allowing testers endless time to find errors as they tear apart the programming, give your testers a goal. Since the goal of testing is to find errors, why not make the completion criterion the detection of some predefined number of errors? This enforces the need to find bugs, but limits the total amount and draws focus toward critical bugs rather than general ones.
I find this view of testing a little bit limited and somewhat misleading, it reduces testing to look like a subset of the activities needed to commercialize a product, such as ethical hacking or penetration testing.
Testing should not be reduced to the art of discovering bugs, but should be framed in the context of Verification and Validation processes. The PMBOK guide defines them as follows:
- “Verification. The evaluation of whether or not a product, service, or system complies with a regulation, requirement, specification, or imposed condition. It is often an internal process. Contrast with validation.”
- “Validation. The assurance that a product, service, or system meets the needs of the customer and other identified stakeholders. It often involves acceptance and suitability with external customers. Contrast with verification.”
So, when testing for verification is enough? When all the regulations, requirements, specifications, or imposed conditions have been tested.
Similarly, testing for validation is enough when it has been verified that the product meets all the clearly stated needs of all the stakeholders (also know as “marketing requirements”).
Does this ensure that the product will be bug free? No. But it gives you more confidence on the quality and reliability of the product.