Beautiful Testing - Beautiful Mutation Testing - chapter 18

In mutation testing what we are doing is; using a slightly and systematically changed version of the program which has defects, and test against our test suit to find whether our test suite can find them.

For an example consider the bellow example code.

if (x||y) {
  z = 1;
}
else{
  z = 0;
}

We can mutate the above code by changing the logical operator as bellows.

if (x&&y) {
  z = 1;
}
else{
  z = 0;
}

Evaluating test case quality

There are several metrices for evaluating test case quality.

1. Code coverage
All the codes are needed to be executed by the test suit without bypassing any.
2. Condition coverage
All the condition statements are needed executed by the test suit

The AspectJ compiler is an extention to java that can be used for testing.
In this chapter it is used Aspectj to view the bug distribution along different classes of a project. It is then obvious that the bud distribution is uneven.
Pareto effect : When the distribution of bugs are uneven then it is called the pareto effect.

According  to the book mostly the system tests are practically carried out through Junit which is used for unit testing in java.

The book has used the term "watching the watchmen" metaphorically to figure out that the 'quality of quality assurance' should be tested.

Mutations : These are the artificial defects introduced to a program. ex: such as changing a constant.
If the test suite is able to find it then, we can be sure that it could find the real bugs.

Benefits of Mutation Testing

1. Can evaluate the test suite's capability
2. Can identify modules with high risk

Some mutation frameworks available for every one;

1. Mothra - For Fortrun
2. uJava - For Java

Drawbacks of Mutation Testing

1. Very time consuming

Tips to make mutation testing efficient;

1. Mutate the binary code so that rebuilding is avoided
2. Use mutant scemata
3. Ignore uncovered code

Adding a mutation;

1. Replace numerical constant
2. Negate jump conditions
3. Replace arithmetic operations
4. Removing method calls

Equivalent mutations: Some operators has the capability to create a similar program like the previous program even after adding a mutation. This results in an equivalent mutation.

Javalanche Framework

1. This is used for mutation testing of Java programs.
2. It is a free for everyone framework.
3. And the other thing is that, it can be extended with additional operators or impact detectors. 

Intersting Facts on Beautifull Testing chapter 2

In this chapter, it is shown the profound beauty of testing rather than it's superficial beauty.

The stake holders(interested parties) of testing are the people who are needed be satisfied. They are of two types.

1. Internal stake holders - doing, leading and managing test work
2. External stake holders - other interested parties

Examples for some of the stake holders can be mentioned as follows;

Fellow testers, test leads and managers, developers, development leads and managers, database and system architect, marketing and business analysts, project managers, tech support and help desk staff, vendors, customers,users etc.

Stake holders can be divided in some other groups depending on their attitude and contribution towards a project. Namely;

1. Nutral Stake holders - like legal bodies
2. Anti Stakeholders - like enemies who wish the unsuccessfulness of the project

Each stakeholder has a set of objectives and expectations  to be satisfied through testing. They need them to be efficiently, effectively and elegantly be carried out.

• Effectiveness - Satisfy the objectives.
• Efficiency - Satisfy the objectives and expectations in away that it maximizes the value.
• Elegance - Achieving efficiency and effectiveness gracefully and nicely.

For some clients the testers measure their effectiveness through the severity of the bugs they find.

External Beauty

DDP (Deffect Detection Percentage) = Bugs Detected/ Bugs Present

DDP (DDP before UAT) = Bugs Detected/ Test Bugs + Production Bugs


A test team’s DDP for a system test or system integration test averages around 85%. But if it is for internal use then DDP should be around 95%. If it is a safety/mission critical application then the DDP should be higher.

In testing, cost of quality refers to the recognized technique for measuring the cost of failures.

Cost of Detection - The testing costs on us even we couldn't find bugs.
Cost of Internal Failure - Testing and development costs such as filing bug reports, fixing bugs as a result of finding bugs.
Cost of External Failure - This includes support, testing and development costs that we have to bear because of a faulty system.

Average Cost of a Test Bug = Cost of Detection + Cost of Internal Failure/ test bugs

Average Cost of Production Bug = Cost of External Failure/Production bugs

Test ROI = (ACPB-ACTB) *test bugs/cost of detection







 

Interesting Facts on Beautiful Testing Book - Chapter 1

Interesting Facts on Beautiful Testing Book - Chapter 1

The 1st chapter of Beautiful Testing E book opens up with an incident an organization has undergone which had been able to embarrass,frighten and amuse different sets of employees in the same organization. 

The people who are amused by this incident are the testers. The testers are a different set of people who own  a different set of attitudes and they are completely different from the other IT staff of an organization.

Their amusement should not be criticized because they do it merely to forget the unpleasant situation. 

The testers are there in a team to find and reveal problems in a system  eliminating the risks to an organization, developers and customers.

Testers are not policemen, instead they needed to be treated as advisers  . Testers have the difficulty in managing risks with cost, marketing requirements and needs. Evaluating and accepting risks is a project for the testers. Another characteristic of the testers is that they are very reluctant to admit the system is 100% error free. 

According to the book the testers differ from other IT staff because they are paid for telling the truth. Its like telling some one that their baby is ugly and reasoning why.

Some qualities of the testers can be mentioned as follows. The testers are ;
curious, experimental, fearless, intelligent, analytical,learns fast and apolitical.

There are some bad qualities which are rooted with a tester. Such as; they are distrustful, critical about every thing, question every thing.

The people who are testing the others test cases just to complete the task can not be called testers. A good tester testes, update and edit others test cases if there are any drawbacks found instead of just ticking the list of test cases when  testing is over. 

Testers highly expect the developing team to fix the errors they find.
Another thing is that the testers figure out the errors using nice words to the team.

Finding a bug to a tester means finding a treasure to them.

The preparation for bug finding can be stepped as bellows;

1. Read all the required and related books.
2. Ask questions till they understand.
3. Prepare a plan

Encountering bugs depends on the learning and motoring rather than the intelligence lies with in them.

Project managers also get involved in testing because then it is easy to get to know the things that a tester is testing and next time can avoid that bug.

Some testers might not report all the bugs they find, thinking that they are never fixed.This is not a good practice for the testers.

The testers find bugs when compare the ability of the programmers, as they try to prove the system is not working properly, rather than proving it is working properly. They really like to find the ways which fails and breaks the system.

The testers needed to be appreciated for their good work. Otherwise the best will leave searching better opportunities.