" Bug Management " - Can we take it for granted?

 An insight to Beautiful Testing Book - Chapter 6

Chapter 6 for beautiful process starts with a bizarre situation. One of the Harvard University's engineering note books has a bug cello taped to it and have a interesting story of the first actual bug. 

Although the note is very very old and found in a museum, the bug report contains two important details that need to be there in every bug report. They are as bellows;

1. Mentioning the root cause to the bug report.
2. Associating the bug closely with the test procedure.

The reason for demanding the attachment of  root cause to the bug report has a very simple consideration.  That is for the others to benefit from it. So when there are different implementations from a single product done by different set of people then they can eliminate the common bug and improve their implementations if they are known to the root causes of the bugs their product has.

Most people do not consider bugs as important. But there are situations where people's lives rely on the correct operation of software. For an example the wrong behavior of a scanning machine or a aeroplane may lead to disastrous situations. Therefore it is important to consider defects of software seriously rather than superficially laughing at the words "bug management". That is why bug/defect management is so important.

The book says that the very initial thing to do in defect management is identifying the defect. For that it is important to consider some vital aspects of a defect. Some of those aspects are; Who,What, When and Where.

1. Who

A good defect tracking system should tack some important details, such as who logged the defect, who fixed the defect and who verified that the defects have been fixed. Ultimately the defect is considered.

2. What

A bug report should consist of the bug's characteristics and also the conditions for the bug to re occur another time. So that the fixing, verification and also the elimination of the bug from newer releases are made easy.

A bug report should essentially posses following details.

• Description - A summary of the bug.
• Detailed Description - Sufficient details to re occur and see the bug which is reported.
• Priority - Importance of the bug.
• Severity - The impact of the defect.

Priority and Severity..... Confused?

For an example lets say that we have developed a web application for a customer. And it's some of the content could not be properly displayed on the customer's standard web browser. Then the bug is of high priority. Lets say the same content is not displayed in an archaic version of the web browser that they are using then we consider the bug is of low severity.

3. When? 

Defect might found at different stages of the software development life cycle. Some bugs may found at Quality Assurance testing and some may found after releasing the product to the customer. If the bug is found at the latter stage then the cost of fixing the bug is very high. So that it is necessary to identify and fix the bugs at the earlier stages.

4. Where?

Some times the bug may found at the public open source defect tracking system. Then there will be a problem for the developers or Quality Assurance to find the code base that the defect lies in.

For a developer it is very easy to find the source code where defect lies in, if they are known about a defect. But for an end user or a support person always the defect they found is existing in the product or the distribution. Because they are unaware of the source code of the product. They can not relate the defect they  found of the product to the actual code base. So that the communication of the defects will not flow very effectively. This is a very big disadvantage.

So that there exist a good solution for this problem. If mentioned in simple terms, we can use the  Original defect repository and extract some lesser information about a bug and place it in a local mete database. And create a key which links the local meta database to the original defect database. So that what happens is the public community can interact with the local meta bug database and apply tags to the bugs and they are tracked. Similarly the company Quality Assurance team can interact with the original defects database and this will provide an effective bug communication methodology. The bellow diagram will provide some insight to defect information communication.

XMPP Testing

Before proceeding to XMPP testing, it is better to get an idea about XMPP.
eXtensible Messaging and Presence Protocol is a;

• Open source
• XML based networking protocol
• Used for real time communication

XMPP is applied in following ares.

• Instant Messaging
• Social Networking and Gaming
• Multimedia (audio and video)
• Micro Blogging

How XMPP works,

• XMPP network consists of servers and clients 
• The job of XMPP is to route small packets (stanzas) of XML between entities in the network.
• Stanzas are of three types ;

                   Message Stanzas - For sending messages

                   Presence Stanzas - For Broadcasting

                   Info/ Query Stanzas - For inquiring. 

• There is an ID called JabberID which is given for each entity in the XMPP network.

Testing XMPP

In testing XMPP what we are doing is testing the functionality of XMPP protocol. Some of the concerns  in functionality testing are as follows;

• Testing whether the client can respond correctly to incoming requests.
• Does client the send right request at the right time?
• Can client handle a specific response on a request?

Testing Request Response Protocols

Unit testing is chosen as the most suitable type for testing the XMPP functionality. Because XMPP's request response rules are local to one class.

When a client creates an XML stanza then it is constructed a Structured document through DOM (Document Object Model) then it is serialized to a textual XML and sent through the network. 

As there can happen slightly varied serialization versions and if the serialized packets are analysed, then there is a possibility for this variations in serialization to break the tests. 

Therefore response is verified by checking the DOM form rather than the serialized textual XML.

An Introduction to MongoDB

MongoDB is a Data Base Management System for web applications and Internet Infrastructure.

It is a scalable, high performance, open source NoSQL database written in C++.

It is a very attractive database. Because of it's intuitive data model.

It also has the capability to represent rich hierarchical data structures. as it has a document data model instead of a relational data model.  
One other instance for a DB using a Document Data Model can be mentioned as Apache's CouchDB.

The disadvantages of relational data bases are not a problem to document based databases. Some of the disadvantages of relational databases can be mentioned as follows;

• Lots of complicated multi-table joins are there in relational data bases. 
• Need to write complicated queries with lot of joins to retrieve data.

If we use a database which has a document data model such as MongoDB, then we have the capability to work with data base objects as a whole rather than working with lots of different tables with joins to each other.

For an example lets say we maintain a "Plantations" database.

In a relational database we have to deal with fertilizers for each plant at each stage, climatic conditions, structural configurations, images, suppliers etc. Those will be different tables in our relational database. But if we deal with a document based database, then we'll have the ability to deal with different plantation objects as a whole rather than considering each and every table separately.

As today's developers are mostly using object oriented languages for developments then it will be very effective to use Object Oriented Databases, so that their objects can be directly mapped to their databases.

The other feature we can have from MongoDB is adhoc querying. Although relational DBs support this, not all DBs support this. In adhoc querying  it is not necessary to define in advance about the types of queries the system will accept.

More over MongoDB supports secondary indexes. Secondary indexing is something like having some piece of information for indexing other than a primary key. For an example lets take a "Student" database and we need to search students using their last name. So that the Student_ID here is the primary key and the last_name will be a secondary key for indexing. B tree is the structure for MongoDB's secondary indexes.

The other key features of MongoDB can be mentioned as Replication support, support for speed and durability. MongoDBs have journals which being defaultly enabled, so that the data files can be recovered.

MongoDB supports scalability. Unlike vertical scaling done by single node expansion through hardware, it supports horizontal scaling where it provides the capability to distribute the database across multiple machines.

There are some advantages of scaling horizontally.Some of them can be mentioned as bellows;

1. Ability to get the use of commodity hardware.

2. Cost for hosting total data set can be reduced.

3. Reduce the consequences of failure.    

MongoDB has a range based partitioning method. It is called Auto Sharding. In here what happens is that the data is automatically distributed among the network nodes. 

MongoDB's core server and tools can operate on Mac OSX, Windows and Linux OSs. It's Java Script shell can be used to administer the database and manipulate data.

So ultimately MogoDB can be selected over other data bases due to its high scalability and fastness.

    

  

Surprizing Facts in "How Google Test Software"

There are quite surprising facts in "How Google Test Software". Some of the surprising and need some thought on type of facts are related to the 'Fatal Flaws in Google’s Process".

What are the things that google find fatal and others don't in their process and why they find them fatal?

• The first fatal flow that none of us finds fatal when we look through traditional testing is that if we ask a developer what he is doing about quality, then his answer often is “testing”. 

But book says that quality is not lying in testing. It has to be embedded to the product it self. As the developers are free to think that the tests are going to be done by testers and should not be considered as a burden to them, and then what happens is that they start to reduce their testing. This should be avoided.

• The second fatal flaw is that the developers and testers are separated and they are walled by different organizations at Google. Which is sometimes a prominent feature in most of the other organizations.

Mostly the testers who work at google are identified with their job and not the product that they are working on. But it is said that, a sign of a healthy organization is when the employees say, that they are working on a particular product, rather than telling their designation. Development and testing should not be separated  If it happens then a role-based association is created and testers find it really difficult to assign themselves to a particular product.

• The third fatal flaw is that, the testers often embrace test artifacts than the software or the product they are creating.

Counting the number of bug reports by a test engineer and being happy over them is not the thing to be in testing. It is like something focusing more on process than the product. The things which are being done need to be directly associated with a value.

• The fourth fatal flow is that almost always the users find problems after releasing the products, even though there are lots of tests had been carried out before releasing.

In here what needs to be considered is that ensuring quality is every one's responsibility. It is not confined to people who are assigned for testing.

And the other surprising facts are related to the future of employees at Google, which will probably rewind the future of SETs TEs and Test Directors/Managers in other IT related organizations in the near future.

It is hoped that there will be no future SETs. How is it going to be managed?

They can be considered as software engineers as a whole. So what happens ultimately is that the 'testing' task is equally distributed among all the developers of the company. Then the development and testing both is done at the developers side. Ultimately all can work as one team.  



TE's tasks will be changed in the future. How is their life cycle being replaced?

In the future TE's job will be something different with respect to the present job of them. In future dogfooders, early adopers and crowd testers etc. will be involved in testing and they will be giving feedback about the system. Then what TEs will have to do is seeing and evaluating whether all these things(feed backs etc) covered the project in testing aspect. He will be involving in calculating the risk impacts, adjusting the test activities like things rather than involved in test creation and execution types of things. So ultimately they will become as specialists or managers for testing.

What happen to Test Directors and Managers in the future?

There will be fewer Test Directors and Managers involved with Google in the near future.   

So ultimately these things will be very surprising without the explanations for them. 

Different types of tests done in the 'Clam Antivirus Project'

Compatibility Testing

Why compatibility tests?

Compatibility test makes sure that the software applications and hardware devices function correctly with all relevant operating systems, platforms, architectures  and with computing environments.

1. A compatibility test is a kind of  Environment Test.

2. 'Buildbot' is a good tool which can be used for compatibility testing.

For antivirus software it is needed to do compatibility testing for both software and database as they are expected to be reliably run on different computer platforms. Therefore compatibility test is an unavoidable test for anti-virus software.   

Performance Testing

In simple terms, performance testing is a type of testing which is intended to determine the responsiveness, throughput, reliability and scalability (if necessary) of a system under a given workload.

Antivirus software need to handle hundreds of thousands of virus signatures and hundreds of file formats, at the same time they must perform all the tasks fast enough to, not to fail the computer system. Therefore performance testing is a must for antivirus software. 

‘Profilers’ are some special tools designed for; code execution performance analysis. It is said that even with profilers, it’s not an easy task to identify parts of the code that can result in slowdowns. One of the reasons for this is; that a code with problems can be hidden in routines that are not called frequently.

Stress testing can be done as a part of the performance testing. Stress testing helps to discover potential problems with stability, robustness, and general efficiency of the software.

User Acceptance Testing

User Acceptance Testing (UAT) is performed by the users of the system to certify the system with respect to the requirements that was agreed upon or it to certify that it is what they wanted.

The usability and user acceptance testing is the last step before giving out the final product.

Advantages of UAT

1. It is able to provide many useful suggestions to the system being developed

2. Can help to verify companie’s development ideas

3. The people who contribute help the project as the best black box testers. 

Methods of User Acceptance Testing

1. Opinion polling 

 Discussions can be held and opinions can be taken from the public about the major changes, so that the complaints can be reduced.

2. Candidate releasing

As a good practice, the releasing of candidates is not done for mission critical   applications. Otherwise the alpha or beta releases of the product can be thrown to use.

 

Beautifull Testing -Testing Open Source with Open Tools

This chapter in beautiful testing book reveals that testing anti-virus software can be considered as a challenging task.

Clam Anti Virus software project has developed a number of tools for testing their software. The methods of testing that they are using are Black box and White box testing.

Black box testing- A surface level testing which only considers the requirements specification.

White box testing- Internal implementation of the system is thoroughly considered and all the units of code is covered.

Statistics Analysis
1. A kind of white box testing
2. Look for common programming errors.
3. Ensure that the source code meets standards.
4. Manual and automatic testing are possible

GCC

1. Compilers are the mostly used tools for statistical analysis. And this is one of them.
2. Syntax and semantic(internal structure of the program) checking is done at the compilation time.
3. GCC refuses to compile code with bugs, therefore can identify and fix the bugs carefully.

Clang Static Analyser

1. Capable of finding critical buggs, such as devide by 0, invalid function calls etc.
2. When compared to other analysers this has a good advantage. That is; it provides a report in html format with the detected bugs.

Spling (Secure Programming Lint)

1. It can detect security-related issues such as null pointer dereferences,
memory management errors, possible infinite loops which other tools are not much capable of.
2. Splint is considered as a very sensitive tool.

Memory Checkers

1. Memory relate issues are captured in these types of testing tools.
2. Some of the examples for memory issues can be mentioned as memory leaks, using undeclared memory, writing out of bounds etc.

Valgrind

1. The advantage of this tool is that, we need not to modify the application that we need to test through it.
2. Can work with the binary files.
3. Can detect synchronization errors

Electric Fence and DUMA

1. Both are malloc() debuggers.
2. Can detect memory leaks
3. Can detect overruns of boundaries for heap-allocated buffers and invalid memory access.

Mudflap

1. It is a pointer debugging tool
2. It can detect memory reference violations
3. Mudflap is a part of GCC and additional components are therefore not required.
4. Can detect out of object access.

Unit Testing

1. This is a kind of white box testing
2. Used to check whether individual units of codes are working properly.
3. Unit tests allow us to quickly check the code before committing any changes to our SVN repository.
4. Can do unit testing in different platforms, this is one of the good advantages of a unit test
5. 'Check' is a popular open source unit-testing framework which is inspired by JUnit for C.

Test Scripts

1. A black box type of texting
2. It focuses only on the specification, not the internal details.
3. When once created, they can be executed automatically, making the testing faster.

Fuzz Testing

1.As the name implies

2.In fuzz testing we provide invalid, unexpected, or random data to the inputs of a computer program. 

3.Then the program is monitored for exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks. 

4.Fuzzing is mainly used to test for security problems in software or computer systems.

5.'Fuzil' is a tool which can be used for fuzzing.

Testing the Environment

1.'Autoconf' is a very popular tool which is used for testing the environment.

2. It configure script to test the system libraries and even the compilers for possible bugs

3. Some compilers create broken binaries while optimizing the code. These types of problems can be found in environment testing done by Autoconf type tools.

Beautiful Testing - Beautiful Mutation Testing - chapter 18

In mutation testing what we are doing is; using a slightly and systematically changed version of the program which has defects, and test against our test suit to find whether our test suite can find them.

For an example consider the bellow example code.

if (x||y) {
  z = 1;
}
else{
  z = 0;
}

We can mutate the above code by changing the logical operator as bellows.

if (x&&y) {
  z = 1;
}
else{
  z = 0;
}

Evaluating test case quality

There are several metrices for evaluating test case quality.

1. Code coverage
All the codes are needed to be executed by the test suit without bypassing any.
2. Condition coverage
All the condition statements are needed executed by the test suit

The AspectJ compiler is an extention to java that can be used for testing.
In this chapter it is used Aspectj to view the bug distribution along different classes of a project. It is then obvious that the bud distribution is uneven.
Pareto effect : When the distribution of bugs are uneven then it is called the pareto effect.

According  to the book mostly the system tests are practically carried out through Junit which is used for unit testing in java.

The book has used the term "watching the watchmen" metaphorically to figure out that the 'quality of quality assurance' should be tested.

Mutations : These are the artificial defects introduced to a program. ex: such as changing a constant.
If the test suite is able to find it then, we can be sure that it could find the real bugs.

Benefits of Mutation Testing

1. Can evaluate the test suite's capability
2. Can identify modules with high risk

Some mutation frameworks available for every one;

1. Mothra - For Fortrun
2. uJava - For Java

Drawbacks of Mutation Testing

1. Very time consuming

Tips to make mutation testing efficient;

1. Mutate the binary code so that rebuilding is avoided
2. Use mutant scemata
3. Ignore uncovered code

Adding a mutation;

1. Replace numerical constant
2. Negate jump conditions
3. Replace arithmetic operations
4. Removing method calls

Equivalent mutations: Some operators has the capability to create a similar program like the previous program even after adding a mutation. This results in an equivalent mutation.

Javalanche Framework

1. This is used for mutation testing of Java programs.
2. It is a free for everyone framework.
3. And the other thing is that, it can be extended with additional operators or impact detectors. 

Intersting Facts on Beautifull Testing chapter 2

In this chapter, it is shown the profound beauty of testing rather than it's superficial beauty.

The stake holders(interested parties) of testing are the people who are needed be satisfied. They are of two types.

1. Internal stake holders - doing, leading and managing test work
2. External stake holders - other interested parties

Examples for some of the stake holders can be mentioned as follows;

Fellow testers, test leads and managers, developers, development leads and managers, database and system architect, marketing and business analysts, project managers, tech support and help desk staff, vendors, customers,users etc.

Stake holders can be divided in some other groups depending on their attitude and contribution towards a project. Namely;

1. Nutral Stake holders - like legal bodies
2. Anti Stakeholders - like enemies who wish the unsuccessfulness of the project

Each stakeholder has a set of objectives and expectations  to be satisfied through testing. They need them to be efficiently, effectively and elegantly be carried out.

• Effectiveness - Satisfy the objectives.
• Efficiency - Satisfy the objectives and expectations in away that it maximizes the value.
• Elegance - Achieving efficiency and effectiveness gracefully and nicely.

For some clients the testers measure their effectiveness through the severity of the bugs they find.

External Beauty

DDP (Deffect Detection Percentage) = Bugs Detected/ Bugs Present

DDP (DDP before UAT) = Bugs Detected/ Test Bugs + Production Bugs


A test team’s DDP for a system test or system integration test averages around 85%. But if it is for internal use then DDP should be around 95%. If it is a safety/mission critical application then the DDP should be higher.

In testing, cost of quality refers to the recognized technique for measuring the cost of failures.

Cost of Detection - The testing costs on us even we couldn't find bugs.
Cost of Internal Failure - Testing and development costs such as filing bug reports, fixing bugs as a result of finding bugs.
Cost of External Failure - This includes support, testing and development costs that we have to bear because of a faulty system.

Average Cost of a Test Bug = Cost of Detection + Cost of Internal Failure/ test bugs

Average Cost of Production Bug = Cost of External Failure/Production bugs

Test ROI = (ACPB-ACTB) *test bugs/cost of detection







 

Interesting Facts on Beautiful Testing Book - Chapter 1

Interesting Facts on Beautiful Testing Book - Chapter 1

The 1st chapter of Beautiful Testing E book opens up with an incident an organization has undergone which had been able to embarrass,frighten and amuse different sets of employees in the same organization. 

The people who are amused by this incident are the testers. The testers are a different set of people who own  a different set of attitudes and they are completely different from the other IT staff of an organization.

Their amusement should not be criticized because they do it merely to forget the unpleasant situation. 

The testers are there in a team to find and reveal problems in a system  eliminating the risks to an organization, developers and customers.

Testers are not policemen, instead they needed to be treated as advisers  . Testers have the difficulty in managing risks with cost, marketing requirements and needs. Evaluating and accepting risks is a project for the testers. Another characteristic of the testers is that they are very reluctant to admit the system is 100% error free. 

According to the book the testers differ from other IT staff because they are paid for telling the truth. Its like telling some one that their baby is ugly and reasoning why.

Some qualities of the testers can be mentioned as follows. The testers are ;
curious, experimental, fearless, intelligent, analytical,learns fast and apolitical.

There are some bad qualities which are rooted with a tester. Such as; they are distrustful, critical about every thing, question every thing.

The people who are testing the others test cases just to complete the task can not be called testers. A good tester testes, update and edit others test cases if there are any drawbacks found instead of just ticking the list of test cases when  testing is over. 

Testers highly expect the developing team to fix the errors they find.
Another thing is that the testers figure out the errors using nice words to the team.

Finding a bug to a tester means finding a treasure to them.

The preparation for bug finding can be stepped as bellows;

1. Read all the required and related books.
2. Ask questions till they understand.
3. Prepare a plan

Encountering bugs depends on the learning and motoring rather than the intelligence lies with in them.

Project managers also get involved in testing because then it is easy to get to know the things that a tester is testing and next time can avoid that bug.

Some testers might not report all the bugs they find, thinking that they are never fixed.This is not a good practice for the testers.

The testers find bugs when compare the ability of the programmers, as they try to prove the system is not working properly, rather than proving it is working properly. They really like to find the ways which fails and breaks the system.

The testers needed to be appreciated for their good work. Otherwise the best will leave searching better opportunities.