December 18, 2015
An Introduction to Exploratory Testing
Curiosity is an inherent trait of humans and so all of us, from the greatest to the simplest minds, explore. We explore from Windows to Mac, from beaches to mountains, from indoor games to outdoor games, from new mobile phones to new software, from Black box to white box testing; we also explore code to test their functionality.
Hearing of the ‘exploratory’ mode of testing, the first thing that comes to mind is ‘ad-hoc’ testing. But there’s a difference between someone who is running when he is late versus a professional athlete that competes in a 100-meter sprint. Exploratory testing is a lot more than wandering aimlessly through an application looking for bugs; it is a ‘thoughtful approach’ which involves simultaneous learning, test design, and test execution.
What is Exploratory Testing?
As its name implies, exploratory testing is about exploring. Exploring the software to discover what the software does and doesn’t do. According to Cem Kaner & James Marcus Bach, exploratory testing is more a mindset or “...a way of thinking about testing” than a methodology. They also say that it crosses a continuum from slightly exploratory (slightly ambiguous or vaguely scripted testing) to highly exploratory (freestyle exploratory testing).
Exploratory testing means the testing of software without any specific plans or schedules. This is a formal testing process where we don’t have any test cases or test planning documents to test the application. Instead, testers identify the functionality of an application by exploring the application, learning the application, and designing and executing the test plans according to their findings.
But can we all agree on such an unplanned process?
A lot of us might still think an exploratory approach is no match to a pre-planned and scripted approach in any situation, but let’s not forget, even an elaborately defined test script may leave a few implicit details to the discretion of the tester, which they will have to explore. In this view, nearly all testing performed by testers is exploratory to some degree and we all perform exploratory testing at one time or another, at least instinctively.
Exploratory testing is simultaneously learning, test design, and test execution. In other words, exploratory testing is any testing to the extent that the tester actively controls the design of the tests as those tests are performed, and then uses that information gained while testing to design new and better tests.
After studying Exploratory Testing, we can say that this type of testing can be extremely fruitful in finding elusive bugs that may not otherwise have been discovered until user acceptance testing.
How can we do Exploratory Testing?
The process of exploratory testing is all about discovery, investigation, and learning. This emphasizes the personal freedom and responsibility of the individual tester. Test cases are not created in advance; instead, testers open the application and start browsing it to find defects. They may note down ideas about what to test before test execution. The focus of exploratory testing is more on testing as a "thinking" activity. Exploratory testing is a hands-on approach in which testers are involved in minimum planning and maximum test execution. The process of exploratory testing is as shown below:
- The planning involves the creation of a test plan, which is a short declaration of the scope for a short time-boxed test effort. In this test plan, we define the objectives and possible approaches to be used in that short time declaration
- In Exploratory testing, both test design and test execution are performed simultaneously and there is no formal documentation for the test scripts, test conditions, or test cases
- Test logging is performed at the same time as test execution. During this, we document the key aspects of what functionality is tested, any defects found, and any thoughts regarding further testing
- By using exploratory testing with other formal testing techniques, we have greater confidence in our software products. Therefore, exploratory testing helps to make sure that the most severe defects have been identified
Some challenges experienced
We can say Exploratory testing is just to explore a module or product on the basis of testing experience. Many times we faced a challenge where the requirement document is much less informative on that situation on which we started our testing on the basis of experience. Let’s look at some examples on the challenges that we have faced.
- Once I was working on a mobile application project where we found a bug. It was a unique kind of bug, which our normal test cases were not able to find. Our product was a mobile application in which a user can recharge their mobile numbers. The bug looked like this: when one user is recharging through a pass code, the application worked fine. However, when multiple users used the same pass code for recharging, all of the users received money in their mobile account. In a normal scenario, we are not supposed to do this kind of test, but because I has been working this project for a long period of time, I used my idea to test this scenario found this bug.
- Data transfer and message systems are a few places where we can find defects of critical severity, because in both cases message/data is transferred from one end to another. Additionally, many times the messages or data are set in a data table and then delivered to the end user. This holds potential failure points if the developer doesn’t clearly communicate which changes are made to which data tables.
- This was a situation where we encountered defects as a result of frequently changing requirements. A business decided that they wanted to change the logic of an application due to which flow is affected. Let's say there is a flag set in a table of the product database--according to the new changes, the flag should be removed from the database and its value added into the configuration file. However, because of urgent requirements, many times the tester forgot to test all configuration setting options. When you explore the depths of configuration, you will definitely find defects. It may take some digging, but eventually we can find some lethal combination
- In this example, according to the client’s requirement there was an offline module, which is a jar of java code. To work off of that jar file, first we have to do an FTP of file, where we transfer a file with a different user mobile number from a different operator, and then process that file by picking a number from the file and inserting that data into the corresponding operator table. Now this module is reading for the testing process. In the testing process, we uploaded a file exceeding the upload limit because we didn’t know the limit, which caused the system to crash and raised a defect. When we asked the developer why this defect came up, they said that they set a size limit on the database table, but not on the file upload in the offline module.
What makes a good Exploratory tester?
In exploratory testing, the quality of testing is completely dependent upon the tester’s skills of designing test cases and finding defects. The testing quality is also dependent upon the tester’s knowledge about the product and the different test methods.
Testers should possess the following skills:
Knowledge of Test Design
An exploratory tester should have knowledge about the test design technique which best suits the product so that the test cases will be capable of finding defects. To achieve this, an exploratory tester should possess skills like the usage of different tools, and the ability to evaluate risks, analyze the product, and think critically.
Excellent exploratory testers are more careful observers than beginners. Beginners or less experienced testers only observe what the functionality tells them, while an exploratory tester must look out for any unusual or mysterious behavior in the system.
Critical thinking is the kind of thinking that specifically looks for problems and mistakes. To be a good exploratory tester, you need to be a good critical thinker. Critically thinking testers save projects from dangerous assumptions and ultimately from disasters. Excellent exploratory testers should be able to review and explain their logic and look for errors in their own thinking.
Excellent exploratory testers produce more and better ideas than beginners. Exploratory testers use documents and thought processes such as guidelines, generic checklists, important indications, and rules of thumb.
Exploratory Testing Tools
Exploratory Testing tools are used as a part of the testing phase to automate certain tasks, improve testing efficiency, and discover issues that might be difficult to find using manual review alone. The section below illustrates a few exploratory testing tools.
Session Tester is an exploratory testing tool for managing and recording Session-Based Testing. It has a timer that limits test sessions to a particular length, and it provides an easy way to record session notes. Notes are stored in an XML format that can be converted to HTML, or be transformed into the desired format.
Test Studio helps to be more efficient in doing exploratory testing. It is an easy to use in-browser extension that allows teams members to send ideas and feedback or to report bugs directly from the application while browsing. Screenshots and annotations can be attached, and feedback via email can be shared, or can be exported to Word and PDF.
qTest explorer is an intelligent capture technology. It tracks user’s interactions with an application and uses that information to automatically create a detailed defect document. It supports the recording of a wide range of applications and technologies on the browser as well as desktop apps.
Microsoft Test Manager
Microsoft Test Manager (MTM) helps you by records the actions a user performs as they work with their application. It also allows the user to record screenshots, comments, file attachments, audio narration and screen video. The recording makes it easier to trace any fault that the user might discover. The user can also store their actions as a test case, so that it is easy for them to replicate the test after the application is updated.
Exploratory Testing (ET) consists of a series of focused sessions where the tester is following a charter document that helps them test a specific product feature or area. ET should be a significant part of any team approach to overall quality strategy. It leverages testers’ skills and experience and frequently find bugs before customers find them. It is a supplement to, rather than a replacement of the manual, automated, performance, and load tests.