Introduction to Exploratory Testing

hi everyone my name is Brittany Stewart I'm a senior QA specialist from kabus today I'm excited to share with you guys about introduction to exploratory testing now this is a 30 minute session or just about um but this is an extraction from a bigger presentation a full day workshop that was done so my aim here is to share with you um as much as I can today about the world of exploratory testing and so on that note let's begin before we begin these presentations i' just like to encourage everyone to listen um and to relax and enjoy the session but the major goals for this session is that by the end of this learning session you should understand the concept of exploratory testing um be familiar with some of the key principles so that's like understanding what where when and how and why to use exploratory testing you'll get an understanding of exploratory testing process and be exposed to some of the techniques and strategies so this is something that I've been doing for the last six years um exploratory testing is something that all of us do in some shape or form but I don't think everybody knows it yet so let me go through and share with you guys what I have and hopefully you learn a lot lot from this session too so for section one we're going to go through the fundamentals of exploratory testing so what exactly is exploratory testing well exploratory testing is an approach to software testing that is described as the simultaneous learning test design and test execution but I do like the definition by Elizabeth Hendrickson which I quote here um so she says it's simultaneously designing executing I um tests and learning about the system using your insights about the last experiment to inform the next so that's just a high level definition in the testing space there's a lot of discussion about which definitions are right and um you know the best way to explore it or sorry in the testing Comm their discussions about the best way to explain it and so this definition is the one that I like to reference when we start talking about exploratory testing now um like uh other testing techniques exploratory testing has its pros and its cons so we're going to get into a bit of that but before I do for those of you who are probably not that familiar with testing I want to explain some differences between exploratory testing as well as um the other types of testing that are out there that we normally use so in this um slide I have three testing Types on one end we have our scripted test which most um persons are used to so this scripted test are the test ones that where we write our test scripts um whether they're automated or not we write test cases and they're designed ahead of time they're usually reviewed by the team and then feedback is given at the end of executing these test cases so with that type of testing feedback is usually slower um execution for this one is also usually follows like a step-by-step instruction approach and we're normally clear about what the expected results are I also have a note here that this can be manual or automated and that is so for for the um scripted tests on the total opposite end of scripted testing is ad hoc tests and I think a lot of people do confuse ad hoc tests with exploratory testing so this is just to clear that up here and now ad hoc tests do not have any test cases there's no strategy there's no sharing of findings no documentation and it cannot be automated now from my experience exploratory testing Falls right in the middle of these two approaches um it's the sweet spot where the process of creating and running tests is simultaneous as we said in the previous slide but with this type of process it allows us to give rapid and valuable feedback to the team um test results are unpredictable which is expected unpredictability and then um it enhances our personal freedom as a tester to be able to execute as we see fit and then this may be automated but that's something that we will talk about a little later on in the session as well now there are many different types of exploratory testing but I've included these um top three so you have the freestyle approach which I think a lot of people do uh it doesn't include any rules no approaches no specifications it's just um like at the beginning of a project you're just going through and you don't note anything it's more of a learning situation and then you have the scenario based approach which is based on real user stories most times or we use end to end scenarios but the main thing here is to base or testing on a specific user um user Persona or p uh use a uh scenario to guide the testing and and then the last one is strategy based testing so this one is more um performed by an experienced test as I have noted here because this one includes different types of testing methodology methodologies and strategies that we use to enhance our expiratory testing experience so with the three definitions that our Le the three types of exploratory testing that I just shared I like to ask a question you know do you think you've already been doing exploratory testing and um I would like it if you think about it are you doing more of an ad hoc style right now based on the the information I give or are you doing more scripted and there's nothing in between um can think about that and let me know as well so the next part is when do you actually use exploratory testing I know a lot of people are confused about this one um so I've designed this slide to highlight the areas where I believe it's most important to use it firstly in an agile environment where our Sprints are short and we have releases um that are very frequent it's important to or you can use exploratory testing to help you in these instances so agile projects definitely do benefit from exploratory testing because we're able to provide um fast feedback secondly when you need to learn about a product functionality or customer requirements so whenever I start on a new project a kabus um you know it's a lot of information to take in and you need to start somewhere so exporter testing is usually my go-to approach to start it helps me to just learn about product functionality I do look at requirements make my assumptions um question things because I'm very curious so exper testing allows us to do that you learn about the product you can review the customer requirements as well thirdly when requirements are unclear or non-existent and you know that happens a lot uh in this time we have clients sometimes that are not clear on what exactly they want so export testing is one of those approaches that you can pull on to help to start somewhere and start getting Clarity on requirements if there are none in place and then as I said before you can provide it allows you to provide faster feedback so in the scenarios where I get a new build on stage for example or the testing environment I will make sure to do an exploratory test before I even going to my sanity or smoke tests um just to be able to do a quick uh run through as I say here you know a quick in andout of how things are looking on the build is it stable enough and then I will go into the detail testing and then lastly when you want to diversify the testing process and this is after you've completed your scripted testing so we can only do so much with our scripted tests they stop and usually when they stop that's where you have to stop but exploiter testing gives us more diversity in terms of our testing and this is one of the reasons I love it especially when um a team has automated regression suits in place and so automation is there to support testing it leaves more time on the testers QA hands to be able to fully um explore the system so these are just some key moments that you can pull on using exploratory testing so during my years of doing this there have been some challenges that I faced and I've also read about and so in this next slide I have propos some solutions to exploratory testing so let's take a look first challenge that I have experienced um and heard of is lack of traceability and that's just due to the nature of it uh exploratory testing doesn't require much planning um because you know there's no predefined test plan or strategy or test scripts but to overcome this challenge what I found helpful is to to log um the test as as and when it's executed and i' share my progress with the team I'm going to talk about this a little bit more further down you'll see the process but that is the solution to the traceability it can be tracked secondly lack of product knowledge so when people start out with um exporter testing they they're like you know I don't know I'm not really sure what to do and that's okay you're not really supposed to know exploratory testing does allow us opportunity to learn while we're test while we're designing our test and while we're executing so the solution for this part is to include another skill tester with you in your exploratory testing session or leverage pair testing and let just pulling on somebody else on a team who may know more about the product than you getting on a call with them and doing a session together and then lastly not knowing when or how to adopt exploitatory testing so to do this it's best to leverage the test method to find out find bgs in the software when time is limited as I said in an agile environment when we're trying to move fast we can pull on exploratory testing um so these are some challenges but we have Solutions and I'm going to show you exactly how I do this shortly I included this slide because I love the image it it basically summarizes all that that I've said um just now about why do we do exploratory testing so firstly it ensures faster testing um it does allow us to deliver feedback at a faster pace and so with that it supports our agile environments it allows us to discover unexpected defects and it promotes realtime thinking which is lovely um and then it enables us to do real time learning which is also lovely for somebody who's very curious about what you're testing um it requires less preparation so that's also a big plus we can get started faster and so with all of that it saves us time find more books discover the unknown unknowns so I don't know if anybody is familiar with this quote but I love it because it in testing it helps to understand why exper testing is important so I'm going to read it it says there are known known there are things we know we know we also know that there are known unknowns and that is to say we know there are some things we do not know but there are also unknown unknowns the ones we don't know we don't know and so in testing basically why um exporter testing is important is it helps us to understand each of these areas I'm going to proceed and let me tell you why because I found this very cool quadrant to explain this when we're testing um risk thinking about risk risk classification is a very important part of what we're doing there are always issues that we have to be able to say okay this one is needs to get fixed first or this one needs to be pushed to the next release and things like that so this quadrant kind of breaks down the known unknowns unknowns un you know those details um but I won't go into too much about it what I wanted to highlight here is the other types of scripted tests allow us to find the known unknowns um the unknown knows but where expiratory testing comes in and is most beneficial is when there are unknown unknown and those are the things that it is impossible for us to to know or to predict you know um what I found is that some of the nastiest bugs hide in these areas in the unknown unknown areas and usually customers find those before QA does or um you know like a big publicity thing happens and it's just because we weren't looking in that area you wouldn't have known to look so the question is you know how can we possibly catch the bugs that live in this unknown unknown area and from my experience expiratory testing is one of the best ways to do that because it gives us the freedom to look um without judgment you know so let me move on to the next one in the testing world and this is something I um experience a lot myself is when we doing our scripted test and we have to run the same test every day uh if we don't have automation for example and you're doing mainly manual testing it's really easy for us to miss things and this is not a fault on any one person it's just human nature it's called intentional um blindness or observational blindness I've also seen it listed as and so it's easy for us to miss things um just because we get fixated on a part particular Target or um we're so fixated on one thing that we Overlook information in another place and exploratory testing can help help us to overcome that I have included this video because it brings awareness to the fact that you know what I'm saying is that we we are only human and we can only notice so much so I'm going to play this video hopefully it can work and we'll see a demonstration of you know how easy it is for us to miss things okay I'm not able to play the video but that's okay essentially whenever we're looking at something for too long it's only natural for us to miss certain things um and so I just want anybody who's out there watching you know if you've ever missed a brog that got lost in production or something of that nature it's okay it's completely normal we're looking at a lot but um exploratory testing can help us to break out of this and to overcome this challenge as testers so the main purpose of this was I wanted to let everyone know that you can do exploratory tting um and people might find it hard to start or you're not really sure where to start so this is going to be a quick start guide to get you up and running in your exploratory testing um sessions first and foremost it's important to have a brain in the loop and so the QA or whoever is doing it is very important and to illustrate that I've included a screen screenshot here of a bug that was found in Google Maps some time ago a long time ago where a couple they were looking for directions from Sweden um no Stockholm Sweden yeah and they were trying to get to Oslo Norway and this is the route that Google gave them took them all around to London and the United Kingdom and then brought them back to to Norway now I had to share this because I doubt any scripted test would have caught this or something like this in in the beginning um maybe visual regression I'm not sure but I really feel like only a human could pick up something like so like this could pick up a bug like this so if you've ever had to use Google Maps to lead you anywhere and it gave you any weird Roots this is this is an example of that but you see here that our exploratory testing with the expert testing concept you do need to have a human being involved D in it and so on that note I want to highlight that with the brain in the loop mindset and how you frame your thoughts is very important before you start it's important to be um have a like a be very curious about what it is you are working on so Embrace that experimentation mindset and at cor of us we like to say keep exploring I love that because that's the mind's that's the mindset that you need to have so be curious you must want to find out you know where is the source of Truth for example um you it's important not to let the requirements hold hold you Ransom because sometimes they don't capture everything so it's good to question it's good to be willing to explore uh the other thing is leverage domain knowledge so I come from my design background and I do pull on my my web design knowledge to help me with testing so I do focus on like the front end there are other people I know who have backgrounds in law and engineering and it's important for QA to understand that you can bring some of your knowledge in if it's from a user perspective it doesn't matter but domain knowledge can really help you um with testing especially exploratory testing so make sure you don't know your domain knowledge is a part of your tool kits when it comes to this one personal freedom my favorite part because the scripted tests are so rigid sometimes you you have to try and stick to the exact steps um being curious you I always want to click out and try different things and so expert testing does offer you the personal freedom to be able to try some things and to see what the results are and then lastly you know expert tests are usually very creative um like I said you think outside the box you critical think you even think about the users perspectives and you use that to guide um your testing sessions so mindset is very important and without a tester in this process it wouldn't be um successful so remember your goals if you were going on any like anything that we're doing we have to have a plan we have to have a goal and so I like to remind myself of these things before I start in exploratory testing and so you can choose one of these it doesn't have to be all three but your goal could be to gain an understanding of how the application works one um what its interface looks like what are the functionality that it has implemented and this could be one of the goals the other one is to force the software to exhibit all its capabilities so you're really working to you know answer some hard questions and probably break it as people would say you want to try to break the software that's fine that could be a goal too and then the last one is to find more bugs you would be exploring those edge cases or as I said the unknown unknowns to see what is actually happening and are there potential spots where bugs are being missed um the goal is very important because it helps to prevent you from going into an ad hoc testing type type of um situation where you kind of aimlessly going through but with this you have a plan you have a goal you know where you're going and you can stay focused with purpose and intent during the exploratory testing session so I love to reference this diagram um as a quick start guide it breaks it down pretty simple to me so first thing you got to do when you're doing exper testing is you have to have a plan as with any like if you were going hiking or on on a trip you would need to plan uh additionally You' need to track then you need to capture some you know whatever your findings are and then it's very important to share feedback because it's a loop so I'm going to go into each of these one by one first of all when it comes to us starting in terms of the planning um in in the software testing world we use test Charters to help us to plan our exploration so what exactly is a test Charter well test Charters are just statements that guide our testing as I said before you know you want to make sure that you have a purpose you have an intent for what you're doing and so the first step is test Charters questions that I like to ask myself at the beginning of the sessions are what am I exploring um if if you were going on a hiking trip for example You' also need to figure out what it is you need to bring with you do you need water or do you need a flashlight similarly with this what resources do I need um I'm going to give some examples shortly and then I like to ask what kind of information am I hoping to find that's just to make the goal a little bit clearer as well so test charts um do help with our Effectiveness and efficiency when we're doing this um but they can be very difficult to create or some people don't really know where to start and so I love to reference Elizabeth Elizabeth Elizabeth Hendrickson's um test chart of format here where she breaks it down for us you're allowed it says you explore the Target and targets can be features or requirements or modules for example with the particular resource resource examples could be be a tool testing tool a data set a particular technique if um certain configurations need to be on and off in a particular environment so things like that what do you need to help you to get the testing done and then to discover information um what is it that you're looking for here are some examples you could be looking at a feature discover more about that you could be checking into the security of the application you could be wanting to find out what's the performance like for this website um how is the usability for it is there any accessibility being consist being considered in this so these are some examples of information that you could be looking to discover just high level ideas to get you started but feel free to Deep dive into this and then as I said like good writing test Charters is something that I'm still working on every day um but but I did find some good examples of test Charters so I wanted to share with you good test Charters usually off a direction without over specifying the test actions because when you get into the details like that that's when you're going back more to the scripted test um as opposed to being exploratory so a good example of a test Chara is one that's not too broad and it's not too specific here's a good example explore the checkout page with each product type available to a new user to discover any issues with the checkout flow so it's not too specific it's not too broad um a bad example of a test Charter would be something like this one where you have test a checkout process for an e-commerce website X that's definitely too broad you're not really sure where you're going with that one and then here's another one testing test clicking a gift card to add it to the cart and view what's on the confirmation page so that one now sounds more like a test case it's telling you exactly what to click where and um the expected results so this is an example of a bad Charter as you gain experience in writing Charters it does become easier to specify your intention and to write these Charters but as I said use the template the Explorer with to discover template to start it does help to just get you in the right direction all right the next point is tracking or exploratory testing how do you do that in the industry best practice is to use session based testing um I'm not going to go into all the details here I have five parts um that you could consider but I'm going to focus on the time boxing the review and the debrief part of this so uh in times of the time in terms of the time boxing it's recommended right now that you spend 45 minutes or to 90 minutes max doing your exploratory testing session so that's just setting a little timer um and making sure that you stop at the end because otherwise you end up two things can happen you end up going down rabbit holes because when you're testing you always see things things that are not within your goal or within your plan will pop up and it will catch you off guard and can you lose um track of what it is you're actually trying to do so the timers the session based approach helps you to stay focused um I mean also it helps you to add this to your time sheets this is something that I I had a problem with in the past and people weren't sure you know what how much time are you spending on this exploratory testing why is it taking so long things like that when you track your time this way it makes it easier um also if you find yourself going over 90 minutes you do run the chance of meeting up on observational blindness that I showed before where you can easily miss things so it's always good to take a break and then to come back and and restart your um testing sessions within this recommended time okay so the next point is how do you capture so when we talk about the review and the debriefing how do you actually capture the exploratory testing notes um um it's not as document sorry it's not as detailed as uh the scripted test but you can capture certain things and so I like to use these tips to guide me when you're making notes for your exploratory testing sessions you can capture things like comments um anything positive that you see happening positive feedback is always good any concerns that you had while you were doing the session any clear problems that you saw if there were any questions that came up you should note those as well if any suggestions any enhancements come to mind you not those and same thing for the ideas to me it's best to Note Everything um and then you come back to it in a debrief session to figure out what's most important to share with the team additionally I love to use visuals to help so uh if I'm using a mind map or I'm using a regular note taking app I do use these icons to represent the status of certain eras that I'm testing or how I feel about it so this is just a guide to show you how you can possibly and then here's an example of an exporter testing session that I did some time ago this is just a regular uh notion note board but you see I have my Charter outlined um I had details on the build I made a lot of notes and I even put it in a table so the format of how you document doesn't really matter but what matters is that you do try to capture as much information as you can with the guide of what I said before okay so how do you now share your findings from exploratory testing with your team um this is a very important part set the process otherwise you don't really grow and so to do this this is my recommendation you can share your findings via this report which is called a pqip report um basically though it's just a report that highlights the Praises the positives um the questions that you have any problems that you see and if any ideas so you can Nar it down to just those four areas capture those share those you have a Blog for examp you can share it there any problems that you found um I've also found it helpful to share it in refinements or even um or retrospectives if there was anything that went well for from a Praises section you can include those if there are any problems which are areas for improvement you highlight those as well so it's nothing too formal um it's just enough for auditing purposes and tracking purposes and then it's just lean documentation and I believe this is a good way to keep track of your exploratory testing sessions and to share the findings okay so session three section three is about generating ideas at some point when you're you're doing an exploratory testing session it becomes hard for you to think of new scenarios and so there are a number of ways that you can actually generate more explorator testing ideas this gets into a little bit more of the advanced side of things but I just want to introduce you to it some ways that you can generate new ideas for further exploration you can use user personas so by placing yourself in the user shoes it's very important to understand who the user users are and you put yourself in that frame of mind and run through the application in that way another thing that you can use are H heuristics and neonics which we have a lot of them in the testing Community too many to list here um but mental short these are basically mental shortcuts that allow you to solve problems really quickly so you can just pull on it um while you're doing the session and it will help you to to guide you another thing is using variations so if you switch up your navigation for example instead of going to the home to to the checkout page via the the icon on the top you can find another way in the photo so switching up navigation thinking of different combinations of inputs for example will help to diversify the tests and give you more ideas for exploring and then lastly peer testing so that's just working closely with another person on your team it doesn't have to be another QA person it could be anybody on the team who has knowledge um during those sessions you asking questions and you get guidance you get ideas from those sessions on how to um probably expand your testing and to look in other areas on this slide I wanted to share with you some of my top heris stics The Helpful ones that have I've been using for a while now to help me um I won't go into detail but there are resources available that will'll talk about each of these um each of them will help you to do more with your expiratory testing and as I said you know using the peer testing approach where two people test the same thing on the application at the same time it doesn't matter who you're with I give an example you as the exploratory tester here but you can call on another team member a product owner a developer a VA and you to in a session it allows you to ask the questions get more um information that can help to expand your testing efforts okay now let's talk about the traceability and coverage part uh so as I said you know the one of the issues that we had was how do you keep track of this and one of the best ways I found to do this was using the Mind map here's an example of how I tracked my findings so you'll see here I capture exactly what it is I'm testing on I included the date the time for the session I included my test Charter and then you see I'm using the um pqip format that I mentioned here so all Praises are highlighted any questions also using the um icons that I mentioned any ideas lots of ideas were were generating during that and then I was able to highlight problems which turned into bugs or um other user stories for action with the team and then just general comments and notes that would have been helpful for testing in the future so this is just an example of how you can use a mind map to help you to track and see the coverage of your testing okay session four has to do with Auto and exploratory testing there's been a lot of discussions around this one and so I like to ask you know can exploratory test be automated I I do want more information on this um but in my experience my answer is yes exploratory testing can be automated to a certain extent I've seen in the space where people are actually you know getting into the code and simulating certain behaviors that are more of an exploratory nature um than just doing our reg regression testing so this is something that can be explored some more um but Automation in exporter testing is possible here I wanted to share some passive expert testing tools so passive meaning you know they're they're tools that you use for documenting uh so to capture if you don't have anything right now to capture your test if you don't use a mind map you can also use this extension it's available free on Chrome exploratory testing extension it allows you to take notes the bugs the questions the ideas so it gives you the pqip format reporting that I mentioned this is a good one when it comes to tracking your time you can use any time tracker but there's this Ultra timer available on Chrome which I use and then you're going through your exporter testing sessions when you want to come up with edge cases bug maget is a nice extension to use because it gives you lots of input variables to include um so you don't have to go search it on the web for like zip codes or invalid emails or inv valid addresses for example it has all of that in there and different languages and then All-In-One tools I used to use x-ray in the past to do some exploratory testing that does help us to capture in um the details in a ticket allows you to create the bug tickets directly from that um still exploring Azure test plans that's what we're exploring now so when I get more information on that I will share but really and truly any other note taking app that you use can work so just to summarize everything that we just um spoke about exploratory testing is a dynamic approach that complements scripted test and it offers its own benefits key principles in this exper testing is that you know you lean on your domain knowledge your expertise is very important being curious and embracing your critical thinking skills and then being creative is there's are also very important in expert testing and um testing process for expiratory testing is design execute document and you use your test Charters to help you to design and you use sessions to help you with execution and then you document your findings but everything happens at once test strategies so you should be pulling on risk based testing as well as time boxing for efficiency and Effectiveness and then lastly collaboration and communication the most important part here is that you feed the information back to your team it's essential to testers it's essential to developers it's essential to everybody on the team and so that concludes this session I hope you learned a lot about exploratory testing um at least just to get started if you have more questions about it there's references to some of the books I've read that have um helped me here those would be helpful there are also additional links to heuristics to help you to expand your exploratory testing and some general resources and links available thank you everyone