Download as PDF, PPTX
More Related Content
Similar to Fundamentals of testing (1)
Fundamentals of testing (1)
- 1. Presentation on UnitI By Prof. R. S. Patwardhan
- 2. When weare testing something we are checking whether it is OK. Testing is necessary because we all make mistakes. Some of those mistakes are unimportant, but some of them are expensive or dangerous. Because we should assume our work contains mistakes, we all need to check our own work. However, some mistakes come from bad assumptions and blind spots, so we might make the same mistakes when we check our own work as we made when we did it. So we should get someone else to check our work.
- 3. Nowadays almosteveryone is aware of softwares. We encounter them in our homes, at work, while shopping etc. However, most people have had an experience with software that did not work as expected. Not all software systems carry the same level of risk and not all problems have the same impact when they occur. A risk is something that has not happened yet and it may never happen; it is a potential problem. We are concerned about these potential problems because, if one of them did happen, we'd feel a negative impact. When we discuss risks, we need to consider how likely it is that the problem would occur and the impact if it happens. Some of the problems we encounter when using software are quite trivial, but others can be costly and damaging - with loss of money, time or business reputation - and even may result in injury or death. For example, suppose a user interface has typographical defects. It may be trivial, but it could have a significant effect, depending on the website and the defect.
- 4. If someonemakes an error or mistake in using the software, this may lead directly to a problem - the software is used incorrectly and so does not behave as we expected. However, people also design and build the software and they can make mistakes during the design and build. These mistakes mean that there are flaws in the software itself. These are called defects or sometimes bugs or faults. When the software code has been built, it is executed and then any defects may cause the system to fail to do what it should do (or do something it shouldn't), causing a failure. Not all defects result in failures; some stay dormant in the code and we may never notice them.
- 5. DO OURMISTAKES MATTER? ◦ Any human being, programmers and testers included, can make an error. ◦ These errors may produce defects in the software code or system, or in a document. ◦ If a defect in code is executed, the system may experience a failure. ◦ So the mistakes we make matter partly because they have consequences for the products for which we are responsible. ◦ Defects in software, systems or documents may result in failures, but not all defects do cause failures. ◦ It is not just defects that give rise to failure. ◦ Failures can also be caused by environmental conditions as well: for example, a radiation burst, a strong magnetic field, electronic fields, or pollution could cause faults in hardware or firmware. ◦ Failures may also arise because of human error in interacting with the software, perhaps a wrong input value being entered or an output being misinterpreted. ◦ Finally, failures may also be caused by someone deliberately trying to cause a failure in a system - malicious damage.
- 6. WHEN DODEFECTS ARISE?
- 7. WHAT ISTHE COST OF DEFECTS?
- 8. Rigorous testingis necessary during development and maintenance to identify defects, in operational environment and increase the quality of the operational system. This includes looking for places in the user interface where a user might make a mistake in input of data or in the interpretation of the output, and looking for potential weak points for intentional and malicious attack. Executing tests help us move towards improved quality of product and service, but that is just one of the verification and validation methods applied to the products. Processes are also checked, for example by audit. A variety of methods may be used to check work, some of which are done by the author of the work and some by others to get an independent view. We may also be required to carry out software testing to meet contractual or legal requirements, or industry specific standards. These standards specify what kind of techniques we must use, or the percentage of software code that must be exercised.
- 9. Testing helpsus to measure the quality of software in terms of number of defects found, the tests run and the system covered by the tests. Testing can give confidence in the quality of software if it finds few or no defects, provided we are happy that the testing is sufficiently rigorous. A poor test may uncover few defects and may leave us with a false sense of security. A well designed test will uncover defects if they are present and so, if such a test passes we will rightly be more confident in the software and be able to assert that the overall level of risk to use the system has been reduced. WHAT IS QUALITY? ◦ Projects aim to deliver software to specification. ◦ For the project to deliver what the customer needs requires a correct specification. ◦ Additionally the delivered system must meet the specification. This is known as verification and validation.
- 10. ◦ As wellas wanting the right software system built correctly, the customer wants the project to be within budget and timescale. ◦ It is important that the project team, the customers and any other stakeholders set and agree expectations. ◦ We need to understand what the customers understand by quality and what their expectations are. ◦ What we as software developers see as quality- that the software meets its defined software specifications, is technically excellent and has few bugs in it- may not provide a quality solution to the customer. ◦ Furthermore if our customers feel that they have spent more money than they wanted or that the software does not help them carry out their tasks, they wont be impressed by the technical excellence of the solution. WHAT IS ROOT CAUSE ANALYSIS? ◦ When we detect failures, we might try to track them back to their root cause, the real reason that they happened. ◦ There are several ways of carrying out root cause analysis, often involving a group brainstorming ideas and discussing them.
- 11. ◦ Understanding theroot causes of defects is an important aspect of quality assurance activities, and testing contributes by helping us to identify defects as early as possible before the software is in use. ◦ As testers, we are also interested in looking at defects found in other projects, so that we can improve our processes. HOW MUCH TESTING IS ENOUGH? ◦ We have a choice: test everything, test nothing or test some of the software. ◦ The immediate response would be ‘Everything must be tested’ because we don’t want to use software that is not tested completely. ◦ This implies that we must test all aspects of a software system. ◦ What we must consider is that whether we must or even can test completely. ◦ Pressures on a project include time and budget as well as pressure to deliver a technical solution that meets the customers' needs. ◦ Customers and project managers will want to spend an amount on testing that provides a return on investment for them. ◦ This return on investment includes preventing failures after release that are costly. ◦ Testing completely - even if that is what customers and project managers ask for - is simply not what they can afford. ◦ Instead we need a test approach which provides the right amount of testing for this project, these customers (and other stakeholders) and this software. ◦ We do this by aligning the testing we do with the risks for the customers, the stakeholders, the project and the software.
- 12. THE DRIVINGTEST – AN ANALOGY FOR SOFTWARE TESTING In a driving test, the examiner critically assesses the candidate's driving, noting every mistake, large or small, made by the driver under test. The examiner takes the driver through a route which tests many possible driving activities, such as road junctions of different types, control and maneuvering of the car, ability to stop safely in an emergency, and awareness of the road, other road users and hazards. Some of the activities must be tested. The examiner bases the judgment on the number and severity of the failures identified, and also whether the driver has been able to meet the driving requirements. A single severe fault is enough to fail the whole test, but a small number of minor faults might still mean the test is passed. Many minor faults would reduce the confidence of the examiner in the quality —of the driving to the point where the driver cannot pass.
- 13. The definitionis broken down into parts; the definition has some key phrases to remember. The definition starts with a description of testing as a process and then lists some objectives of the test processes. Testing as a process: ◦ Process - Testing is a process rather than a single activity - there are a series of activities involved. ◦ All life cycle activities – Testing as a process takes place throughout the software development life cycle. We saw earlier that the later in the life cycle we find bugs, the more expensive they are to fix. ◦ Both static and dynamic -as well as tests where the software code is executed to demonstrate the results of running tests (often called dynamic testing) we can also test and find defects without executing code. This is called static testing. This testing includes reviewing of documents (including source code) and static analysis. This is a useful and cost effective way of testing. ◦ Planning - Activities take place before and after test execution. We need to manage the testing; for example, we plan what we want to do; we control the test activities; we report on testing progress and the status of the software under test; and we finalize or close testing when a phase completes.
- 14. ◦ Preparation –We need to choose what testing we’ll do, by selecting test conditions and designing test cases. ◦ Evaluation – As well as executing the tests, we must check the results and evaluate the software under test and the completion criteria, which help us decide whether we have finished testing and whether the software product has passed the tests. ◦ Software products and related work products – We don’t just test code. We test the design and requirement specifications, and we test related documents such as operation, user and training material. The second part of the definition covers some of the objectives for testing- the reasons why we do it. ◦ Determine that software products satisfy specified requirements ◦ Demonstrate that software products are fit for purpose ◦ Detect defects
- 15. The drivingexaminer becomes the software tester. The driver being examined becomes the system or software under test. Planning and preparation – Both the examiner and the tester need a plan of action and need to prepare for the test, which is not exhaustive, but is representative and allows risk – based decisions about the outcome. Static and dynamic – Both dynamic (driving the car or executing the software) and static(questions to the driver or review of the software) tests are useful. Evaluation – The examiner and the tester must make an objective evaluation, log the test outcome and report factual observations about the tests. Determine that they satisfy specified requirements – The examiner and the tester both check against requirements to carry out particular tasks successfully.
- 16. Demonstrate thatthey are fit for purpose – The examiner and the tester are not evaluating for perfection but for meeting sufficient attributes required to pass the test. Detect defects – The examiner and tester both look for and log faults.
- 17. We canuse both static and dynamic testing as means for achieving similar test objectives. Both provide information to improve both the system to be tested , and the development and testing processes. Testing often includes the following goals and objectives: ◦ Finding defects ◦ Gaining confidence in and providing information about the level of quality ◦ Preventing defects Many types of review and testing activities take place at different stages in the life cycle. These have different objectives. Early testing- such as early test design and review activities – finds defects early on when they are cheap to find and fix. Fixing the defects may not always be the test objective or the desired outcome. Sometimes we simply want to gather information and measure the software.
- 18. When maintainingsoftware by enhancing it or fixing bugs, we are changing software that is already being used. In that case an objective of testing may be to ensure that we have not made errors and introduced defects when we changed the software. This is called regression testing. We may continue to test the system once it is in operational use. In this case, the main objective may be to assess the system characteristics such as reliability or availability.
- 19. Reviewing defectsand failures in order to improve processes allows us to improve our testing and our requirements, design and development processes. One phenomenon that many testers have observed is that defects tend to cluster. Testers will often use this information when making their risk assessment for planning the tests, and will focus on known hot spots. A main focus of reviews and other static tests is to carry out testing as early as possible, finding and fixing defects more cheaply, and preventing defects from appearing at later stages of the project. These activities help us find out about defects earlier and identify potential clusters. Additionally, an important outcome of all testing is information that assists in risk assessment; these reviews will contribute to the planning for the tests executed later in the software development life cycle.
- 20. Over time,as we improve our whole software development life cycle and the early static testing, we may well find that dynamic test levels find fewer defects. As the hot spots for bugs get cleaned up we need to move our focus elsewhere, to the next set of risks. Over time, our focus may change from finding coding bugs, to looking at the requirements and design documents for defects, and to looking for process improvements so that we prevent defects in the product.
- 21. When atest finds a defect that must be fixed, a programmer must do some work to locate the defect in the code and make the fix. In this process called debugging, a programmer will examine the code for the immediate cause of the problem, repair the code and check that the code now executes as expected. The fix is often then tested separately to confirm the fix.
- 22. This principlearises from the theory of the process of scientific experimentation and has been adopted by testers. It says, however many white swans we see, we cannot say ‘All swans are white’. However, as soon as we see one black swan we can say ‘Not all swans are white’. In the same way, however many tests we execute without finding a bug, we have not shown ‘There are no bugs’. As soon as we find a bug, we have shown ‘This code is not bug-free’.
- 23. Testing showspresence of defects Exhaustive testing is impossible Early testing Defect clustering Pesticide paradox ◦ If the same tests are repeated over and over again, eventually the same set of test cases will no longer find any new bugs. To overcome this 'pesticide paradox', the test cases need to be regularly reviewed and revised, and new and different tests need to be written to exercise different parts of the software or system to potentially find more defects. Testing is context dependent ◦ Testing is done differently in different contexts. For eg., safety – critical software is tested differently from an e-commerce site. Absence of errors fallacy ◦ Finding and fixing defects does not help if the system built is unusable and does not fulfill the users’ needs and expectations.
- 24. The activitiesin the fundamental test process are divided into the following basic steps: ◦ Planning and Control; ◦ Analysis and Design; ◦ Implementation and Execution; ◦ Evaluating exit criteria and Reporting; ◦ Test closure activities.
- 25. Test planninghas the following major tasks, which help us build a test plan: ◦ Determine the scope and risks and identify the objectives of testing. ◦ Determine the test approach. ◦ Implement the test policy and/or the test strategy ◦ Determine the required test resources ◦ Schedule test analysis and design tasks, test implementation , execution and evaluation. ◦ Determine the exit criteria. This will show us which tasks and checks we must complete for a particular level of testing before we can say that testing is finished. Management of any activity does not stop with planning it. We need to control and measure progress against the plan. So test control is an ongoing activity. Test control has the following major tasks: ◦ Measure and analyze the results of reviews and testing. ◦ Monitor and document progress test coverage and exit criteria ◦ Provide information on testing.
- 26. Initiate correctiveactions: For example, tighten exit criteria for defects fixed, ask for more effort to be put into debugging or prioritize defects for fixing test blockers. Make decisions: Based on the measures and information gathered during testing and any changes to business and project risks or our increased understanding of technical and product risks, we'll make decisions or enable others to make decisions: to continue testing, to stop testing, to release the software or to retain it for further work for example.
- 27. During testanalysis and design, we take general testing objectives identified during planning and build test designs and test procedures. Test analysis and design has the following major tasks: ◦ Review the test basis (such as the product risk analysis, requirements, architecture, design specifications, and interfaces), examining the specifications for the software we are testing. We use the test basis to help us build our tests. ◦ Identify test conditions based on analysis of test items, their specifications, and what we know about their behavior and structure. ◦ Design the tests, using techniques to help select representative tests that relate to particular aspects of the software which carry risks or which are of particular interest, based on the test conditions and going into more detail. ◦ Evaluate testability of the requirements and system. The requirements may be written in a way that allows a tester to design tests. ◦ Design the test environment set-up and identify any required infrastructure and tools.
- 28. During testimplementation and execution, we take the test conditions and make them into test cases and set up the test environment. Test implementation and execution has the following major tasks: IMPLEMENTATION ◦ Develop and prioritize our test cases, and create test data for those tests. We will also write instructions for tests. ◦ Create test suites from the test cases for efficient test execution. A test suite is a logical collection of test cases which naturally work together. We’ll also set up a test execution schedule. ◦ Implement and verify the environment. We make sure that the test environment has been set up correctly, possibly even running specific tests on it. EXECUTION ◦ Execute the test suites and individual test cases, following our test procedures. ◦ Log the outcome of test execution and record the identities and versions of the software under test, test tools and testware. ◦ Compare actual results (what happened when we ran the tests) with expected results (what we anticipated would happen).
- 29. ◦ Where thereare differences between actual and expected results, report discrepancies as incidents. ◦ Repeat test activities as a result of action taken for each discrepancy. We need to re-execute tests that previously failed in order to confirm a fix.
- 30. Evaluating exitcriteria is the activity where test execution is assessed against the defined objectives. This should be done for each test level, as for each we need to know whether we have done enough testing. Evaluating exit criteria has the following major tasks: ◦ Check test logs against the exit criteria specified in test planning: We look to see what evidence we have for which tests have been executed and checked, and what defects have been raised, fixed, confirmation tested, or are outstanding. ◦ Assess if more tests are needed or if the exit criteria specified should be changed. ◦ Write a test summary report for stakeholders: It is not enough that the testers know the outcome of the test. All the stakeholders need to know what testing has been done and the outcome of the testing, in order to make informed decisions about the software.
- 31. During testclosure activities, we collect data from completed test activities to consolidate experience, including checking and filing testware, and analyzing facts and numbers. Test closure activities include the following major tasks: ◦ Check which planned deliverables we actually delivered and ensure all incident reports have been resolved through defect repair or deferral. ◦ Finalize and archive testware, such as scripts, the test environment, and any other test infrastructure, for later reuse. ◦ Hand over testware to the maintenance organization who will support the software and make any bug fixes or maintenance changes, for use in confirmation testing and regression testing. ◦ Evaluate how the testing went and analyze lessons learned for future releases and projects.
- 32. Independent Testing-Who is a tester? The mindset we want to use while testing and reviewing is different from the one we use while analyzing or developing. By this we mean that, if we are building something we are working positively to solve problems in the design and to realize a product that meets some need. However, when we test or review a product, we are looking for defects in the product and thus are critical of it. We do not mean that a tester cannot be a programmer, or that a programmer cannot be a tester, although they often are separate roles. In fact, programmers are testers - they test the components which they build, and the integration of the components into the system. With the right mindset, programmers can test their own code; indeed programmers do test their own code and find many problems, resolving them before anyone else sees the code.
- 33. However, weall know it is difficult to find our own mistakes. So programmers often rely on others to help test their work. This other person might be a fellow analyst, designer or developer. A person who will use the software may help test it. Testing specialists - professional testers – are often involved. In fact, testing may involve a succession of people each carrying out a different level of testing. This allows an independent test of the system. Several levels of independence can be identified, listed here from the lowest level of independence to the highest: ◦ tests by the person who wrote the item under test; ◦ tests by another person within the same team, such as another programmer; ◦ tests by a person from a different organizational group, such as a independent test team; ◦ tests designed by a person from a different-organization or company, such as outsourced testing or certification by an external body.