العربية
中国
Français
Deutsch
Ελληνική
हिन्दी
Italiano
日本語
Português
Русский
EspañolSOFTWARE TESTING
(Redirected from Beta testing)
'Software testing' is the process used to measure the quality of developed computer software. Usually, quality is constrained to such topics as correctness, completeness, security, but can also include more technical requirements as described under the ISO standard ISO 9126, such as capability, reliability, efficiency, portability, maintainability, compatibility, and usability. Testing is a process of technical investigation, performed on behalf of stakeholders, that is intended to reveal quality-related information about the product with respect to the context in which it is intended to operate. This includes, but is not limited to, the process of executing a program or application with the intent of finding errors. Quality is not an absolute; it is value to some person. With that in mind, testing can never completely establish the correctness of arbitrary computer software; testing furnishes a ''criticism'' or ''comparison'' that compares the state and behaviour of the product against a specification. An important point is that ''software testing'' should be distinguished from the separate discipline of ''Software Quality Assurance'' (SQA), which encompasses all business process areas, not just testing.
There are many approaches to software testing, but effective testing of complex products is essentially a process of investigation, not merely a matter of creating and following routine procedure. One definition of testing is "the process of questioning a product in order to evaluate it", where the "questions" are operations the tester attempts to execute with the product, and the product answers with its behavior in reaction to the probing of the tester[1]. Although most of the intellectual processes of testing are nearly identical to that of review or inspection, the word testing is also used to connote the dynamic analysis of the product—putting the product through its paces. Sometimes one therefore refers to reviews, walkthroughs or inspections as "static testing", whereas actually running the program with a given set of test cases in a given development stage is often referred to as "dynamic testing", to emphasize the fact that formal review processes form part of the overall testing scope.
In general, software engineers distinguish software faults from software failures. In case of a failure, the software does not do what the user expects. A fault is a programming error that may or may not actually manifest as a failure. A fault can also be described as an error in the correctness of the semantic of a computer program. A fault will become a failure if the exact computation conditions are met, one of them being that the faulty portion of computer software executes on the CPU. A fault can also turn into a failure when the software is ported to a different hardware platform or a different compiler, or when the software gets extended.
Software testing may be viewed as a sub-field of Software Quality Assurance but typically exists independently (and there may be no SQA areas in some companies). In SQA, software process specialists and auditors take a broader view on software and its development. They examine and change the software engineering process itself to reduce the amount of faults that end up in the code or deliver faster.
Regardless of the methods used or level of formality involved, the desired result of testing is a level of confidence in the software so that the organization is confident that the software has an acceptable defect rate. What constitutes an acceptable defect rate depends on the nature of the software. An arcade video game designed to simulate flying an airplane would presumably have a much higher tolerance for defects than software used to control an actual airliner.
A problem with software testing is that the number of defects in a software product can be very large, and the number of configurations of the product larger still. Bugs that occur infrequently are difficult to find in testing. A rule of thumb is that a system that is expected to function without faults for a certain length of time must have already been tested for at least that length of time. This has severe consequences for projects to write long-lived reliable software, since it is not usually commercially viable to test over the proposed length of time unless this is a relatively short period. A few days or a week would normally be acceptable, but any longer period would usually have to be simulated according to carefully prescribed start and end conditions.
A common practice of software testing is that it is performed by an independent group of testers after the functionality is developed but before it is shipped to the customer. This practice often results in the testing phase being used as project buffer to compensate for project delays, thereby compromising the time devoted to testing. Another practice is to start software testing at the same moment the project starts and it is a continuous process until the project finishes.
This is highly problematic in terms of controlling changes to software: if faults or failures are found part way into the project, the decision to correct the software needs to be taken on the basis of whether or not these defects will delay the remainder of the project. If the software does need correction, this needs to be rigorously controlled using a version numbering system, and software testers need to be accurate in knowing that they are testing the correct version, and will need to re-test the part of the software wherein the defects were found. The correct start point needs to be identified for retesting. There are added risks in that new defects may be introduced as part of the corrections, and the original requirement can also change part way through, in which instance previous successful tests may no longer meet the requirement and will need to be re-specified and redone (part of regression testing). Clearly the possibilities for projects being delayed and running over budget are significant.
Another common practice is for test suites to be developed during technical support escalation procedures. Such tests are then maintained in regression testing suites to ensure that future updates to the software don't repeat any of the known mistakes.
It is commonly believed that the earlier a defect is found the cheaper it is to fix it. This is reasonable based on the risk of any given defect contributing to or being confused with further defects later in the system or process. In particular, if a defect erroneously changes the state of the data on which the software is operating, that data is no longer reliable and therefore any testing after that point cannot be relied on even if there are no further actual software defects.
In counterpoint, some emerging software disciplines such as extreme programming and the agile software development movement, adhere to a "test-driven software development" model. In this process unit tests are written first, by the software engineers (often with pair programming in the extreme programming methodology). Of course these tests fail initially; as they are expected to. Then as code is written
it passes incrementally larger portions of the test suites. The test suites are continuously updated
as new failure conditions and corner cases are discovered, and they are integrated with any regression
tests that are developed.
Unit tests are maintained along with the rest of the software source code and generally integrated into the build process (with inherently interactive tests being relegated to a partially manual build acceptance process).
The software, tools, samples of data input and output, and configurations are all referred to collectively as a test harness.
The separation of debugging from testing was initially introduced by Glenford J. Myers in 1979.[3] Although his attention was on breakage testing it illustrated the desire of the software engineering community to separate fundamental development activities, such as debugging, from that of verification. Drs. Dave Gelperin and William C. Hetzel classified in 1988 the phases and goals in software testing as follows:[4]
Dr. Gelperin chaired the IEEE 829-1989 (Test Documentation Standard) with Dr. Hetzel writing the book ''The Complete Guide to Software Testing''. Both works were pivotal in to today's testing culture and remain a consistent source of reference. Dr. Gelperin and Jerry E. Durant also went on to develop High Impact Inspection Technology that builds upon traditional Inspections but utilizes a test driven additive.
White box and black box testing are terms used to describe the point of view a test engineer takes when designing test cases. 'Black box' testing assumes an external view of the test object; one inputs data and one sees only outputs from the test object. 'White box' testing provides an internal view of the test object and its processes.
In recent years the term gray box testing has come into common usage. The typical gray box tester is permitted to set up or manipulate the testing environment, such as by seeding a database, and can view the state of the product after his actions, such as performing a SQL query on the database to be certain of the values of columns.
'Gray box' testing is used almost exclusively by client-server testers or others who use a database as a repository of information, but can also apply to a tester who has to manipulate input or configuration files directly, or perform testing like SQL injection. It can also be used by testers who know the internal workings or algorithm of the software under test and can write tests specifically for the anticipated results. For example, testing a data warehouse implementation involves loading the target database with information, and verifying the correctness of data population and loading of data into the correct tables.
Software testing is used in association with verification and validation (V&V). ''Verification'' is the checking of or testing of items, including software, for conformance and consistency with an associated specification. Software testing is just one kind of verification, which also uses techniques such as reviews, inspections, and walkthroughs. ''Validation'' is the process of checking what has been specified is what the user actually wanted.
★ Verification: Are we doing the job right?
★ Validation: Have we done the right job?
★ Unit testing tests the minimal software component, or module. Each unit (basic component) of the software is tested to verify that the detailed design for the unit has been correctly implemented.
★ Integration testing exposes defects in the interfaces and interaction between integrated components (modules). Progressively larger groups of tested software components corresponding to elements of the architectural design are integrated and tested until the software works as a whole.
★ System testing tests an integrated system to verify that it meets its requirements.
★ System integration testing verifies that a system is integrated to any external or third party systems defined in the system requirements.
★ Acceptance testing can be conducted by the end-user, customer, or client to validate whether or not to accept the product. Acceptance testing may be performed after the testing and before the implementation phase. ''See also Development stage''
★
★ '''Alpha testing''' is simulated or actual operational testing by potential users/customers or an independent test team at the developers' site. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing, before the software goes to beta testing.
★
★ '''Beta testing''' comes after alpha testing. Versions of the software, known as beta versions, are released to a limited audience outside of the company. The software is released to groups of people so that further testing can ensure the product has few faults or bugs. Sometimes, beta versions are made available to the open public to increase the feedback field to a maximal number of future users.
It should be noted that although both Alpha and Beta are referred to as testing it is in fact use immersion. The rigors that are applied are often unsystematic and many of the basic tenets of testing process are not used. The Alpha and Beta period provides insight into environmental and utilization conditions that can impact the software.
After modifying software, either for a change in functionality or to fix defects, a regression test re-runs previously passing tests on the modified software to ensure that the modifications haven't unintentionally caused a ''regression'' of previous functionality.
Regression testing can be performed at any or all of the above test levels. These regression tests are often automated.
A test case is a software testing document,which consists of event, action, input, output, expected result, and actual result. Clinically defined (IEEE 829-1998) a test case is an input and an expected result. This can be as pragmatic as 'for condition x your derived result is y', whereas other test cases described in more detail the input scenario and what results might be expected. It can occasionally be a series of steps (but often steps are contained in a separate test procedure that can be exercised against multiple test cases, as a matter of economy) but with one expected result or expected outcome. The optional fields are a test case ID, test step or order of execution number, related requirement(s), depth, test category, author, and check boxes for whether the test is automatable and has been automated. Larger test cases may also contain prerequisite states or steps, and descriptions. A test case should also contain a place for the actual result. These steps can be stored in a word processor document, spreadsheet, database, or other common repository. In a database system, you may also be able to see past test results and who generated the results and the system configuration used to generate those results. These past results would usually be stored in a separate table.
The term test script is the combination of a test case, test procedure, and test data. Initially the term was derived from the product of work created by automated regression test tools. Today, test scripts can be manual, automated, or a combination of both.
The most common term for a collection of test cases is a test suite. The test suite often also contains more detailed instructions or goals for each collection of test cases. It definitely contains a section where the tester identifies the system configuration used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests.
Collections of test cases are sometimes incorrectly termed a test plan. They might correctly be called a test specification. If sequence is specified, it can be called a test script, scenario, or procedure.
Although testing varies between organizations, there is a cycle to testing:
# Requirements Analysis: Testing should begin in the requirements phase of the software development life cycle.
#: During the design phase, testers work with developers in determining what aspects of a design are testable and under what parameter those tests work.
# Test Planning: Test Strategy, Test Plan(s), Test Bed creation.
# Test Development: Test Procedures, Test Scenarios, Test Cases, Test Scripts to use in testing software.
# Test Execution: Testers execute the software based on the plans and tests and report any errors found to the development team.
# Test Reporting: Once testing is completed, testers generate metrics and make final reports on their test effort and whether or not the software tested is ready for release.
# Retesting the Defects
Not all errors or defects reported must be fixed by a software development team. Some may be caused by errors in configuring the test software to match the development or production environment. Some defects can be handled by a workaround in the production environment. Others might be deferred to future releases of the software, or the deficiency might be accepted by the business user. There are yet other defects that may be rejected by the development team (of course, with due reason) if they deem it
Main articles: Code coverage
Code Coverage is inherently a white box testing activity. The target software is built with special options or libraries and/or run under a special environment such that every function that is exercised (executed) in the program(s) are mapped back to the function points in the source code. This process allows developers and quality assurance personnel to look for parts of a system that are rarely or never accessed under normal conditions (error handling and the like) and helps reassure test engineers that the most important conditions (function points) have been tested.
Test engineers can look at code coverage test results to help them devise test cases and input or configuration sets that will increase the code coverage over vital functions. Two common forms of code coverage used by testers are statement (or line) coverage, and path (or edge) coverage. Line coverage reports on the execution footprint of testing in terms of which lines of code were executed to complete the test. Edge coverage reports which branches, or code decision points were executed to complete the test. They both report a coverage metric, measured as a percentage.
Generally code coverage tools and libraries exact a performance and/or memory or other resource cost which is unacceptable to normal operations of the software. Thus they are only used in the lab. As one might expect there are classes of software that cannot be feasibly subjected to these coverage tests, though a degree of coverage mapping can be approximated through analysis rather than direct testing.
There are also some sorts of defects which are affected by such tools. In particular some race conditions or similar real time sensitive operations can be masked when run under code coverage environments; and conversely some of these defects may become easier to find as a result of the additional overhead of the testing code.
Code coverage may be regarded as a more up-to-date incarnation of debugging in that the automated tools used to achieve statement and path coverage are often referred to as “debugging utilities”. These tools allow the program code under test to be observed on screen whilst the program is executing, and commands and keyboard function keys are available to allow the code to be “stepped” through literally line by line. Alternatively it is possible to define pinpointed lines of code as “breakpoints” which will allow a large section of the code to be executed, then stopping at that point and displaying that part of the program on screen. Judging where to put breakpoints is based on a reasonable understanding of the program indicating that a particular defect is thought to exist around that point. The data values held in program variables can also be examined and in some instances (with care) altered to try out “what if” scenarios. Clearly use of a debugging tool is more the domain of the software engineer at a unit test level, and it is more likely that the software tester will ask the software engineer to perform this. However, it is useful for the tester to understand the concept of a debugging tool.
There is considerable ''controversy'' among testing writers and consultants about what constitutes responsible software testing. Members of the "context-driven" school of testing believe that there are no "best practices" of testing, but rather that testing is a set of skills that allow the tester to select or invent testing practices to suit each unique situation. In addition, prominent members of the community consider much of the writing about software testing to be doctrine, mythology, and folklore. Some might contend that this belief directly contradicts standards such as the IEEE 829 test documentation standard, and organizations such as the Food and Drug Administration who promote them. The context-driven school's retort is that Lessons Learned in Software Testing includes one lesson supporting the use IEEE 829 and another opposing it; that not all software testing occurs in a regulated environment and that practices appropriate for such environments would be ruinously expensive, unnecessary, and inappropriate for other contexts; and that in any case the FDA generally promotes the principle of the least burdensome approach.
Some of the major controversies include:
Starting around 1990, a new style of writing about testing began to challenge what had come before. The seminal work in this regard is widely considered to be ''Testing Computer Software'', by Cem Kaner.[5] Instead of assuming that testers have full access to source code and complete specifications, these writers, including Kaner and James Bach, argued that testers must learn to work under conditions of uncertainty and constant change. Meanwhile, an opposing trend toward process "maturity" also gained ground, in the form of the Capability Maturity Model. The agile testing movement (which includes but is not limited to forms of testing practiced on agile development projects) has popularity mainly in commercial circles, whereas the CMM was embraced by government and military software providers.
However, saying that "maturity models" like CMM gained ground against or opposing Agile testing may not be right. Agile movement is a 'way of working', while CMM is a process improvement idea.
But another point of view must be considered: the operational culture of an organization. While it may be true that testers must have an ability to work in a world of uncertainty, it is also true that their flexibility must have direction. In many cases test cultures are self-directed and as a result fruitless; unproductive results can ensue. Furthermore, providing positive evidence of defects may either indicate that you have found the tip of a much larger problem, or that you have exhausted all possibilities. A framework is a test of Testing. It provides a boundary that can measure (validate) the capacity of our work. Both sides have, and will continue to argue the virtues of their work. The proof however is in each and every assessment of delivery quality. It does little good to test systematically if you are too narrowly focused. On the other hand, finding a bunch of errors is not an indicator that Agile methods was the driving force; you may simply have stumbled upon an obviously poor piece of work.
Exploratory testing means simultaneous test design and test execution with an emphasis on learning. Scripted testing means that learning and test design happen prior to test execution, and quite often the learning has to be done again during test execution. Exploratory testing is very common, but in most writing and training about testing it is barely mentioned and generally misunderstood. Some writers consider it a primary and essential practice. Structured exploratory testing is a compromise when the testers are familiar with the software. A vague test plan, known as a test charter, is written up, describing what functionalities need to be tested but not how, allowing the individual testers to choose the method and steps of testing.
There are two main disadvantages associated with a primarily exploratory testing approach. The first is that there is no opportunity to prevent defects, which can happen when the designing of tests in advance serves as a form of structured static testing that often reveals problems in system requirements and design. The second is that, even with test charters, demonstrating test coverage and achieving repeatability of tests using a purely exploratory testing approach is difficult. For this reason, a blended approach of scripted and exploratory testing is often used to reap the benefits while mitigating each approach's disadvantages.
Some writers believe that test automation is so expensive relative to its value that it should be used sparingly. Others, such as advocates of agile development, recommend automating 100% of all tests. A challenge with automation is that automated testing requires automated test oracles (an oracle is a mechanism or principle by which a problem in the software can be recognized). Such tools have value in load testing software (by signing on to an application with hundreds or thousands of instances simultaneously), or in checking for intermittent errors in software. The success of automated software testing depends on complete and comprehensive test planning. Software development strategies such as test-driven development are highly compatible with the idea of devoting a large part of an organization's testing resources to automated testing. Many large software organizations perform automated testing. Some have developed their own automated testing environments specifically for internal development, and not for resale.
Software testers should not be limited only to testing software implementation, but also to testing software design. With this assumption, the role and involvement of testers will change dramatically. The test cycle will change too. To test software design, testers will review requirement and design specifications together with designer and programmer. This will help to identify bugs earlier.
Several certification programs exist to support the professional aspirations of software testers and quality assurance specialists. No certification currently offered actually requires the applicant to demonstrate the ability to test software. No certification is based on a widely accepted body of knowledge. No certification board decertifies individuals. This has led some to declare that the testing field is not ready for certification.[6] Certification itself cannot measure an individual's productivity, their skill, or practical knowledge, and cannot guarantee their competence, or professionalism as a tester.[7]
Certifications can be grouped into: exam-based and education-based. Exam-based certifications: For these there is the need to pass an exam, which can also be learned by self-study: e.g. for ISTQB or QAI. Education-based certifications are instructor-led sessions, where each course has to be passed, e.g. IIST (International Institute for Software Testing).
★ CSTE offered by the Quality Assurance Institute (QAI)
★ CSTP offered by the International Institute for Software Testing
★ ISEB offered by the Information Systems Examinations Board
★ ISTQB offered by the International Software Testing Qualification Board
★ CSQE offered by the American Society for Quality (ASQ)
★ CSQA offered by the Quality Assurance Institute (QAI)
One principle in software testing is summed up by the classical Latin question posed by Juvenal:
''Quis Custodiet Ipsos Custodes'' (Who watches the watchmen?), or is alternatively referred
informally, as the "Heisenbug" concept (a common misconception that confuses Heisenberg's uncertainty principle with observer effect). The idea is that any form of observation is also an interaction, that the act of testing can also affect that which is being tested.
In practical terms the test engineer is testing software (and sometimes hardware or firmware) with other software (and hardware and firmware). The process can fail in ways that are not the result of defects in the target but rather result from defects in (or indeed intended features of) the testing tool.
There are metrics being developed to measure the effectiveness of testing. One method is by analyzing code coverage (this is highly controversial) - where every one can agree what areas are not at all being covered and try to improve coverage on these areas.
Bugs can also be placed into code on purpose, and the number of bugs that have not been found can be predicted based on the percentage of intentionally placed bugs that were found. The problem is that it assumes that the intentional bugs are the same type of bug as the unintentional ones.
Finally, there is the analysis of historical find-rates. By measuring how many bugs are found and comparing them to predicted numbers (based on past experience with similar projects), certain assumptions regarding the effectiveness of testing can be made. While not an absolute measurement of quality, if a project is halfway complete and there have been no defects found, then changes may be needed to the procedures being employed by QA.
Software testing can be done by software testers. Until the 1950s the term software tester was used generally, but later it was also seen as a separate profession. Regarding the periods and the different goals in software testing (see D. Gelperin and W.C. Hetzel) there have been established different roles: test lead/manager, tester, test designer, test automater/automation developer, and test administrator.
Participants of testing team:
#Tester
#Developer
#Business Analyst
#Customer
#Information Service Management
#Senior Organization Management
#Quality team
★ All-pairs testing
★ Automated testing
★ Betaware
★ Black box testing
★ Characterization Test
★ Code coverage
★ Defect tracking
★ Design predicates
★ Development stage
★ Dynamic program analysis
★ Elephant in Cairo
★ Equivalence partitioning
★ Exploratory test
★ Formal verification
★ Functional testing
★ Fuzz testing
★ GUI software testing
★ IEEE 829 Standard for software test documentation
★ Integration testing
★ Keyword-driven testing
★ Model-based testing
★ Performance Engineering
★ Pseudolocalization
★ Quality assurance
★ Regression testing
★ Risk-based testing
★ Sanity test
★ Search Based Software Engineering
★ Session-based test
★ Security Testing
★ Smoke testing
★ Soak testing
★ Software engineering
★ Software Quality Assurance
★ Static code analysis
★ Static testing
★ System testing
★ Test case
★ Test Effort
★ Test-driven development
★ Test Vector
★ Tester Driven Development
★ Traceability matrix
★ Unit test
★ White box testing
★ "''An effective way to test code is to exercise it at its natural boundaries.''" -- Brian Kernighan
★ "''Program testing can be used to show the presence of bugs, but never to show their absence!''" -- Edsger Dijkstra
★ "''Beware of bugs in the above code; I have only proved it correct, not tried it.''" -- Donald Knuth
★ "''A relatively small number of causes will typically produce a large majority of the problems or defects (80/20 Rule).''" -- Pareto principle
1. www.cirrusgs.com/FAQs.htm
2. Code Complete, Second Edition, , McConnell, Steve, Microsoft Press, , ISBN 0-7356-1967-0
3. The Art of Software Testing, , Glenford J., Myers, John Wiley and Sons, 1979, ISBN 0-471-04328-1
4. The Growth of Software Testing, , D., Gelperin, CACM, 1988
5. Testing Computer Software, , Cem, Kaner, John Wiley and Sons, 1993, ISBN 1-85032-908-7
6. NSF grant proposal to "lay a foundation for significant improvements in the quality of academic and commercial courses in software testing"
7. Measuring the Effectiveness of Software Testers
★ Boris Beizer: ''Software Testing Techniques.'' Second Edition, International Thomson Computer Press, 1990, ISBN 1-85032-880-3
★ Elfriede Dustin: ''Effective Software Testing.'' Addison Wesley, 2002, ISBN 0-20179-429-2
★ Elfriede Dustin, et al: ''Automated Software Testing.'' Addison Wesley, 1999, ISBN 0-20143-287-0
★ Robert V. Binder: ''Testing Object-Oriented Systems: Objects, Patterns, and Tools.'' Addison-Wesley Professional, 1999, ISBN 0-201-80938-9
★ Rex Black: ''Managing the Testing Process.'' Second Edition, John Wiley and Sons, 2002, ISBN 0-471-22398-0
★ Mark Fewster, Dorothy Graham: ''Software Test Automation.'' Addison Wesley, 1999, ISBN 0-201-33140-3
★ Cem Kaner, James Bach, Bret Pettichord: ''Lessons Learned in Software Testing. A Context-Driven Approach.'' John Wiley & Sons, 2001, ISBN 0-471-08112-4
★ G. T. Laycock: ''The Theory and Practice of Specification Based Software Testing.'' PhD Thesis, Dept of Computer Science, Sheffield University, UK, 1993, free PS-version
★ AppLab's CEO Sashi Reddi sees potential of global software testing Development Market
★ Hung Nguyen, Robert Johnson, Michael Hackett: ''Testing Applications on the Web (2nd Edition): Test Planning for Mobile and Internet-Based Systems'' ISBN 0-471-20100-6
★ James A. Whittaker: ''How to Break Web Software: Functional and Security Testing of Web Applications and Web Services'', Addison-Wesley Professional, February 2, 2006. ISBN 0-321-36944-0
★ Lydia Ash: ''The Web Testing Companion: The Insider's Guide to Efficient and Effective Tests'', Wiley, May 2, 2003. ISBN 0471430218
★ Ron Patton: ''Software Testing'', Second Edition, Sams, July 26, 2005.
★ Andreas Spillner, Tilo Linz, Hans Schäfer (2007) 'Software Testing Foundations, 2nd Edition: ''A Study Guide for the Certified Tester Exam''' (Foundation Level – ISTQB compliant), Rocky Nook, Inc., Santa Barbara, California, USA. [ISBN 978-1-933952-08-6].
★
★ Software Testing Q&A
★ Application Note: Structured Testing - The Path to Thorough Code Coverage
★ Software Test Management Guide, A to Z
★ CSTE Resources and Subjective answers....
★ Is Application Testing in Financial Services Making the Grade?
★ Software Testing tutorials & FAQs
★ Software Testing Life Cycle
★ Blogs on Software Quality & Testing
'Software testing' is the process used to measure the quality of developed computer software. Usually, quality is constrained to such topics as correctness, completeness, security, but can also include more technical requirements as described under the ISO standard ISO 9126, such as capability, reliability, efficiency, portability, maintainability, compatibility, and usability. Testing is a process of technical investigation, performed on behalf of stakeholders, that is intended to reveal quality-related information about the product with respect to the context in which it is intended to operate. This includes, but is not limited to, the process of executing a program or application with the intent of finding errors. Quality is not an absolute; it is value to some person. With that in mind, testing can never completely establish the correctness of arbitrary computer software; testing furnishes a ''criticism'' or ''comparison'' that compares the state and behaviour of the product against a specification. An important point is that ''software testing'' should be distinguished from the separate discipline of ''Software Quality Assurance'' (SQA), which encompasses all business process areas, not just testing.
There are many approaches to software testing, but effective testing of complex products is essentially a process of investigation, not merely a matter of creating and following routine procedure. One definition of testing is "the process of questioning a product in order to evaluate it", where the "questions" are operations the tester attempts to execute with the product, and the product answers with its behavior in reaction to the probing of the tester[1]. Although most of the intellectual processes of testing are nearly identical to that of review or inspection, the word testing is also used to connote the dynamic analysis of the product—putting the product through its paces. Sometimes one therefore refers to reviews, walkthroughs or inspections as "static testing", whereas actually running the program with a given set of test cases in a given development stage is often referred to as "dynamic testing", to emphasize the fact that formal review processes form part of the overall testing scope.
Introduction
In general, software engineers distinguish software faults from software failures. In case of a failure, the software does not do what the user expects. A fault is a programming error that may or may not actually manifest as a failure. A fault can also be described as an error in the correctness of the semantic of a computer program. A fault will become a failure if the exact computation conditions are met, one of them being that the faulty portion of computer software executes on the CPU. A fault can also turn into a failure when the software is ported to a different hardware platform or a different compiler, or when the software gets extended.
Software testing may be viewed as a sub-field of Software Quality Assurance but typically exists independently (and there may be no SQA areas in some companies). In SQA, software process specialists and auditors take a broader view on software and its development. They examine and change the software engineering process itself to reduce the amount of faults that end up in the code or deliver faster.
Regardless of the methods used or level of formality involved, the desired result of testing is a level of confidence in the software so that the organization is confident that the software has an acceptable defect rate. What constitutes an acceptable defect rate depends on the nature of the software. An arcade video game designed to simulate flying an airplane would presumably have a much higher tolerance for defects than software used to control an actual airliner.
A problem with software testing is that the number of defects in a software product can be very large, and the number of configurations of the product larger still. Bugs that occur infrequently are difficult to find in testing. A rule of thumb is that a system that is expected to function without faults for a certain length of time must have already been tested for at least that length of time. This has severe consequences for projects to write long-lived reliable software, since it is not usually commercially viable to test over the proposed length of time unless this is a relatively short period. A few days or a week would normally be acceptable, but any longer period would usually have to be simulated according to carefully prescribed start and end conditions.
A common practice of software testing is that it is performed by an independent group of testers after the functionality is developed but before it is shipped to the customer. This practice often results in the testing phase being used as project buffer to compensate for project delays, thereby compromising the time devoted to testing. Another practice is to start software testing at the same moment the project starts and it is a continuous process until the project finishes.
This is highly problematic in terms of controlling changes to software: if faults or failures are found part way into the project, the decision to correct the software needs to be taken on the basis of whether or not these defects will delay the remainder of the project. If the software does need correction, this needs to be rigorously controlled using a version numbering system, and software testers need to be accurate in knowing that they are testing the correct version, and will need to re-test the part of the software wherein the defects were found. The correct start point needs to be identified for retesting. There are added risks in that new defects may be introduced as part of the corrections, and the original requirement can also change part way through, in which instance previous successful tests may no longer meet the requirement and will need to be re-specified and redone (part of regression testing). Clearly the possibilities for projects being delayed and running over budget are significant.
Another common practice is for test suites to be developed during technical support escalation procedures. Such tests are then maintained in regression testing suites to ensure that future updates to the software don't repeat any of the known mistakes.
It is commonly believed that the earlier a defect is found the cheaper it is to fix it. This is reasonable based on the risk of any given defect contributing to or being confused with further defects later in the system or process. In particular, if a defect erroneously changes the state of the data on which the software is operating, that data is no longer reliable and therefore any testing after that point cannot be relied on even if there are no further actual software defects.
| Time Introduced | Requirements | Architecture | Construction | System Test | Post-Release |
|---|---|---|---|---|---|
| Requirements | 1 | 3 | 5-10 | 10 | 10-100 |
| Architecture | - | 1 | 10 | 15 | 25-100 |
| Construction | - | - | 1 | 10 | 10-25 |
In counterpoint, some emerging software disciplines such as extreme programming and the agile software development movement, adhere to a "test-driven software development" model. In this process unit tests are written first, by the software engineers (often with pair programming in the extreme programming methodology). Of course these tests fail initially; as they are expected to. Then as code is written
it passes incrementally larger portions of the test suites. The test suites are continuously updated
as new failure conditions and corner cases are discovered, and they are integrated with any regression
tests that are developed.
Unit tests are maintained along with the rest of the software source code and generally integrated into the build process (with inherently interactive tests being relegated to a partially manual build acceptance process).
The software, tools, samples of data input and output, and configurations are all referred to collectively as a test harness.
History
The separation of debugging from testing was initially introduced by Glenford J. Myers in 1979.[3] Although his attention was on breakage testing it illustrated the desire of the software engineering community to separate fundamental development activities, such as debugging, from that of verification. Drs. Dave Gelperin and William C. Hetzel classified in 1988 the phases and goals in software testing as follows:[4]
Dr. Gelperin chaired the IEEE 829-1989 (Test Documentation Standard) with Dr. Hetzel writing the book ''The Complete Guide to Software Testing''. Both works were pivotal in to today's testing culture and remain a consistent source of reference. Dr. Gelperin and Jerry E. Durant also went on to develop High Impact Inspection Technology that builds upon traditional Inspections but utilizes a test driven additive.
White-box, black-box, and gray-box testing
White box and black box testing are terms used to describe the point of view a test engineer takes when designing test cases. 'Black box' testing assumes an external view of the test object; one inputs data and one sees only outputs from the test object. 'White box' testing provides an internal view of the test object and its processes.
In recent years the term gray box testing has come into common usage. The typical gray box tester is permitted to set up or manipulate the testing environment, such as by seeding a database, and can view the state of the product after his actions, such as performing a SQL query on the database to be certain of the values of columns.
'Gray box' testing is used almost exclusively by client-server testers or others who use a database as a repository of information, but can also apply to a tester who has to manipulate input or configuration files directly, or perform testing like SQL injection. It can also be used by testers who know the internal workings or algorithm of the software under test and can write tests specifically for the anticipated results. For example, testing a data warehouse implementation involves loading the target database with information, and verifying the correctness of data population and loading of data into the correct tables.
Verification and validation
Software testing is used in association with verification and validation (V&V). ''Verification'' is the checking of or testing of items, including software, for conformance and consistency with an associated specification. Software testing is just one kind of verification, which also uses techniques such as reviews, inspections, and walkthroughs. ''Validation'' is the process of checking what has been specified is what the user actually wanted.
★ Verification: Are we doing the job right?
★ Validation: Have we done the right job?
Levels of testing
★ Unit testing tests the minimal software component, or module. Each unit (basic component) of the software is tested to verify that the detailed design for the unit has been correctly implemented.
★ Integration testing exposes defects in the interfaces and interaction between integrated components (modules). Progressively larger groups of tested software components corresponding to elements of the architectural design are integrated and tested until the software works as a whole.
★ System testing tests an integrated system to verify that it meets its requirements.
★ System integration testing verifies that a system is integrated to any external or third party systems defined in the system requirements.
★ Acceptance testing can be conducted by the end-user, customer, or client to validate whether or not to accept the product. Acceptance testing may be performed after the testing and before the implementation phase. ''See also Development stage''
★
★ '''Alpha testing''' is simulated or actual operational testing by potential users/customers or an independent test team at the developers' site. Alpha testing is often employed for off-the-shelf software as a form of internal acceptance testing, before the software goes to beta testing.
★
★ '''Beta testing''' comes after alpha testing. Versions of the software, known as beta versions, are released to a limited audience outside of the company. The software is released to groups of people so that further testing can ensure the product has few faults or bugs. Sometimes, beta versions are made available to the open public to increase the feedback field to a maximal number of future users.
It should be noted that although both Alpha and Beta are referred to as testing it is in fact use immersion. The rigors that are applied are often unsystematic and many of the basic tenets of testing process are not used. The Alpha and Beta period provides insight into environmental and utilization conditions that can impact the software.
After modifying software, either for a change in functionality or to fix defects, a regression test re-runs previously passing tests on the modified software to ensure that the modifications haven't unintentionally caused a ''regression'' of previous functionality.
Regression testing can be performed at any or all of the above test levels. These regression tests are often automated.
Test cases, suites, scripts, and scenarios
A test case is a software testing document,which consists of event, action, input, output, expected result, and actual result. Clinically defined (IEEE 829-1998) a test case is an input and an expected result. This can be as pragmatic as 'for condition x your derived result is y', whereas other test cases described in more detail the input scenario and what results might be expected. It can occasionally be a series of steps (but often steps are contained in a separate test procedure that can be exercised against multiple test cases, as a matter of economy) but with one expected result or expected outcome. The optional fields are a test case ID, test step or order of execution number, related requirement(s), depth, test category, author, and check boxes for whether the test is automatable and has been automated. Larger test cases may also contain prerequisite states or steps, and descriptions. A test case should also contain a place for the actual result. These steps can be stored in a word processor document, spreadsheet, database, or other common repository. In a database system, you may also be able to see past test results and who generated the results and the system configuration used to generate those results. These past results would usually be stored in a separate table.
The term test script is the combination of a test case, test procedure, and test data. Initially the term was derived from the product of work created by automated regression test tools. Today, test scripts can be manual, automated, or a combination of both.
The most common term for a collection of test cases is a test suite. The test suite often also contains more detailed instructions or goals for each collection of test cases. It definitely contains a section where the tester identifies the system configuration used during testing. A group of test cases may also contain prerequisite states or steps, and descriptions of the following tests.
Collections of test cases are sometimes incorrectly termed a test plan. They might correctly be called a test specification. If sequence is specified, it can be called a test script, scenario, or procedure.
A sample testing cycle
Although testing varies between organizations, there is a cycle to testing:
# Requirements Analysis: Testing should begin in the requirements phase of the software development life cycle.
#: During the design phase, testers work with developers in determining what aspects of a design are testable and under what parameter those tests work.
# Test Planning: Test Strategy, Test Plan(s), Test Bed creation.
# Test Development: Test Procedures, Test Scenarios, Test Cases, Test Scripts to use in testing software.
# Test Execution: Testers execute the software based on the plans and tests and report any errors found to the development team.
# Test Reporting: Once testing is completed, testers generate metrics and make final reports on their test effort and whether or not the software tested is ready for release.
# Retesting the Defects
Not all errors or defects reported must be fixed by a software development team. Some may be caused by errors in configuring the test software to match the development or production environment. Some defects can be handled by a workaround in the production environment. Others might be deferred to future releases of the software, or the deficiency might be accepted by the business user. There are yet other defects that may be rejected by the development team (of course, with due reason) if they deem it
Code coverage
Main articles: Code coverage
Code Coverage is inherently a white box testing activity. The target software is built with special options or libraries and/or run under a special environment such that every function that is exercised (executed) in the program(s) are mapped back to the function points in the source code. This process allows developers and quality assurance personnel to look for parts of a system that are rarely or never accessed under normal conditions (error handling and the like) and helps reassure test engineers that the most important conditions (function points) have been tested.
Test engineers can look at code coverage test results to help them devise test cases and input or configuration sets that will increase the code coverage over vital functions. Two common forms of code coverage used by testers are statement (or line) coverage, and path (or edge) coverage. Line coverage reports on the execution footprint of testing in terms of which lines of code were executed to complete the test. Edge coverage reports which branches, or code decision points were executed to complete the test. They both report a coverage metric, measured as a percentage.
Generally code coverage tools and libraries exact a performance and/or memory or other resource cost which is unacceptable to normal operations of the software. Thus they are only used in the lab. As one might expect there are classes of software that cannot be feasibly subjected to these coverage tests, though a degree of coverage mapping can be approximated through analysis rather than direct testing.
There are also some sorts of defects which are affected by such tools. In particular some race conditions or similar real time sensitive operations can be masked when run under code coverage environments; and conversely some of these defects may become easier to find as a result of the additional overhead of the testing code.
Code coverage may be regarded as a more up-to-date incarnation of debugging in that the automated tools used to achieve statement and path coverage are often referred to as “debugging utilities”. These tools allow the program code under test to be observed on screen whilst the program is executing, and commands and keyboard function keys are available to allow the code to be “stepped” through literally line by line. Alternatively it is possible to define pinpointed lines of code as “breakpoints” which will allow a large section of the code to be executed, then stopping at that point and displaying that part of the program on screen. Judging where to put breakpoints is based on a reasonable understanding of the program indicating that a particular defect is thought to exist around that point. The data values held in program variables can also be examined and in some instances (with care) altered to try out “what if” scenarios. Clearly use of a debugging tool is more the domain of the software engineer at a unit test level, and it is more likely that the software tester will ask the software engineer to perform this. However, it is useful for the tester to understand the concept of a debugging tool.
Controversy
There is considerable ''controversy'' among testing writers and consultants about what constitutes responsible software testing. Members of the "context-driven" school of testing believe that there are no "best practices" of testing, but rather that testing is a set of skills that allow the tester to select or invent testing practices to suit each unique situation. In addition, prominent members of the community consider much of the writing about software testing to be doctrine, mythology, and folklore. Some might contend that this belief directly contradicts standards such as the IEEE 829 test documentation standard, and organizations such as the Food and Drug Administration who promote them. The context-driven school's retort is that Lessons Learned in Software Testing includes one lesson supporting the use IEEE 829 and another opposing it; that not all software testing occurs in a regulated environment and that practices appropriate for such environments would be ruinously expensive, unnecessary, and inappropriate for other contexts; and that in any case the FDA generally promotes the principle of the least burdensome approach.
Some of the major controversies include:
Agile vs. traditional
Starting around 1990, a new style of writing about testing began to challenge what had come before. The seminal work in this regard is widely considered to be ''Testing Computer Software'', by Cem Kaner.[5] Instead of assuming that testers have full access to source code and complete specifications, these writers, including Kaner and James Bach, argued that testers must learn to work under conditions of uncertainty and constant change. Meanwhile, an opposing trend toward process "maturity" also gained ground, in the form of the Capability Maturity Model. The agile testing movement (which includes but is not limited to forms of testing practiced on agile development projects) has popularity mainly in commercial circles, whereas the CMM was embraced by government and military software providers.
However, saying that "maturity models" like CMM gained ground against or opposing Agile testing may not be right. Agile movement is a 'way of working', while CMM is a process improvement idea.
But another point of view must be considered: the operational culture of an organization. While it may be true that testers must have an ability to work in a world of uncertainty, it is also true that their flexibility must have direction. In many cases test cultures are self-directed and as a result fruitless; unproductive results can ensue. Furthermore, providing positive evidence of defects may either indicate that you have found the tip of a much larger problem, or that you have exhausted all possibilities. A framework is a test of Testing. It provides a boundary that can measure (validate) the capacity of our work. Both sides have, and will continue to argue the virtues of their work. The proof however is in each and every assessment of delivery quality. It does little good to test systematically if you are too narrowly focused. On the other hand, finding a bunch of errors is not an indicator that Agile methods was the driving force; you may simply have stumbled upon an obviously poor piece of work.
Exploratory vs. scripted
Exploratory testing means simultaneous test design and test execution with an emphasis on learning. Scripted testing means that learning and test design happen prior to test execution, and quite often the learning has to be done again during test execution. Exploratory testing is very common, but in most writing and training about testing it is barely mentioned and generally misunderstood. Some writers consider it a primary and essential practice. Structured exploratory testing is a compromise when the testers are familiar with the software. A vague test plan, known as a test charter, is written up, describing what functionalities need to be tested but not how, allowing the individual testers to choose the method and steps of testing.
There are two main disadvantages associated with a primarily exploratory testing approach. The first is that there is no opportunity to prevent defects, which can happen when the designing of tests in advance serves as a form of structured static testing that often reveals problems in system requirements and design. The second is that, even with test charters, demonstrating test coverage and achieving repeatability of tests using a purely exploratory testing approach is difficult. For this reason, a blended approach of scripted and exploratory testing is often used to reap the benefits while mitigating each approach's disadvantages.
Manual vs. automated
Some writers believe that test automation is so expensive relative to its value that it should be used sparingly. Others, such as advocates of agile development, recommend automating 100% of all tests. A challenge with automation is that automated testing requires automated test oracles (an oracle is a mechanism or principle by which a problem in the software can be recognized). Such tools have value in load testing software (by signing on to an application with hundreds or thousands of instances simultaneously), or in checking for intermittent errors in software. The success of automated software testing depends on complete and comprehensive test planning. Software development strategies such as test-driven development are highly compatible with the idea of devoting a large part of an organization's testing resources to automated testing. Many large software organizations perform automated testing. Some have developed their own automated testing environments specifically for internal development, and not for resale.
Software design vs. software implementation
Software testers should not be limited only to testing software implementation, but also to testing software design. With this assumption, the role and involvement of testers will change dramatically. The test cycle will change too. To test software design, testers will review requirement and design specifications together with designer and programmer. This will help to identify bugs earlier.
Certification
Several certification programs exist to support the professional aspirations of software testers and quality assurance specialists. No certification currently offered actually requires the applicant to demonstrate the ability to test software. No certification is based on a widely accepted body of knowledge. No certification board decertifies individuals. This has led some to declare that the testing field is not ready for certification.[6] Certification itself cannot measure an individual's productivity, their skill, or practical knowledge, and cannot guarantee their competence, or professionalism as a tester.[7]
Certifications can be grouped into: exam-based and education-based. Exam-based certifications: For these there is the need to pass an exam, which can also be learned by self-study: e.g. for ISTQB or QAI. Education-based certifications are instructor-led sessions, where each course has to be passed, e.g. IIST (International Institute for Software Testing).
Testing certifications
★ CSTE offered by the Quality Assurance Institute (QAI)
★ CSTP offered by the International Institute for Software Testing
★ ISEB offered by the Information Systems Examinations Board
★ ISTQB offered by the International Software Testing Qualification Board
Quality assurance certifications
★ CSQE offered by the American Society for Quality (ASQ)
★ CSQA offered by the Quality Assurance Institute (QAI)
Who watches the watchmen?
One principle in software testing is summed up by the classical Latin question posed by Juvenal:
''Quis Custodiet Ipsos Custodes'' (Who watches the watchmen?), or is alternatively referred
informally, as the "Heisenbug" concept (a common misconception that confuses Heisenberg's uncertainty principle with observer effect). The idea is that any form of observation is also an interaction, that the act of testing can also affect that which is being tested.
In practical terms the test engineer is testing software (and sometimes hardware or firmware) with other software (and hardware and firmware). The process can fail in ways that are not the result of defects in the target but rather result from defects in (or indeed intended features of) the testing tool.
There are metrics being developed to measure the effectiveness of testing. One method is by analyzing code coverage (this is highly controversial) - where every one can agree what areas are not at all being covered and try to improve coverage on these areas.
Bugs can also be placed into code on purpose, and the number of bugs that have not been found can be predicted based on the percentage of intentionally placed bugs that were found. The problem is that it assumes that the intentional bugs are the same type of bug as the unintentional ones.
Finally, there is the analysis of historical find-rates. By measuring how many bugs are found and comparing them to predicted numbers (based on past experience with similar projects), certain assumptions regarding the effectiveness of testing can be made. While not an absolute measurement of quality, if a project is halfway complete and there have been no defects found, then changes may be needed to the procedures being employed by QA.
Roles in software testing
Software testing can be done by software testers. Until the 1950s the term software tester was used generally, but later it was also seen as a separate profession. Regarding the periods and the different goals in software testing (see D. Gelperin and W.C. Hetzel) there have been established different roles: test lead/manager, tester, test designer, test automater/automation developer, and test administrator.
Participants of testing team:
#Tester
#Developer
#Business Analyst
#Customer
#Information Service Management
#Senior Organization Management
#Quality team
See also
★ All-pairs testing
★ Automated testing
★ Betaware
★ Black box testing
★ Characterization Test
★ Code coverage
★ Defect tracking
★ Design predicates
★ Development stage
★ Dynamic program analysis
★ Elephant in Cairo
★ Equivalence partitioning
★ Exploratory test
★ Formal verification
★ Functional testing
★ Fuzz testing
★ GUI software testing
★ IEEE 829 Standard for software test documentation
★ Integration testing
★ Keyword-driven testing
★ Model-based testing
★ Performance Engineering
★ Pseudolocalization
★ Quality assurance
★ Regression testing
★ Risk-based testing
★ Sanity test
★ Search Based Software Engineering
★ Session-based test
★ Security Testing
★ Smoke testing
★ Soak testing
★ Software engineering
★ Software Quality Assurance
★ Static code analysis
★ Static testing
★ System testing
★ Test case
★ Test Effort
★ Test-driven development
★ Test Vector
★ Tester Driven Development
★ Traceability matrix
★ Unit test
★ White box testing
Quotes
★ "''An effective way to test code is to exercise it at its natural boundaries.''" -- Brian Kernighan
★ "''Program testing can be used to show the presence of bugs, but never to show their absence!''" -- Edsger Dijkstra
★ "''Beware of bugs in the above code; I have only proved it correct, not tried it.''" -- Donald Knuth
★ "''A relatively small number of causes will typically produce a large majority of the problems or defects (80/20 Rule).''" -- Pareto principle
References
1. www.cirrusgs.com/FAQs.htm
2. Code Complete, Second Edition, , McConnell, Steve, Microsoft Press, , ISBN 0-7356-1967-0
3. The Art of Software Testing, , Glenford J., Myers, John Wiley and Sons, 1979, ISBN 0-471-04328-1
4. The Growth of Software Testing, , D., Gelperin, CACM, 1988
5. Testing Computer Software, , Cem, Kaner, John Wiley and Sons, 1993, ISBN 1-85032-908-7
6. NSF grant proposal to "lay a foundation for significant improvements in the quality of academic and commercial courses in software testing"
7. Measuring the Effectiveness of Software Testers
★ Boris Beizer: ''Software Testing Techniques.'' Second Edition, International Thomson Computer Press, 1990, ISBN 1-85032-880-3
★ Elfriede Dustin: ''Effective Software Testing.'' Addison Wesley, 2002, ISBN 0-20179-429-2
★ Elfriede Dustin, et al: ''Automated Software Testing.'' Addison Wesley, 1999, ISBN 0-20143-287-0
★ Robert V. Binder: ''Testing Object-Oriented Systems: Objects, Patterns, and Tools.'' Addison-Wesley Professional, 1999, ISBN 0-201-80938-9
★ Rex Black: ''Managing the Testing Process.'' Second Edition, John Wiley and Sons, 2002, ISBN 0-471-22398-0
★ Mark Fewster, Dorothy Graham: ''Software Test Automation.'' Addison Wesley, 1999, ISBN 0-201-33140-3
★ Cem Kaner, James Bach, Bret Pettichord: ''Lessons Learned in Software Testing. A Context-Driven Approach.'' John Wiley & Sons, 2001, ISBN 0-471-08112-4
★ G. T. Laycock: ''The Theory and Practice of Specification Based Software Testing.'' PhD Thesis, Dept of Computer Science, Sheffield University, UK, 1993, free PS-version
★ AppLab's CEO Sashi Reddi sees potential of global software testing Development Market
★ Hung Nguyen, Robert Johnson, Michael Hackett: ''Testing Applications on the Web (2nd Edition): Test Planning for Mobile and Internet-Based Systems'' ISBN 0-471-20100-6
Further reading
★ James A. Whittaker: ''How to Break Web Software: Functional and Security Testing of Web Applications and Web Services'', Addison-Wesley Professional, February 2, 2006. ISBN 0-321-36944-0
★ Lydia Ash: ''The Web Testing Companion: The Insider's Guide to Efficient and Effective Tests'', Wiley, May 2, 2003. ISBN 0471430218
★ Ron Patton: ''Software Testing'', Second Edition, Sams, July 26, 2005.
★ Andreas Spillner, Tilo Linz, Hans Schäfer (2007) 'Software Testing Foundations, 2nd Edition: ''A Study Guide for the Certified Tester Exam''' (Foundation Level – ISTQB compliant), Rocky Nook, Inc., Santa Barbara, California, USA. [ISBN 978-1-933952-08-6].
External links
★
★ Software Testing Q&A
★ Application Note: Structured Testing - The Path to Thorough Code Coverage
★ Software Test Management Guide, A to Z
★ CSTE Resources and Subjective answers....
★ Is Application Testing in Financial Services Making the Grade?
★ Software Testing tutorials & FAQs
★ Software Testing Life Cycle
★ Blogs on Software Quality & Testing
This article provided by Wikipedia. To edit the contents of this article, click here for original source.
psst.. try this: add to faves
Featured Companies
| Dancing Moon Travel | |
| Selloffvacations.com Oakville |
Software testing Videos
 | Mike Mullane Fall 2009 |
 | STPConFall08attendees |
 | FT2008 |
 | STP VIDEO FINAL |
