Certain words have different meaning to different people. So here are definitions of some words used during the testing of systems and applications. Differences in meaning can often be emotional because of a misunderstanding about the concreteness and formality of discussions.
• AUT is an acronym for Application Under Test.
• SUT is an acronym for System Under Test.
• CUT is an acronym for Component Under Test.
• Test results from running (executing) test cases are evaluated using a verdict. In test management systems, a verdict is a property of a test case within its test suite. Verdict values focus on whether test results give evidence for the correctness of the SUT:
o Pass = test results conclusively give evidence for the correctness of the SUT examined by a specific valid test case.
o Fail = test results conclusively give evidence that the SUT does not meet the requirements and specifications examined by a specific valid test case.
o Inconclusive = test results do not conclusively prove that a Pass or Fail verdict can be given. An example is when a test run stopped earlier than planned.
o Test Error = the test case or test system itself cannot properly examine the SUT. An example is when test script encounters a value not anticpated by the test case.
• An incident is an event occuring during testing that requires attention (investigation). [IEEE 1008] Incidents do not necessarily justify the filing of a formal Problem Report. However, information describing incidents are often attachments to a Problem Report.
Examples: "When a word is copied from Microsoft Word 2003 document and pasted into a text field, the AUT does not recognize the entry." or "Reports are not being produced." or "Users must wait a minimum of 5 minutes for logins to authenticate."
• A problem refers to one or more incidents with unknown underlying cause. This definition is used byITIL
• A Known Error is when the root cause of an incident is known and a temporary work around or permanent alternative has been identified.
• A failure is the misbehavior of some program or process which results in the AUT ending in a failure mode due to a set of conditions.
Examples: "The AUT does not recongize whitespace (a space character) at the beginning and end of text as extraneous and returns a soft error about text which visually appears correct to users."
o The failure mode of an incident is the physical or functional manifestation of a failure.
Examples: A system in failure mode may be characterized by slow operation, incorrect outputs, or complete termination of execution. [IEEE 610]
A soft error does not cause the AUT to terminate. The typical outcomes is that processing continues after an error message is issued. Optionally, a prompt may be issued so that the tester can decide whether to continue processing.
A hard error causes the AUT to terminate. Also known as an abend (abnormal ending) or a “Show Stopper” for the testing effort. Such defects cannot typically be reported by the program which contains them.
o The failure condition is what triggers a failure to occur. An example in functional testing is the input of a particular value or some specific user action such as clicking on an errant button. An example in load testing is the incremental number of simultaneous users or data items which causes the system under test to fail.
A bug is a slang term for the cause of faults. This term began early in the history of computers when a moth was found inside a computer and identified as the cause for a fault in the circuit boards.
• An issue is a point of disagreement requiring clarification and discussion. Issues are often about differences in mindset and philosophy that may be codified in a corporate Test Policy issued by managements.
Examples: "It's too much to ask users to remove invisibles space character automatically inserted when copying a block of text within Microsoft Word 2003." or
"It is unacceptable for users waiting over 5 minutes to login.".
• A contention is an assertion, such as a conjecture about the possible cause of a problem or a remedy to that problem. The word "should" usually appears in contentions.
Examples: "Text fields such as UserID should be stripped of whitespace (spaces, tabs, etc.) before being matched against the database." or
"The authentication module should not be single threaded."
I don't use this word to avoid confusion with the conflict between systems in load testing.
• A concern is a non-specific desire for an improvement in or the avoidance of threats to the AUT or work process related to it.
Examples: "I am concerned that login is too slow for our users".
• A defect is a specific deviation from expectations that usually requires corrective action to the implementation (programming code) or design (requirements) of the AUT. A formal Problem Report or ticket is usually raised for each defect uncovered. A defect may be manifested in several occurences(incidents). A statement defect should ideally be descriptive of the functionality not provided (or should not be sprovided) rather than prescriptive.
Examples: "There is no mechanism for controlling the verbosity of logging at run-time."
• A fault is less specific than a defect in the AUT. Like a geologic fault line which may or may not be the location of an earthquake, a computer system fault may indicate a potential for error rather than the existence of a defect in the product.
Examples: "The authentication module writes 250 lines into its std log detailing each of the 10 calls made to the authentication server per transaction. This slows processing time and can cause disk space overflow under load."
Verification vs. Validation
Testing is a process of reducing risk by comparing "what is" against "what should be".
Software verification is often confused with software validation. The difference between 'verification and validation:
Difference Verification
Validation
Asks: "Are we building the product right?"
Does the software conform to itsspecification? "Are we building the right product?"
Is the software doing what the user really need/want?
Focus: verifies that the final product satisfies or matches the validates that the product designsatisfies the
Basis: original design (from low-level engineering). intended usage (from high-level marketing).
Conclusion fromCapability Maturity Model (CMMI-SW v1.1)
the work products properly reflect therequirements specified for them. the product, as provided, will fulfill its intended use.
The aim of testing: Find errors introduced by an activity, i.e. check if the product of the activity is as correct as it was at the beginning of the activity. Declare whether the product of an activity is indeed what is expected, i.e. the activity extended the product successfully.
In the electronics industry:
Near the end of the Prototyping stage, after engineers create actual working samples of the product they plan to produce, Engineering Verification Testing (EVT) uses prototypes to verify that the design meets pre-determined specifications and design goals. This is done to validate the design as is, or identify areas that need to be modified.
After prototyping, and after the product goes though the Design Refinement cycle when engineers revise and improve the design to meet performance and design requirements and specifications, objective, comprehensive Design Verification Testing (DVT) is performed to verify all product specifications, interface standards, OEM requirements, and diagnostic commands.
Process (or Pilot) Verification Test (PVT) is a subset of Design Verification Tests (DVT) performed on pre-production or production units to Verify that the design has been correctly implemented into production.
Application Version, Release, Build, Handoff
The Application-under-test (AUT) is the software application that is currently being tested. But which part (components or features) of the AUT is being tested?
Which release and build and handoff?
• A Build is a single set of revisions to the AUT, created to correct defects in, or adds new functionality to, a previous revision.
• Developers may create several formal or informal builds before issuing ahandoff of the AUT for testing.
• Testing may be conducted on several handoffs before making a conclusion about a formal release of the AUT for duplication and mass distribution.
What user work process does the AUT seem fit to do? To whom should it be delivered to (the next test group, Business Users, etc.)?
Specify the detailed specific version of each component under discussion. Also specify the version of the requirements document which server as the basis for determining whether the application is behaving as expected.
Tester Authority
What is the responsibility and authority assumed by testers? This may be indicated by the word the tester uses to describe (record) his/her conclusionabout the AUT:
• Approved (after approval) - the AUT satisfies the standards of the person or oganization granting the approval.
• Authorized (after authorization) - the AUT has been cleared for use by a person with authority.
• Verified (after verification testing) - the AUT satisfies some quality standard.
• Certified (after certification testing) - the AUT has been passed a defined series of tests. Several experts in the testing field have advised testers to avoid using this formal term because the term can be misinterpreted to mean that the AUT is “fault-free” or that the organization performing the certification accepts some liability.
• Accepted (after acceptance testing) - the AUT satisfies some set ofrequirements, as in “Crest has been Accepted by the Dental Association”.
• Run Conditions
Define the conditions of the test run. Examples of this are:
• Platform characteristics (CPU type, RAM memory, etc.).
• Type of test run (maximum values, load test, time-out, etc.)
• Focus of test run (negative test, etc.)
• Date and time settings (all machines)
• The people involved (Developer, Tester, etc.)
This gives debuggers the information they need to narrow down the cause of the problem. Some problems take months to solve, bouncing from one person or team to another. So the more information the better.
Steps to Reproduce
Write step-by-step instructions to reproduce the bug. Number each step.
Example:
1. As user ___ from an NT4 Pro client, sign in the ___ server with a blank database.
2. Click menu item ...
3. Select From any account.
4. Enter ...
5. Select Hungary.
6. Click Next.
Notice that ...
Start each step with an active verb such as .click OK., .select checking account., .type value 99 in the Transaction Amount field., etc. The last step should reveal the concern stated in the Summary sentence (.Temp delay EOT.). Note whether .Temp delay. screens results in an End of Transaction or End of Session (EOS) which causes another logon.
To avoid needless repetition, produce a Test Specification document that defines the meaning of special words.
If a special program not used by typical users is needed to view results (such as a log viewer), note its use.
Analysis of Symptoms
Expore the extent of a bug's impact. State the result from various conditions under which the bug was found/verified. Examples:
• This occurred for all currencies.
• This did not occur for other countries (Spain, etc.).
• This occurred for account type A and B but not C.
• This occurred only for users with guest rights, not users with admin rights.
• This occurred only when values have not been specified. (Populated fields are OK)
• This occurred only for values loaded with thebatch back-end process. (Values entered interactively using the GUI are OK).
• This occurred for all functions using the underlying subroutine xyz.
Other examples:
• different routes through the application to the same point of failure (pulling down a menu vs. pressing a shortcut key),
• similar functions in the same application,
• different browsers,
• different software,
• different hardware configurations,
• different locales,
• different security contexts,
• previous versions,
• different run dates (accounting cycle, etc.),
Severity and Priority Levels
Indicate the impact each defect has on testing efforts or users and administrators of the application under test. This information is used by devlopers and management as thebasis for assigning priority of work on defects.
A sample guideline for assignment of Priority Levels during the product test phase include:
1. Critical / Show Stopper - an item that prevents further testing of the product or function under test. No workaround is possible. Examples of this include an installation process which does not load a component; a GPF (General Protection Fault) or other situation which freezes the system (requiring a reboot); or a missing menu option or security permission required to access a function under test.
Within a production test context, this category could also include incorrect financial calculations and even cosmetic items.
Major / High — a defect that does not function as expected/designed or cause other functionality to fail to meet requirements. The workaround may be to reboot the system or run a fix-up program/hassle. Examples of this include inaccurate calculations; the wrong field being updated; the wrong rule, phrase, or data being retrieved; an update operation that fails to complete; slow system turn-around performance; or a transaction journal record which fails to occur as expected.
Average / Medium — Annoyances which do not conform to standards and conventions. Easy workarounds exists to achieve functionality objectives. Examples include incorrect/missing hot-key operation; an error condition which is not trapped; or matching visual and text links which lead to different end points.
Minor / Low — Cosmetic defects which does not affect the functionality of the system. Examples of this include misspelled or ungrammatical text; inappropriate or incorrect formatting (such as text font, size, alignment, color, etc.); or inconsistencies between product and documented text or formatting.
Enhancement — Additional features that would improve the product for users, administrators, or other stakeholders.
Emergency is a term defined in ITIL.
Managementmakes the decision whether an application should be shipped in light of the number and types of defects open.
Owner Assign To
Understand the organizational structure of those who will work on resolving your bugs. Consider putting the name of the person who is first most likely to work on each particular bug.
Most organizations who value fast turnaround of bug fixes prefer this approach.
However, some (usually complex) organizations prefer all bugs for a project to flow through a coordinator or manager who then assigns the individuals to work on each particular bug.
For example, if the development team is divided by people who write reports and people who write GUI code, analyze bugs so that you can specify who should review each specific bug report.
"GUI dialog xxx does not ... "
"REPORT yyy does not ... "
"REQUIREMENT yyy.xx does not define ... "
This avoids the common situation of a bug report from being closed by one person for a single sub-item when additional sub-items going unsolved.
Summary / Headline
Follow these general rules when crafting statements:
Be specific. Don't use vague phrases such as "problems with", "is incorrect", or "issue with". State the expected behavior which did not occur - such as "after pop-up ___, ____ did not appear." and the behavior which occurred instead.
Use present or past tense. Say "appears" instead of "will appear", which may be confusing to readers.
Don't use unnecessary words such as "apparently". State observed facts.
Don't add multiple exclamation points!!!! (We do want to help.) End sentences with a period.
DON'T USE ALL CAPS (That's the same as shouting.) Format words in upper and lower case (mixed case).Other examples are:
• different routes throught the application to the same point of failure,
• different browsers,
• different software and hardware configurations,
• different locales,
• different security contexts,
• previous versions, etc.
• Status of Incident Resolution
• similar functions in the same application,
A defect item attains these states of status:
1. A new defect item is created with aEntered,Started, orSubmittedstatus. NOTE: Avoid using the word "New" for this state because newness can be calculated by the date created.
2. A test supervisor may confirm the need for the bug item and place it inAssignedorOpenstatus.
Alternately, an item may be be called inEvaluationstatus while the item isassignedfor analysis by a developer or whoever "owns" the bug. Some organizations also add an intermediate "Prioritized" status.
3. A tester may place a defect item intoWithdrawnstatus items whichduplicateanother defect item already reported or is the result of tester misunderstanding about how the application should operate.
4. Management may allow the application to be shipped “as-is” with the defect by marking certain defect items as:
a. Refused/Rejected(developers feel reported item is not recognized as a defect); Some automatically reassign this to the submitter's boss.
b. Waivered(defect is accepted, but no action will be taken); or
c. Forwarded/Postponed/Deferredto a future release
This usually happens because of time or cost constraints.
5. Analysts place a defect item intoDevelopmentstatus when analysis is complete and changes are being made to the application by developers.
6. Developers change the status of a defect item toTestingafter unit testing. This signals to testers that application changes can be tested again. Such items are ideally noted on a Hand-off memo ("manifest") from developers to testers.
7. If changes by developers did not fix the problem reported, that defect item can beReturned or Reopenedto development. Another defect item is created if a fix introduces another problem.
8. After a tester verifies that a defect has indeed been fixed, the defect item isFixed. NOTE: Avoid the wordClosedbecause it can be confused with other stuatus such as Refused/Forwarded, etc.
No comments:
Post a Comment