Software Validation, Verification and Testing

Course Notes - First Midterm

! !

Fall 2010

Syllabus
CSE 565 Software Verification, Validation and Testing

8/19/10 3:52 PM

Topics
01. 02. 03. 04. 05. 06. 07. 08. 09. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. Week 01 - Testing background Week 01 - Testing process activities Week 02 - Requirements-based testing techniques Feature interaction testing Exploratory testing Structure-based testing techniques Integration testing System testing Regression testing Usability testing Performance testing Security testing Design of Experiments Testing tools Software Reliability Engineering Reliability models Statistical testing Test planning Risk based testing Tracking testing progress Testing measures Test documentation Testing standards Test process improvement Test process patterns

[week 01 Class 01 Thursday]

Testing Background

8/19/10 3:52 PM

1. Objective of Testing a. b. c. d. Minimize risks Find faults (things that not works) How robust is the software: Do not fail and if fails it recover very fast Assess performance

2. When To Stop Testing? a. - Demonstrate requirements are met b. - When it is good enough????

Monthly Meetings Of IEEE-CS and ACM - Sponsor the main conferences - Phoenix (search for it!)

3. Definitions IEEE RELIABILITY: Probability that a give SW application will operate for some given time period without SW failure. MTBF - mean time between failure. It is a related concept with Reliability. Agencies which certified SW. Each of them have standards: FAA. Aviation Agency. Standard D0178b. Probability (10^-9). B787 is another standard (Boeing) FDA is another agency.

VALIDATION: Tends to answer the question "are we building the right product? Is this software do what it needs to do? (cumple requerimientos de usuario) VERIFICATION: Are we building the product right? Does this meets it requirements. It is the easy one. (esta implementado acorde a diseo) Industry talks about V&V concepts which stands for Validation & Verification. Other concept is IV&V which stands for Independent V&V

TESTING: Examination of the behavior of a program by executing it on sample data sets. Testers VS Pollsters Analogies [Poll - encuestas] [Scientific pollsters such as CNN, NBC News. They take a representative group for the poll, where the demographics are well defined which means that the MOE (margin of error) is less than 3.] Pollsters are paid to make predictions, while Testers . does tester make predictions?. Testers are expected to do predictions about the reliability of the SW. Both Testers and Pollsters works with significant constrains, mainly money and time. Pollsters cannot do an exhaustive poll. Testers cannot either do exhaustive testing. Some cases maybe when the SW is very simple and has very specific inputs.

If you cannot make a exhaustive poll then you have to assure you have a good sample (representative and valid statistically). The same happened for a test process.

4. Effective Testing Examination of the behavior of a program by executing it on REPRESENTATIVE sample data sets. ROI (Return of Investment): Is a development effort but also a V&V effort. How many effort we put on doing the development and the V&V. Time used for different activities 1/3 - A&D 1/3 Coding 1/3 - V&V Cost = f(productivity)The SW productivity is 1 LOC / hr. ROI, on software is all about the effort we put on doing the V&V.

5. Measures Of Reliability CRUD - Customer Reported Unique Defects. It is expressed by defects/KLOC. In 2010 the average is 1/KLOC. defects/Kloc = 1 or 0.1 (2010) Another way to measure this is using 6sigma =). (six sigma) = 3.14/million. Even here is A LOT!

See The Risk Digest http://catless.ncl.ac.uk/risks

[week 01 Class 01 Thursday]

Testing Process Activities

8/19/10 3:52 PM

1. Software Testing Process vs. Development Process a. b. c. d. Development process Requirements [What to build use case scenarios] Design [Pick the Arch patterns. How to meet requirements] Implementation V&V These 4 activities on development has a analogy on testing process. - Testing process a. Test objectives (plan, conf management) b. Sampling strategy [Test Design] c. Write test cases and test scripts d. Execute test Testing is a mature process and requires specifics skills. There are some other activities that are analogous and that affects both, testing and development: a. b. c. d. Planning Configuration Management Changes of requirements Review

2. Testing Limitations And Constraints 1. Constraint by the requirements (missing or bad reqs by example) 2. exhaustive testing is not possible 3. Limited time and money

3. Testing Attitudes Independence. IV&V, independence test, or independence team. Customer perspective: understand the customer; know about the domain. Demonstrate that the system works. Go through all the requirements and be sure that we meet them all. This is kind of clean process. Dirty testing (try to break the system) Professionalism. Testing should be an add value to the product. Should be something where a person can have a career. The different paths of career on SW could be: development, mgnt, testing. It could be a good idea to get a Licensing of SE, there is a certification on Texas (fee, test, and educational background). The test covers: development and reason about correctness (testing, implementation, prove of correctness).

Note: A good test engineer has: - an ability to take the point of view of the customer, - a strong desire for quality, - a 'test to break' attitude, - and an attention to detail.

Homework
Homework for Tuesday
3 first readings of the list of the readings:

8/19/10 3:52 PM

Classic Testing Mistakes, Marick, Brian,

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34.9940&rep=rep1&type=pdf

Software Testing Best Practices, Chillarege, Ram,

http://www.chillarege.com/authwork/papers1990s/TestingBestPractice.pdf

The Software Quality Challenge, Watts S. Humphrey,

CrossTalk, June 2008.

[week 02 Class 02 Tuesday]

8/24/10 2:48 PM

0. From the last class

Reading Papers ! Classic Mistakes in Testing ! Testing Best Practices ! Crosstalk about Quality Testing Attitudes ! Professionalism: testing is a carrier, as develop and as management. ! License as SE in Texas! :-0 (develop and reasoning about correctness). Include testing, Proof of correctness and Implement ! Testing is not a course into university :-0 but there are a lot of books!

1. Testing Principles
a) Do not throw away your test. They are going to be useful in the future. Reuse, maintain your product (the product will be out for a while, it is better to save your case of testing) regression analysis. b) Probability of the existence of more errors in a section of code is proportional to the # of errors already found. ! Remember we have 1 error/Kloc ! Errors tend to generate clusters (because complexity, dependences of the code, skills of the developer of that section of code, poor or changing requirements that affect that part). c) A necessary part of a test case is the definition of the expected result A test case consist on: input and expected results. ! Example: Testing a new optimized C++ compiler. The input would be a file1 with code, and the exit a file2 with code that satisfied defined characteristics.

level of independence

d) Faster and Cheaper Lean manufacturing, lean testing, lean software development :-o ! This has to be with technology, techniques, and reuse.

2. Paper 1. Classic testing mistakes by Brian Marick

! ! Brian Marick is a king of leader in the area [Page 24] Some classic testing mistakes 2. (Role of Testing). Thinking that the purpose of testing is to find bugs. There is not problem in the state. However we should remind that the objective of testing is to find important bugs. 4. (Role of Testing). Not reporting usability problems. It is common to report only functionality errors. Remain that the user do not use a tool functionally but by tasks. 7. (Role of Testing). Starting testing too late (bug detection not bug reduction). Testing should begin in front. We should write test at the same time we are writing requirements. You write your test cases before you write your code. The idea is that developer have more information about what it is expected. 10. (Planning the complete testing effort). Putting stress and load testing off to the last minute. It is a problem due the fact there would not be enough time to fix any issue. 11. (Planning the complete testing effort). Not testing the documentation. Normally the testing is done to the requirements, but it is also necessary to test the documentation (installations procedures and configuration instructions).

19. (Personnel issues). Testers are not domain experts. We need to have the people who really understand the application. 23. (Personnel issues). A physical separation between developers and testers. Separated groups. It is not good to have this separation.

3. Paper 2. Testing Best Practices by Ram Chillarege

Report 28 best practices 1. (Incremental) Asses software reliability via statistical testing. One way to talk about reliability is MTBF (Mean Time Between Failures), the only way to measure this is through statistical testing. 2. (Basis) Develop an agile test design. Agile means: flexible, short cycles, small pieces at the same time. 3. (Foundational) Utilize model-base testing-techniques Whle working on the development we use Model Base Software Engineering (MBSE) and Software Development, this means have models such as UML, state machine, Petri nets, and then from here move to the code. UML - "state machine" go to test and code! Something similar happen for testing. We need models and the move from there to testing cases.

4. (Incremental) Cross-functional teams. It is a very good practice to have tester and developers together at the same time. Use (IPDT) - Integrated Product Development Team

5. (Foundational) Use ODC (Orthogonal Defect Classification). This technique is used to improve test effectiveness. For each founded defect we will have attributes associated with it. Defect Attributes Defect classification.

6. (Basis) Automated Test This is the first homework "

7. (Foundational) Perform scenario-based testing. Scenario comes from Uses cases and Use-case modeling. 8. (Foundational) Emphasize usability testing 9. (Basis) Test over multiple platforms

! !

4. Paper 3. Software Quality Challenge by W. Humphrey

Some analogies: Compare SW with other human generated media (paper/book) 1 defect per page. (software) 1,000,000 LOC 40,000 pages / errors [Page 4, paragraph 3] The problem with the software is that there is a lot of combinations and permutations of possible errors. So many permutations to test all cases software options State of the art in SW development. 100 defect / 1000 lines Then we filter errors: first the compiler, then static analyzer (search for error patterns), then inspection, then unit testing (UT), then Integration Testing(IT), then System Testing (ST) And we can obtain 10-20/1000 error per LOC

And then finally some how 1 /1000 error per LOC Then worry about variation en configuration Review the Table 1 and Table 2. (testing variations and possible paths through a network). Think on how to test all possible options on the software. [Page 5] The eight elements of software quality management 1. Establish quality policies, goals and plans. (V&V). 2. Properly training, couch and support. Make sure that all the team understand the principles, objectives and tools for testing. 3. Establish and maintain a requirements quality-management process (V&V). 4. Establish and maintain statistical control of the Software Engineering process. 5. Review, inspect and evaluate all artifacts (test). Evaluate the products and process! 6. Evaluate all defects for correction and prevent other similar problems. (ODC - Orthogonal Defect Classification) 7. Establish and maintain Configuration Management (CM) System (testing). 8. Continually improve the Process. (testing), postmortem and root case analysis (CMMI, PSP)

This course is going to be focused on Software Reliability!

[week 02 Class 02 Tuesday]

8/24/10 2:48 PM

1. Testing Techniques
There are two types of testing techniques ! Functional testing: (black box oriented = tests the functionality of an application as opposed to its internal structures or workings). This are applicable on any level. Typical black-box test design techniques include: Decision table testing, All-pairs testing, State transition tables, Equivalence partitioning, Boundary value analysis. ! Structural Testing: (white box oriented = tests internal structures or workings of an application as opposed to its functionality). The tester chooses inputs to exercise paths through the code and determine the appropriate outputs. It is analogous to testing nodes in a circuit White-box test design techniques include: Control-flow testing, Data flow testing, Branch testing, Path testing. We are going to review some of the Functional Testing techniques.

2. Functional Testing (black box)

Useful for all levels of Testing! a) Scenario-base Testing: directly tined to use-cases (use case driven testing). The requirements are expressed on use-cases. (histories with several steps)

Example: software for AMT system; examples of use-cases: withdraw, deposit, check balance. For the use-case withdraw could be different scenarios: normal (sunshine case) alternative 1(no cash) alternative 2(wrong PIN) alternative 3 (bad something) Sampling Strategies for used for case-based testing b) Requirement-base testing. Is an alternative for Scenario-base testing. The requirements are in a document as a list of Shalls (system shall do this), then for each of the req. we match one or more testing (we use a matrix of reqs vs test). This are functional requirements Make a table with requirements and testing associated with each requirement. (a table)

3. Structural Testing (white box)

4. Testing Levels
! ! ! Unit Testing Integration Testing System Testing

Homework

8/24/10 2:48 PM

Software Unit Testing Frameworks Due: September 7 Write a Paper about JUnit

[week 02 Class 03 Thursday]

8/26/10 2:57 PM

1. Functional Testing 1.1. Sampling Strategies (Scenario| Use Case)-Based 1.1.1. Equivalence Partitioning

technique that divides the input data of a software unit (function, service, component, program) into partitions of data from which test cases can be derived. In principle, test cases are designed to cover each partition at least once. This technique tries to define test cases that uncover classes of errors, thereby reducing the total number of test cases that must be developed.

!"""""""""""""#"""""""""$"

Example: Domain = integer 32 bits Test Cases: 1. 2. 3. 4. 5. -5 -> 5 0 -> 0 5 -> 5 max min

Partition: negative, positive and zero. Take a sample of each partition (valid and invalid partition) Invalid Partition greater than max less that min characters float

2. Equivalent Partitioning Technique

1. Identify Partition. 2. Write test covering as many uncovered valid partitions as possible. 3. For each invalid partition write a test that cover one and only one at time. ! Assumption: Input are Independent. BUT this assumption is normally not True.

Example One: Test a function f(x, y) that receive two input values in the ranges of X: 1 .. 10; Y: 50 .. 100 Partitions X 1...10 <1 >10 Y 50100 <50 >100 V I I V V I I V V I I V I I V V Test cases 1 2 3 4 5

remember step 3, only one invalid value at one time Test 1 2 3 4 5 X 5 -1 5 Y 55 60 45

EP requires at least 5 test cases for this example (according with the three steps described before)

Example Two: (scenario testing) ATM system: scenario one withdraw; scenario two deposit; scenario three - atm pin change. Let us see a simple use case for amt pin change. And try to do scenario-base testing: User System 1. validate the user 2. display options 3. Pin change 4. Prompt new pin 5. enter pin 6. Validate pin 7. Reenter 8. Confirm/Update

Scenarios: a) normal one b) alternative 1 (format problem) c) alternative 2 (pin not match) How to test normal and alternative scenarios? We need extra requirements What is a valid PIN? 4-8 digits 1st and last must be different cannot have 3 or more digits the same

For this create equivalent partitions (EP). Our INPUT is PIN Think about ATTRIBUTES like: length - 1. 4-8 is (v) - 2. <4 is (i) - 3. >8 is (i) first/last - 4. They are different (v) - 5. They are the same (i) number of same digits - 6. <3 (v) - 7. >=3 (i) AS developer you think in architecture, design and algorithm AS tester you identify partitions and make sampling Then create coverage matrix: Partitions Length 4-8 <4 >8 First/Last Different Equals # same digits <3 >=3 V I V V V V I V I V V V I V V I I V I I V V Test cases 1 2 3 4 5

Then create test data: Test 1 2 3 4 5 PIN 1234 (VVV) 123 (IVV) 123456789 (IVV) 1231(VIV) 244443(VVI)

Testing is all about sampling! For the normal scenario 1 test {t1} For alternative 1 (invalid pin) {t2, t3, t4, t5}

Problem (in class): A Payroll System with 2 input: - Id (5 digits) - Hours Worked And three Output - Id (5 digits) - Type employee, is based on the Id 00000 - 29999 engineer 30000 99999 support - Gross pay $10 hr for 1st 40 hr $15 hr after 40 hr Additional information: - max number hours a person can work 100 hr. If exceeded then send error. - min number 0 hr - format of the hrs is integer and round down!

Testers need Requirements as Development!

Create coverage matrix: Partitions Id 5 digit Engineer Support <5 >5 ~digit Hours 0-40 41-100 <0 >100 ~integer V V I I I V V I I I V V V V V V I I I V V V V V V V I I I V V V Test cases 1 2 3 4 5 6 7 8

Remember the steps: 1 the matrix; 2 the valid as many as possible; 3 one invalid at time. Valid cases are tested with two test cases Then go for the invalids (only one each time).

Homework
0. Not collectable Homework
On line registration system for university Inputs ! course # (5 digits and exist in course database) ! semester # (1,2,3) ! enrollee 1 year in advance Outputs ! status of the class requested (cancelled, open, close) ! error messages based on input

8/26/10 2:57 PM

[week 03 Class 04 Tuesday]

8/31/10 2:59 PM

0. From the last class

! We talk about Functional Testing Scenario-Based (Sampling Strategies) Equivalence Partitioning and work with some examples Let us check the homework about the Registration System

Partitions Line # 5 digit in DB <5 >5 ~digit 5 digit not in DB Semester # 1 2 3 other Output (status) Cancel Open Closed V V V V V V I V I I I I

Test cases 1 V 2 V 3 V I I I I 4 5 6 7 8

V V V I

V V V

Output are test variations. We want to try each of them. They are like States. Create samples for each semester and each Status. But not combinations, because we assume are independents.

1. Paper 4. An Introduction to Scenario Testing by Cem Kaner

! ! ! For each scenario 1 or more test Scenarios are histories that are motivating, and credible, complex, and easy to evaluate. Differences between requirements analysis and create test cases: Tester exploits disagreement.

The main idea is to create histories (original and also taken from others similar systems) with objects inside, with user (good and bad) interacting with the objects, with system events

1. Functional Testing 1.1. Sampling Strategies (Scenario| Use Case)-Based 1.1.2. Boundary Value Analysis
! Each of the inputs of the scenario must be tested with max, min and values around the max and min. (test the edges and around the edges of EP) ! Example (Payroll system from the previous session): Partitions Hours 0-40 41-100 <0 >100 ~integer V V I I I 0-40 ; 41 -100 ; >100

(1) Test the Edges means: <0 ; (2) Test around the Edges:

-1, 0, 1 39, 40, 41 99, 100, 101

3. Paper 5. A review of Boundary Value Analysis Techniques by David J. Coe.

! Fault (in Spanish: defecto) incorrect step , process or data definition in a computer program. A fault causes Failure. Detect fault before cause a Failure. Failure (in Spanish: falla) inability of a program or component to meet its function. ! Software testing identifies Failures, which indicate the presence of one or more faults. (The goal of testing is to find faults!) {FAULT - tolerant} Software Detect the fault and avoid the Failure! Single Variable and Multi-Variable

! !

4. Merging EP and BVA

! ! ! EP where input independent. BVA where EP x BVA (both at the same time)

Go back to the problem:

Partitions Hours 0-40 V

41-100 <0 >100 ~integer

V I I I

EP defines the following sections <0 0-40 41-100 >100 BVA defines the followings values for each section -1 0, 1, 39, 40 41, 42, 99, 100 101 Hit the border and around there. BVA is done with INPUT but can also be about OUTPUT Example: Suppose that the maximum check amount that the system should generate is $9,999.99 [max] if the program is Input Process Output then create input that generate $9,999.99 9,999.90 OK --- NO .97 10000.00 reject --- NO .03 Output BVA is testing (validation and verification) but is more about Review & Inspection.

1. Functional Testing 1.1. Sampling Strategies (Scenario| Use Case)-Based 1.1.3. Decision Trees, Decision Tables and DOE
! ! DOE = Design of Experiments This family of testing techniques are for test combinations of DEPENDENT Input! Example Inputs: - Customer [A,B,C] - Order [1- 1000] Output: - Discount Rules: 1. Customer A gets 0% discount to < 10 gets 5% discount to 10 99 gets 10% discount >= 100 2. Customer B 5% < 10 15% 10-99 25% >= 100 3. Customer C 0% < 10 20% 10-99 25% >=100 With EP, we can do 3 testing cases Partitions Test Cases 1 2 3

Customer A B C Order 1-9 10-99 100-1000 V V V V V V V V V V V V

The question is, are those three cases enough to test the whole system. Is it possible to ship the system with only this test? The answer is NO! But, in this case the type of customer and the amount of the order are not independent. So it is not possible to use EP.

Decision Tables. 1 A B C 1-9 10-99 10-100 X X X X X X X 2 X 3 X X X X X X X X X X 4 5 6 7 8 9

No think about a problem where you have five inputs values that are not independent. F (A, B, C, D, E) and A have 4 partitions, B 5, C 10, D 4 and E 3 this makes 2400 test cases table, which is huge. One way to reduce this is reduces the terms. For instance reviewing the partitions of C we can reduce from 10 to 3.

Other way is to reduce the number of inputs, and instead of testing all 5 inputs at the same time, we look for which inputs should be really tested together. And we might find out that A,B,C has to be tested together but not with D and E. ! Example In the previous example: If we add the D (I) case to the customer, and we add the values out of the boundaries, this make 4 partitions for customer and 5 to order amount, what makes 20 test cases. But in fact instead of having the 20 of them, we can go with only 12, the 9 basics and then only three extras WITH ERRORS INDEPENDENTS! 1 A B C D (I) 1-9 10-99 10100 <1 >1000 V X V V I V X V V I I X X X X X X X X X X 2 X 3 X X X X X X X X X 4 5 6 7 8 9 10 11 X X 12

This technique becomes practical doing assumptions Columns becomes test cases!

Decision Tree ! Customer

A ! 1-9 ! 10-99 ! 99-100 ! ! ! 1-9 10-99 99-100 1-9 10-99 99-100

o B

o C ! ! !

NOTICE that the decision tree generates the same 9 test cases as the table decision. Each PATH correspond with one Test Case! Trees and Tables are analogous!

Homework

8/31/10 2:59 PM

0. Not collectable Homework

Online Ordering System Input ! product # ! customer ID ! quantity Output ! shipping cost Rules to calculate the cost 1. If customer is a 1st time user, free shipping world wide. 2. If the product is a featured product or quantity is >100, then free US shipping. 3. Otherwise standard shipping rates. The partitions are (only valid cases): a. customer (first time or not first time) b. product (featured or not) c. quantity (<=100 o >100)

As Tester study the problem, study the Inputs and attempt to think about variations and model the test cases.

The Decision Tree ! customer 1 time o free earth shipping (1st case) ~ 1 time o product ! featured

Shipping at USA free us shipping (2nd case) " Nota USA Standard Shipping ~featured " qty >100 o Shipping at USA ! Free US Shipping (3th case) o Not a USA ! Standard Shipping <=100 o Standard shipping (4th case) "

The test cases are the PATHS This tree suggest 4 or 6 test cases!

The Decision Table 1 First time Yes No Product Featured Not Featured QTY <100 >100 V V V V V V V V V V V V V V V V V V V V V V V V 2 3 4 5 6 7 8

This table suggest 8 test cases!

Why?

[week 03 Class 05 Thursday]

9/2/10 2:49 PM

0. From the last class

! ! Last class we talk about different forms of testing and use 3 techniques, combinatorial, decision table and trees. We finish with a simple problem about online order system and we obtain 4 test cases with decision tree and 8 with the table. Putting all Together: EP+BV+DTT The tree in the product node -> featured -> ship USA split in two cases: Yes and No. Yes -> Free No -> Standard Shipping The same happends to Qty node ! The tree have 6 paths customer o 1 time ! free earth shipping (1st case) o ~ 1 time ! product " featured Shipping at USA o free us shipping (2nd case) Nota USA o Standard Shipping " ~featured qty o >100 ! Shipping at USA " Free US Shipping ! Not a USA " Standard Shipping o <=100

! ! Decision table
1 1 time Yes No Product Featured Not F. USA Add Yes No QTY <100 >100 V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V
st

Standard shipping (6th case)

10

11

12

13

14

15

16

V V V V V

V V V

V V V

V V

V V

V V

V V

We have 16 vs. 6? Remember that for first time user, I do not care about Anything? Most of the time 6 test cases would be sufficient, however we might want to do the 16 just to be sure that the programmers do not do anything wrong.

1. First Problem in Class Scenario-based Testing to a Tuition System Inputs - Status (instate, outstate) - Hours # - Dependent of university employment

Rates - In-state: $50/hr for first 12hrs and $600 >12 - Out-state: $500/hrs for first 12hrs and $6000 > 12 - Dependant: Receive 50% discount not matter if it is in-state or out-st

Decision Tree Status o In-state ! Dependant " Yes Hours <=12 o $25/hr " No Hours <=12 o $50/hr Hours >12 o 600/hr Hours >12 o 300/hr

o Out-state ! Dependant " Yes Hours <=12 o $250/hr Hours >12 o 3000/hr " No Hours <=12 o $500/hr Hours >12 o 6000/hr ! Decision Table 1 Status 2 3 4 5 6 7 8

In Out Hours <=12 >12 Depend Yes No

X X X X

X X

X X X X

X X

X X X X

X X X X

In this case both tree and table generate the same amount of test cases. When building a tree it could get different number of test cases depending on with which input you start. The table is recommended to force us in think about combinations! But Trees show sequences!

2. Second Problem in Class Combining EP x BVA x DT See the document: Data Base Exercise. (entry validation) First step: look the inputs. All the inputs are the ones underlined. 1
Part # In list No in list Part status code New Old Unknown Req. Inventory V V V V V V V V I V V

3..5

Positive Integer Spaces Supplier In DB and PP In DB not in PP NO in DB Auto Reorder Flag Y N Reorder Qty Positive Integer Other Remaining Inv T Positive Integer Other Purchase Terms Positive Integer Blank Part price Dollar value Other

V V V

V V

V V

V V

(same color means dependency) They are not independent so not use EP Decision Table could generate 1152 combinations :-( But they are not all dependent. So we can combine! Independents: Part # y Part Price. Dependents Part Status Code y Req. Inventory. Dependents Supplier y Purchase Term. Dependents Auto Reorder Flag, Reorder Qty y Remaining Inventory. ! ! ! We are going to have 3 to 5 valid Test Cases, creating valid situations Case 6 is an invalid option for input independent 1 Case 7 is an invalid combination of input 2 and 3 that are dependents

! !

Create Valid Inputs and Valid Combinations Then Crete Invalid inputs and Invalid Combination separately. The final table has approximately 5 test cases for valid and a total of 13 to 14 test cases.

And at the end BVA :-o This three techniques together in real problems

Homework
Read the papers about model based Testing. 1. Gold Practice: Model Based Testing 2. A survey of Model Driven Testing Techniques 3. Model Based Testing Using Software Architecture

9/2/10 2:49 PM

[week 04 Class 06 Tuesday]

9/7/10 3:00 PM

0. From the last class

! We have been working with sampling data and synchronous testing techniques. REQ BASED TESTING TECHNIQUES are applicable to all levels: Unit Testing, System Testing, etc. For sampling we see: EP, BV, Decision tree and Decision table

1. Asynchronous Testing

The king of testing that we will when there are input that can take place in any point in time during the execution. Start wit a simple example TIME LINE: system engineering tool for looking in a scenario. Example One: ATM system in a normal scenario. Sequence: 1. Customer insert a card 2. Customer request withdraw 3. Update account 4. Dispose cash 5. Take cash

! ! !

But, what happens if: 1. lost of power (in any point of time)

Example Two: The call waiting feature in a cel phone. A talking with B, C tries to call A. We will have a time line for a normal scenario. 1. A enter digits for B 2. A presses send button 3. 4. 5. 6. Network connects to B B line is ringing B answer A and B taking

During this time line could happen synchronous events. For instant, C calling A.

BUT what happened if: C could call A during different point in the time line. ! ! ! ! C calls when A is entering the digits. As phone will ring. C calls just in the same time A hit the send button. A will answer Cs calling. C calls when the network is trying to connects B. You might lost the call. C calls when B line is ringing. If A get the call, does B get on hold ?

These examples are to illustrate these techniques scenario-based. And see what happen when an asynchronous event happen during a time-line. These asynchronous events could just happen or interrupt the time-line.

[week 04 Class 06 Tuesday]

9/7/10 3:00 PM

1. Functional Testing 1.2. Requirement-Based Techniques 1.2.1. Model-Based Testing

(Paper 6,7,8) Use the model to generate the solution UML tool to create the model of the system And we can create the implementation from the model!

2. Paper 6. Gold Practice, Model Based Testing

! MBT is the automatic generation of efficient test procedures using models. Model, expected input, expected output, run test, compare output. Techniques to build the model: decision table, finite state machine, grammars, Markov chains, state charts (UML).

! !

3. Paper 7. A survey of Model-Driven Testing Techniques

! ! UML for models UML Testing Profile (extension of UML) (BPEL) Business Process Execution Language. Used for services! Both describe behavior! Survey: 1. Based on languages used 2. Degree of automatic generation of test scenario 3. Testing target: - implementation code

- abstract model (simulation) 4. Tools supports ! ! SUT (system under test) For Software Architecture: Requirements -> Design in UML For Software Testing: Requirements -> Design in UML Testing Profile

But this is not the goal of the class!!

4. Paper 8. Model Based Testing Using Software Architecture

! Software Architecture - use UML - or use ADL (architecture description language) Syntaxes and semantics The paper talk about a technique called acme develop by ABLE/CMU Example with a client / server system The idea connect model and test UML Test Profile feed into Tools, like JUnit

! ! ! ! !

Now let put the theory in practice!

5. State Machines

We are going to illustrate how to do Model-Based testing with State 23%405"6#*%)"(*%7% Machines !"#$%&'(%"% .(('(% )"(*% Example: ATM Machine =3%>$'6.0%)"(*% ?% 83%9"6#*%)"(*%7% ,(':,$%&'(%"%;4<% +",$-(.% $/.%)"(*% A% D3%@"*%;4<%7% E.$(B% @% F3%9"6#*%%;4<%

!"#$#01%&'(% "%,#0%

D3%@"*%;4<%%

!"#$#01%&'(% "%,#0%7%=0*% $(B% D3%@"*%;4<%% +%

F3%9"6#*%%;4<%

!"#$#01%&'(%"% $("0C")$#'0% G%

!"#$#01%&'(% "%,#0%7%80*% $(B%

F3%9"6#*%%;4<% H%

So if this is the way the system works Test Each Path. For each state try each input or stimulates

State machine Coverage: to reach each state and test each stimuli or event from the state. We need to be sure that each state is tested and covered.

Try to test each event/state/event combination (3, 1)(4,1) (3,2)(4,2) (3,7)(4,7)

Let us see a Simple Way to attempt to achieve this coverage

Testing cover creation ! 3 step process: 1. develop a state testing tree from the final state machine 2. identify test sequences from the tree (paths) 3. develop tests that the contain the sequences (start " end/terminal state)

STEP 1. Develop State Testing Tree 1. Start state = ROOT. From the example it would be A. 2. 1st. level. Identify the events and new states reached from the root. From the example it would includes: A ->1 -> A (A-1) A-> 2 -> F (F-2)

A-> 3 -> B (B-3) where the first letter is the state reached and the number indicates the event. 3. 2nd and remaining levels, expand each state not previously expanded in the tree B -> 4 -> D (D-4) B -> 5 -> C (C-5) C -> 4 -> D (D-4) C -> 5 -> E (E-5) E -> 4 -> D (D-4) E -> 5 -> F (E-5)

STEP 2. Identify Test Sequences (Path) The terminal states in this example are D and F. 1 2 3,4 3,5,4 3,5,5,4 3,5,5,5

Step 3. Develop Test Cases That Contains The Sequences We need to include the sequences in the exact order!! T1 T2 T3 T4 could could could could be be be be 1, 3,4 this include (1) (3,4) 2 3,4,5 3,5,5,4

T5 could be 3,5,5,5 The goal is not the minimum but if we can it is ok. In this case we have 5 test cases to cover 6 paths. 3,4 is different from 3,5,4 (remember same order)

Homework

9/7/10 3:00 PM

Read the Next Papers

09. The AETG System, An Approach to Testing Based on Combinatorial Design 10. Automated Combinatorial Test Methods Beyond Pair wise Testing 11. An innovative approach for Testing Bioinformatics Programs Using Metamorphic Testing

[week 04 Class 07 Thursday]

(" )"

9/9/10 2:56 PM

1. Continue '" with Model Based Testing Techniques.

#" $" *" /" !" ," &" +" -" %" ."

STEP 1. Build the tree A -> 1 -> B A -> 2 -> A A -> 3 -> C B-> 4 -> A B-> 5 -> D C ->9 -> E D -> 6 -> C D -> 7 ->D D -> 8 -> E Do not extend previous visited nodes! Take care of LEVELS

STEP 2. Sequences 1,4 1,5,6 1,5,7 1,5,8 2, 3,9

STEP 3. Test Cases All of them should start with A and finish on E 1,5,8 3,9 1,4,3,9 2,1,4,3,9 1,5,7,7,8 1,5,6,9 2,3,9

[week 04 Class 07 Thursday]

9/9/10 2:56 PM

1. Functional Testing 1.4. Design of Experiments (DOE) ! So far we have covered this: EP BVA Decision Tree and Decision Tables Model-Based Testing State Based Testing (state machine) Now we will review a new technique: Design of experiments (DoE) Sometimes called Pair-wise Interaction testing We have to be careful when we use this tech Is sound cool but some times could be difficult and cause more problems than solutions This is for decision tables that are Too large (a lot combinations)

! ! ! !

Example One: Let see the following problem: Performance testing for a car: Engine (3 options): 3.0, 3.8, 5.0 Transmission (2 options): manual, auto Body style (2 options): 2Doors, 4Doors Tires (2 options): normal, high performance

Having all this inputs we will have 24 possible configurations (3*2*2*2 options = 24 configurations) But It could be exhaustive DOE ( it is a sampling technique). Instead of test all, use small sample.

! !

DOE Pair Wise Combination Testing Steps: 1. Identify the inputs parameters to the SUT (systematic under Test) 2. Partition the inputs/ create samples (remember EP) 3. Specify constraints prohibit combinations. In some systems something can not happen together, for instance having two buttons to be pressed at the same time. This would a restriction. 4. Develop test wich satisfy: For any two parameters/inputs P1 and P2 and for any partition value V1 for P1 and V2 for P2 there is a test where P1 has the value V1 and P2 has the value V2.

Example Two: P1 (3 values v1, v2, v3) and P2 (v4, v5) , P3 ( 4 partitions), P4 (1 partition), P5 (2 partition) in SUT (system under test) so 48 different combinations. Example One (continuation): Take the column with more values P1 combine with a second column. Verify that combinations are satisfied. Engine (3) 3.0 3.0 3.8 3.8 5.0 5.0 Transition(2) A M A M A M Body(2) 2D 4D 2D 4D 4D 2D Tires(2) N HP HP N HP N

You can do 24 cases (all) but DOE warranty that this 6 are enough.

Example Three CPU - A B C OS D E H DB F, G All combinations are 18

! !

With Pair wise combination testing at least 9. The parameters with the most number of values (3 each) that would generate 9 combination, this would be the minimum number of combinations. To create the list, first create all possible combinations of the two partitions with the greater number of values. And then complete the third column assuring that we have a combination that covers all options. NOTE: The table below is not complete, it is needed to complete the third column.

! !

CREATE AN ORTOGONAL ARRAY " Software for this: AETG, Allpairs, tvguwm. A A A B B B C C C D E H D E H D E H F G F F G G G F F

2. Paper 9. The AETG System: An Approach to Testing Based on Combinational Design

! ! Describe a tool to do DOE Basic combinatorial design paradigm

See Table 9.

3. Paper 10. Automated Combinatorial Test Methods Beyond Pair wise Testing
! ! They review pair wise as an introductions Unlike pair wise technique instead of testing in pair, we can take groups of inputs, for instance P1-P2-P3 or P1-P2-P4. This would help, but it is kind of difficult. So here is were automation comes =)!.

4. Problem about Database

! ! 8 inputs with 2 o 3 partitions each one. Applying pair wise will be 9 test (because we have 2 cases with 3 partitions (take the 2 with more partitions and multiply). If we use pair-wise combination, we will have as minimum as 9 test cases, but does this test cases would be enough?... No due the dependency we have in the inputs. This happen even if we use a 3x or 4x flavor of pair-wise. Here maybe it is do not going to work Why? Because dependences This technique must be a very careful!!! Pair wise combinations cold cause problems. THIS is good for configuration testing! Something that make sense I try this and that! This software interact with this device driver by example or for performance or stress testing

This is also used for Interaction Testing between Black boxes.

[week 04 Class 07 Thursday]

9/9/10 2:56 PM

1. Functional Testing 1.4. Design of Experiments (DOE) ! So far we have covered this: EP BVA Decision Tree and Decision Tables Model-Based Testing State Based Testing (state machine) Now we will review a new technique: Design of experiments (DoE) Sometimes called Pair-wise Interaction testing We have to be careful when we use this tech Is sound cool but some times could be difficult and cause more problems than solutions This is for decision tables that are Too large (a lot combinations)

! ! ! !

Example One: Let see the following problem: Performance testing for a car: Engine (3 options): 3.0, 3.8, 5.0 Transmission (2 options): manual, auto Body style (2 options): 2Doors, 4Doors Tires (2 options): normal, high performance

Having all this inputs we will have 24 possible configurations (3*2*2*2 options = 24 configurations) But It could be exhaustive DOE ( it is a sampling technique). Instead of test all, use small sample.

! !

DOE Pair Wise Combination Testing Steps: 1. Identify the inputs parameters to the SUT (systematic under Test) 2. Partition the inputs/ create samples (remember EP) 3. Specify constraints prohibit combinations. In some systems something can not happen together, for instance having two buttons to be pressed at the same time. This would a restriction. 4. Develop test wich satisfy: For any two parameters/inputs P1 and P2 and for any partition value V1 for P1 and V2 for P2 there is a test where P1 has the value V1 and P2 has the value V2.

Example Two: P1 (3 values v1, v2, v3) and P2 (v4, v5) , P3 ( 4 partitions), P4 (1 partition), P5 (2 partition) in SUT (system under test) so 48 different combinations. Example One (continuation): Take the column with more values P1 combine with a second column. Verify that combinations are satisfied. Engine (3) 3.0 3.0 3.8 3.8 5.0 5.0 Transition(2) A M A M A M Body(2) 2D 4D 2D 4D 4D 2D Tires(2) N HP HP N HP N

You can do 24 cases (all) but DOE warranty that this 6 are enough.

Example Three CPU - A B C OS D E H DB F, G All combinations are 18

! !

With Pair wise combination testing at least 9. The parameters with the most number of values (3 each) that would generate 9 combination, this would be the minimum number of combinations. To create the list, first create all possible combinations of the two partitions with the greater number of values. And then complete the third column assuring that we have a combination that covers all options. NOTE: The table below is not complete, it is needed to complete the third column.

! !

CREATE AN ORTOGONAL ARRAY " Software for this: AETG, Allpairs, tvguwm. A A A B B B C C C D E H D E H D E H F G F F G G G F F

2. Paper 9. The AETG System: An Approach to Testing Based on Combinational Design

! ! Describe a tool to do DOE Basic combinatorial design paradigm

See Table 9.

3. Paper 10. Automated Combinatorial Test Methods Beyond Pair wise Testing
! ! They review pair wise as an introductions Unlike pair wise technique instead of testing in pair, we can take groups of inputs, for instance P1-P2-P3 or P1-P2-P4. This would help, but it is kind of difficult. So here is were automation comes =)!.

4. Problem about Database

! ! 8 inputs with 2 o 3 partitions each one. Applying pair wise will be 9 test (because we have 2 cases with 3 partitions (take the 2 with more partitions and multiply). If we use pair-wise combination, we will have as minimum as 9 test cases, but does this test cases would be enough?... No due the dependency we have in the inputs. This happen even if we use a 3x or 4x flavor of pair-wise. Here maybe it is do not going to work Why? Because dependences This technique must be a very careful!!! Pair wise combinations cold cause problems. THIS is good for configuration testing! Something that make sense I try this and that! This software interact with this device driver by example or for performance or stress testing

This is also used for Interaction Testing between Black boxes.

Homework

9/9/10 2:56 PM

Assignment 2. Feature Interaction Testing.

! ! What is this? Approaches?

Exam
! ! Lecture Papers

[week 05 Class 08 Tuesday]

9/14/10 2:57 PM

1. Paper 11. An Innovative Approach for Testing Bioinformatics Programs Using Metamorphic Testing.
! Bioinformatics programs: organize and analyze large & complex biological dataset. Invoke complex algorithms to extract useful information. Test case: input -> expected results Testing oracle problem 1- N-version programming Input to several version of the program (v1, v2, v3) and compare. (version means working implementations) ! Metamorphic Testing: ! Been able to look at the input domain and search REASONABLE outputs. ! Define properties about the output. Another example: graph analysis: search for shortest path (G,a,b) Another example: graphics (huge data sets) Moving a Light source to cause a shadow In summary, the idea that testing need inputs and predefined outputs works very well in theory, but there are a lot of fields where this do not work in the same way. So the methods reviewed above could help for these fields.

! ! !

! !

2. Structure-Based Testing (White Box Testing)

! A white box testing methods gets its name due the fact it see the code. Unlike black box testing methods that are all based only in requirements.

Tend to be techniques to be applied into lower level (small units of code) unit, services, components. Not for entire systems Not fot Integration or Qualification Level of Testing

Two categories ! Static Techniques Analysis of the code. Example: symbolic execution. ! Dynamic Techniques Execute the code: run for test. Structure - control flow. o Statement coverage o Decision coverage o Decision-Condition Coverage o Multiple decision Coverage Data flow

3. Statement Coverage
! ! Develop test cases such that every statement is execute at least once Example if a <10 or b>5 (1) then x<-50 (2) else x <-0; if w = 5 or y>0 (3) then z<-48 (4) else z<-5;

Flow Chart = Control Flow Diagram. Test predicate (diamond) Statement (oval) 4 statement in the previous code Path Testing to execute every Path into the code! Testing strategy execute all path into the code Loops are going to make the numbers of paths bigger. So path coverage is complicate. Values of (a, b, w, y) to execute all statements (with 2 cases we do it)! a = 0, b=0, w=0, y=0 (execute 1 and 4) a=10, b=4, w=5, y = dont care (execute 2 and 3) Whit these two tests cases we are covering all the statements.

! !

However, this kind of testing is the minimum need. FAA D0178B standard / best practices: 1. statements 100% statement coverage! Compilers help us to do that. By example the PROFILER measure the time of each statement.

4. Decision (branch) Coverage ! Develop test cases such that each branch (decision point) is executed at least once. Decision point -> conditional statement = test predicates = diamonds

EJEMPLO UN MAPA CUADRICULAR CON CALLES Y CADA INTERSECION ES DECIDIR UNA DIRECCION Even If I use structured programming (no GOTO) then if I have NOT 100% statement coverage then I have 100% test predicated. s1 if x<10 then s2 s3 x =5 I have 100% statement coverage But NOT 100% decision coverage, false case is MISSING!

! !

Statement Coverage Do Not Imply Decision Coverage! Decision Coverage satisfies Statement but Statement not necessarily satisfies Decision! So Software must cover 100% Decision Coverage

3. Decision-Condition Coverage
! Develop test such that each condition in a decision takes on all possible outcomes and each decision takes on all possible outcomes This is like Decision++ (the second part of the definition, is Decision Coverage) if x<10 or y>50 and z!=0 or flag= T and status = NIL then S1 else S2 2 test to statement coverage 2 test to decision coverage

But that not enough for Decision-Condition Coverage. We need to open the test predicate and look inside: there are 5 conditions! For each condition we need to test their TRUE and FALSE cases We need 2 cases. All false and All True.

4. Multiple Condition
! Develop test to execute all combinations of conditions within a decision. This is the most powefull.

5. Example, Test Binary Search Algorithm)

(1) start <-1; (2) end <-num; (3) found <- false; (4) while starts <= end and not found //(C1) and (C2) (5) middle <-(start = end)/2 (6) if key > table[middle] //(C3) (7) then start <- middle +1 (8) else if key = table[middle] //(c4) (9) then found <-T (10) LOC <- middle; (11) else end <- middle -1

We have 4 conditions, and we have 11 statements (each row). We have three decision points and we have one case with multiple (2) conditions.

if4 T F if6 T F if8 T F

For multiple-condition coverage C1 y C2 make combinations of T y F For conditions C3 y C4 simply try one T and one F of each one

Example of Execution 10 20 30 40 50 60 70 Key=55 1 2 3 4 with T, T 5 6T 7 4 done before T, T 5 6F 8 with F 11

4 con T, T 5 6T 7 4 con F, T ! ! ! ! Multiple-condition Coverage: 100% NO Decision-condition Coverage: 100% NO Decision Coverage: NO Statement coverage: NO

Now execute this: 10 20 30 40 50 60 70 Key = 40 1 2 3 4 with T, T 5 6 with F 8 with T 9 10 4 with T,F ! 4. 3. 2. 1. Coverage is cumulative Multiple-condition Coverage NOT 100% (75%) Decision-condition Coverage 100% Decision Coverage 100% Statement Coverage 100%

[week 05 Class 09 Thursday]

9/16/10 2:53 PM

0. Last Class
! We review different types of Coverage in particular Control-Flow related. We review the Binary Search Algorithm We do not have 100% Multiple-Condition Coverage. What can we do? Test Case 3. Search for the missing case that give me 100% Multiplecondition coverage. Do not EXIST! Loop Entry 1 2 T T circle middle E diamante T y F if T start E if False diamante if if true found E if false end E So I have 3 path to test T {impact condition 1} FT {impact condition 2} FF {impact condition 1} But any single path affect both conditions! TT FT

! ! !

TF So F F can not occurs!

1. Problem
Code coverage for midterm exam if x <10 or y>50 (1) then S1 (2) else S2 if w=50 or z>10 (3) then S3 (4) else S4 Whit this four test cases x T1 T2 T3 T4 0 11 5 15 y 0 75 15 75 w 0 50 10 50 Z 0 12 5 3

The question is, do we have 100% coverage with this four test cases?

Statement Coverage We have 4 statements (each s) Statement Test Case 1 2 3 4 T2, T4 T1, T3 T1, T2, T3,T4

Decision Coverage We have 2 decisions points (ifs) Decision 1T 1F 2T 2F T2(T o T), T4(F o T), T1 (F o F), T3(F o F), Test Case T1(T o F), T2(F o T),T3(T o F),T4(F o T)

Decision-Condition Coverage We have 4 conditions (underlined) Decision 1T 1F 2T 2F 3T 3F 4T 4F Test Case T1, T3 T2, T4 T2, T4 T1, T3 T2, T4 T1, T3 T2 T1, T3, T4

Multiple-Condition Coverage We have 2 multiple conditions (line 1 and 4) Decision 1 TT 1 TF 1 FT 1 FF 2 TT 2 TF 2 FT 2 FF T1,T3 T2 T4 T1, T3 T2, T4 Test Case

So, we have: Multiple-Condition Coverage: 5/8 = 62.5% Decision-Condition Coverage and Decision Coverage: 11/12 = 91.6% Sentence Coverage: 3/4 = 75% '" '" '" %" &" !" #" $"

2. Path Expressions '"

Convert Control Flow Graph to path expression! '"

! !

We can label the edges We have 4 paths for 1 to 5

Approach to convert a Control Flow Graph (CFG) into a Path Expression: 1. Combine serial links by multiplying path expressions 2. combine parallel links by adding, + their path expressions 3. Remove self loop by replacing them with a link of the form x* Lets make it more meaningful

Example:

!" +" ( 1) ( 2) ( 3) ( 4) ( 5) ( 6) ( 7) ( 8) ( 9) (10) (11) read (,y) while x!=0 y <- x + y if y>10 then put (y) else put (x) if x>10 then x<-x-5 else x<-x-1 read (x,y) end; !!" #"

'"

(" $" ," &" 3" 5" 0" 4" !2" /" 6" 1" 8" )" %" 7" ." *" -"

Convert the code into graph

The nodes on the CFG are not diamonds and ovals, but circles that represent the nodes. The labeling of the edges is totally arbitrary.

And then into Path Expression a ( bc (df + eg) (hj + ik) l ) * m So, whit this notation we somehow can define how many paths we have from 1 to 11. In this case it is a infinite situation. So if we define how

many times we will run the loop we could mathematically define how many paths we have.

3. Anomaly Detection
! ! Static analysis to find potential errors in code! It is all about to find anomalies in the code, such as open a file that is already open, or close a file that is already close. Or assign a value to a variable in the statement 10 e.g. x=3 and the next line (11) we assign another different value e.g. x=5. There is a theorem that helps to define these situations. Huans Theorem Let A,B,C be nonempty sets of character sequences whose smallest string is at least 1 character long. Let T be a 2-character string. Then if T is a substring of AB^NC, then T will appear in AB^2C. ! We are going to get sophisticated on the CFG, where edges labels are not going to be arbitrary but they will represent what is going on in that path. With this kind of labels it would be easier to find any anomaly. Hvans theorem works with the path expressions to find anomalies.

[week 06 Class 10 Tuesday]

9/21/10 2:57 PM

0. Last Class
! ! Continue with Testing Approaches We looking at control flow graph , path expression, data anomalies, continue with that and introduce symbolic execution.

1. Symbolic Execution

Definition from Paper 3 (homework list): Symbolic execution represents values of program inputs with symbolic values instead of concrete (initialized) data and executes the program by manipulating program expressions involving the symbolic values. (like algebra)

Uses Probe of correctness Generate test data Anomalies Detection

Example One: (0) (1) (2) (3) (4) ! input A, A <- A + B <- A A <- 2 * C <- A + B B B A + B 4

(0) A0 value of A after execute step 0 (defined) B0 (defined) C0 (undefined)

(1) A1 = A0 + B0 B1 = B0 C1 = C0 = (undefined) (2) A2 = A1 = A0 + B0 B2 = A1 - B1 = A0 + B0 B0 = A0 C2 = C1 = C2 = (undefined) (3) A3 = 2 *A2 + B2 = 3A0 + 2B0 B3 = A0 C3 = (undefined) (4) A4 = 3A0 + 2B0 B4 = A0 C4 = 3A0 + 2B0 + 4 !"

$" %" #"

Example Two:

if (x<=0) or (y<=0) then x<-x^2; y <- y^2 else x<-x+1; y <-y+1; endif; if (x<1) or (y<1) then x<-x+1; y <- y+1 else x<-x-1; y <-y-1; endif;

%" &"

$" '"

Then, create a Control Flow Diagram (flowchart diagram) Now for symbolic execution we have 4 paths into the diagram

In these cases we need to execute the code and do the symbolic execution for each of the paths. So if we do the True-True case we will have: TT path: x1 y1 x3 y3 = = = = X0^2 y0^2 x1+1 = X0^2 + 1 y1+1 = y0^2 + 1 %" Example Three: !" read(x, y) if(x>7) (1) then y<-x+10 (2) else y<-x-10; if y > 0 (3) then x<-y+5 (4) else x<-y-5 ," if x + y > 10 (5) then x<-x+10 (6) else x<-x-10; #"

()*+"

$"

%" &"

$" '"

%"

$" -"

TTT Path: x1 = x0 y1 = x0 + 10

x3 = x0 + 15 y3 = x0 + 10 x5 = x3 + 10 = x0+25 y5 = x0 + 10 The numbers correspond to the sentence number ! The main use of this is to create test data, in particular path conditions Things gets ugly when we add loops, references inside data structures. ! Assuming things simple this is useful to create test data, in particular Path Conditions: Necessary Values for the path to be covered. Use in conjunction with the branch condition for all the branches along the path with symbolic variables substituted in.

3. Path Condition
! ! What values do we need in a path to be transverse. Conjunction of the branch conditions for all the branches along the path with symbolic variables substituted in. Example Two (TT Path): We identify 4 paths. For TT Path ( (x0<=0) or (y0<=0) ) ! ( (x1 < 1) or (y1<1) ) Then, What test data we need to drive down the path? ( (x0<=0) or (y0<=0)) ! ( (x0^2<1) or (y0^2<1) ) x = -0.5

! !

y = -0.5 This are the values to test the path ! Example Two (FT Path): Calculate the symbolic variables. x2 = x0 + 1 y2 = y0 + 1 x3 = x0 + 2 y3 = y0 + 2 Then define the path condition ( (x0>0) ^ (y0>0) ) ^ ( (x0<0) or (y0<0) ) Then, create test data for that! This show that, this is a infeasible path, it is a path that can not to be executed (FT is a path invisible THAT CAN NOT BE EXECUTED)

4. Problem in class: Path Condition

! Develop the path condition for TTT path in code from example 3 x1 = x0 y1 = x0 + 10 x3 = x0 + 15 y3 = x0 + 10 x5 = x3 + 10 = x0+25 y5 = x0 + 10

And the conditions are: initial: (x0 > 7) and (y1 > 0) and (x3 + y3 > 10) then, (x0 > 7) and (x0 > -10) and (x0 > -15/2) Easy, just make substitutions!

[week 06 Class 10 Tuesday]

9/21/10 2:57 PM

1. Complexity Measure: McCabe Number. ! Now we are going to talk about Complexity measure this is a tool that take a piece a of code and generate a complexity number for this code. This is know as McCabe number. This is equal to the complexity of a graph. Code -> # (calculate mcCabe complexity number) Based of [graph] The formal definition is V(G) = edges nodes + 2(connected components) = e n + 2p The informal definition is V(G) = test predicates + 1, where test predicates (points of decisions) For the example 3 is 4 (3if plus one) If a piece of code has a high complexity number it means that it has a lot of test predicates which means it has a lot of points of decision. On companies that develop software they define a threshold for this complexity and if a code gets a number that is over this threshold the code is flag. It does not means that the code is wrong but it has to be reviewed more careful. McCabe complexity testing Strategy Structured testing V(G) = #of basis paths that must be tested = minimum # of test Let see an example S1 If x<10

! ! !

! !

'"

&" Then s2 Else if y>0 Then s3 Else s4 If z=5 Then s6 Else s7

%"

!"

&"

%"

#"

$"

&"

%"

(" We have 6 paths (3 x 2). Complexity = 4 (diamantes + 1) So, the recommendation us to test 4 cases

)"

! !

So for the code above we will choose the basis paths, that we call them basis paths. This paths are the ones which can be used to create all other paths.

Approach: 1. Select an arbitrary path 2. Flip the first condition, attempt to return to 1st path 3. Reset first condition, flip the second condition, attempt to return to 1st path 4. Repeat (no audio since last exercise, audio continue here)

Example One: ! This are the 4 basis path in the example: (1) (2) (3) (4) F T F F T T (arbitrary) _T FT TF

*+,-"

This would be the four basis paths according with McCabe technique. We have 4 basis paths due the fact the complexity of the code is 4. %" &" '" Example Two (with 3 consecutive ifs) ! ! Complexity : 4 (diamonds + 1) For basics Paths (1) (2) (3) (4) T F T T T T F T T T T F #" &" %" $" !"

&" ("

%" )"

! ! !

In the past classes we talk about Code coverage (white box). We mention: Control and Data flow coverage. Data Flow Coverage, look definitions an uses of data and create paths. (next class)

[week 06 Class 11 Thursday]

9/23/10 2:55 PM

0. Last class
! Control Flow Coverage: McCabe Testing, Decision coverage, Multiple Condition, etc. Today Data flow coverage.

1. Data Flow Coverage

! Even 100% Control Flow coverage, that means even 100% multiple condition coverage do not warranty no errors. Try to do all path coverage it still do not warranty no errors) ! Data flow coverage extend the capability of detect errors. Data is going to search data problems like point with cero division

Definition: Variable Occurrence Classifications ! ! ! def: (variable defined) x:=0 c- use: (computational use) variable used in a computation. y = x +1 p-use: (predicate use) variable is used in a predicate. If x<1

Approach: 1 Classify Variables. 2. Construct Control Flow Graph (CFG) with annotations about (def/use)

For each node: o def(i) : set of variables for node i which are globally defined o c-use(i): set of variables for which node i contains a c-use o p-use(TP) : set of variable for which the TP contains a p-use (TP = test predicate) %" $"

Example One: if x<10 then y<-0; z<-0 else y<-5; y<-10 endif
!" #"

!" ! In this example we can see that we have nodes in our graph that have %" $" multiple statements. Node (1) has 2 statements Y=0 and z = 0.
%&!"

Example Two: (1) get x,z y <- 0 if x > 10 (2) then y <- 15; if z > 0 (3) then w <- y + 1 (4) else w <- y - 1

#"

%"

$"

%&#"

'"

("

def(1) = {x,y,z} def(2) = {y} def(3) = {w} def(4)= {w}

c-use(1) c-use(2) c-use(3) c-use(4)

= = = =

0 (nothing) 0 (nothing) {y} {y}

p-use(TP1) = {x} p-use(TP2) = {z} ! Then, a new definition, CLEAR PATH w. r. t (whit respect to) "some variable"

Definition: Clear Path (def-clr) It is a path between two nodes i and j with intermediate nodes (i, N1, N2, Nm, j) which contains no definitions of x in (N1, , Nm)

Definition: Definition Computation Use (dcu) dcu(x, i) Where i is any node, and x is defined in i dcu(x, i) is the set of all nodes j such that x is in c-use(j) and for which there is a definition-clear path w.r.t x from i to j

Definition: Definition Predicate Use Path (dpu) dpu (x, i) [definition /predicate use set] is the set of all TP such that x is in p-use(TP) and for which there is a defclr path w.r.t. x from i to TP

Definition: Use Path (du) It is a path (n,. Nj, Nk) is a du path w.r.t. x if N1 has a global definition of X and either: 1. Nk has a c-use of x and (n1, nj) is a def-clr path w.r.t. x 2. Mk is a TP and has a p-use of X and (ni, ..nj) is a def-clr path w.r.t. x (the path from definition to use ")

!" Go back to Example Two: def(1) = {x, y, z} def(2) = {y} def(3) = {w} def(4)= {w} c-use(1) c-use(2) c-use(3) c-use(4) = = = = 0 (nothing) 0 (nothing) {y} {y} '" %"
%&!"

$"

#"

%"

$"

%&#"

p-use(Tp1) = {x} p-use(Tp2) = {z} !

("

dcu(var, node) nodes that use the var defined in node dcu (x,1) = empty dcu (z,1) = empty dcu (y,1) = {3,4} dcu (y,2) = {3,4} dcu (w,3) = vacio dcu (w,4) = vacio

dpu(var, node) dpu (x,1) = {TP1} dpu(z,1) = {Tp2}

Test Coverage Criteria for data flow means "all-uses coverage" We want 100%

!"

3. All-Uses Coverage
! %" $" Goal: develop test cases such that for each definition, there is a %&!" definition use path (du) to every use of the definition. #" Problem One get x, y; a <- 0; b <- 0; if x>10 then w <-a + 1; b <- 4 else w <-b + 1; a <- 4; if y>10 then z <- a + w else z <- b + w; %" $" '"

%&#"

("

)"

def(1) def(2) def(3) def(4) def(5)

= = = = =

{x,y,a,b} {w,b} {w,a} {z} {z} = = = = = empty {a} {b} {a, w} {b, w}

c-use(1) c-use(2) c-use(3) c-use(4) c-use(5)

p-use(TP1) = {x} p-use(TP2) = {y} ! Create a dcu for each pair def(N) = {i1, i2.}

Search {i1. I2} en c-use(M) = {i1, i2} and add the numbers M to the set dcu(x,1) = {} dcu(y,1) = {} dcu(a,1) = {2, 4} dcu(b,1) = {3, 5} dcu(w,2) = {4} dcu(b,2) = {5} dcu(w,3) = {4} dcu(a,3) = {5} dcu(z,4) = {} dcu(z,5) = {} dpu(x,1) = TP1 dpu(y,1) = TP2

X Test 1. TT Test 2. FT 15 5

Y 15 15

Coverage Def(a1) & c-use(a2) & c-use(a4) Def(a1) & c-use(a4)

! !

For the 4 paths. Test EVERY definition Give me all-uses coverage with respect to a but not respect to b You need to do this for all-uses. The same use cases give me all te missing ones " X Test 1 Test 2 Test 3 Test 4 5 5 Y 15 5

15 15

15 5

What control Flow Coverage do I have? 100% statement coverage

100% condition coverage All definitions criteria is satisfied if for each definition in the program some use of this definition is executed by at least one input in the test suite.

Homework

9/23/10 2:55 PM

For Exam:
! ! Create EP Asses level of Coverage

[week 07 Class 12 Tuesday]

9/28/10 3:05 PM

0. Last Class
! Examples from the last week (1) x=10; y=5; if a >25 (2) then x=3; (3) else y=3; if b>0 (4) then z=x+5; (5) else z=y+5; ! A exam question could be: Does the following test set of data satisfy the all uses coverage criterion w.r.t. x and y, if not complement. def(1)={x, y} def(2)={x} def(3)={y} def(4)={z} def(5)={z} c-use(1)=empty c-use(2)=empty c-use(3)=empty c-use(4)={x} c-use(5)={y} p-use(TP1)={a} p-use(TP2)={b} dcu(x,1)= {4} 1-3-4 //define los paths

dcu(y,1)= {5} dcu(x,2)= {4} dcu(y,3)= {5} dcu(z,4)= {} dcu(z,5)= {} dcu(a,1)= {} dcu(b,1)= {} dpu(a,1)= {TP1} dpu(b,1)= {TP2}

1-2-5 2-4 3-5

1-TP1 1-TP2

Test case 1 2 3 4

a 20 20 30 30

b 5 -3 -3 5

Coverage x(1-3-4), a(1-TP1), b(1-3-TP2) y(3-5) y(1-2-5) b(1-2-TP2) x(2-4)

For exam, only answer yes or not. If not then why? ! for each one of the definitions of X, we need to test that there is a definition clear path between the node where we define x and the node where we use x. Than means that the value of x do not change. X is defined in node 0, and it is used in node 4. So to have a clear path we need to avoid to enter to node 2. So if a is less or equal to 25 and b greater that 0 we avoid the node 2 and we get to node 4. The next case for X is, that it is defined on node 2 and used in node 4. So if a is greater than 25 and b greater than, we will have that covered. The answer for this problem is YES!

1. Paper: An empirical study of the use of Frankl-Weyuker Data Flow Testing Criteria to Test BPEL Web Services.
BPEL Business Process Execution Language. Is used to coordinates the webservices. So far we are testing programs based on languages such as C, C++, Java, etc., but what if you need to test a software based on webservices. The language BPEL can be converted into a graph (CFG). Then we will have the ability to test the control flow coverage. When you are testing and SOA app, you might want to test all services. The BPEL helps to abstract all of them from the graph. Creating new methodologies for testing. When you create a new methodology you need to prove that it is better. One way to prove this is putting faults in the code and then evaluate how well your methodology detects this faults. See table 3. Is an experiment of one data set. They test three test suites with 100% all-uses coverage, but now they are evaluating the faultdetection rate. The message of the paper is that with this kind of applications (webservices) we may need more tools and techniques that better fix for this kind of appl. Think in the testing methodologies or techniques as spray to kill bugs. Not all sprays works for all kind of bugs. So we need different techniques depending on the type of appl.

2. Paper: The Use and Limitations of Static Analysis Tools to improve Software Quality.

Static tools all need a control flow graphs, then annotate the node with the data used on it, and then look for anomalies in the code. Problems with this type of tools: Remember: This tools goes to different paths in the code and try to find bugs One problem that we have in this techniques is that the detect bugs that are not really bugs. There are a couple of metrics to evaluate this tools. * Recall metric: it is #of flows found / #of flows present. We want that this number is close to one. * Precision-metric: this number is true positives/reported problems. We want that this number is close to one too. * Performance-metric: CPU time and memory. How many CPU and memory does this tool take to come up with a result. Another problems that we have are loops, dynamic variables. Due they make things difficult to review. Then besides static analysis, we will need dynamic analysis to complement these tecnniques.

3. Paper: A Survey of New Trends in Symbolic Execution for Software Testing and Analysis. This tool creates a symbolic execution tree. (where PC = Path Condition) See it on the paper. There are some different tools to implement this. Having this tree helps to use different techniques to test our tool.

Feature Interaction Testing / Service Interaction Testing Is a another important methodology, to test apps focus on featured of the system. Testers test every feature/service separately and they work wall. The problem became when you want to combine them, you must test their interaction. Problems doing this: 1. Define feature interaction [being as more specific as possible] 2. test unspecified / unplanned interactions. A way to do this is using a model, but most of the times it requires and adhoc. Another way to do it is been analytical. But talking about adhoc we can mention the following:

Systematic VS adhoc - Adhoc: by my god, expert testing, trial and error, are other names for this type of testing. The question is, is this bad?... NO!. - Recommendation. Try to do as much as possible systematic testing. But there should be a little bit of novelty, surprise. Out there is the term Exploratory testing, is not 100% adhoc but is kind of. Exploratory Science (analogy) 1. Learn as much as possible prior to exploration. 2. Develop a systematic strategy for exploring: follow the stars, follow the river. 3. Keep track of where you have been. Do journals. 4. Be extremely observant of side effects. 5. Document findings. Now let's talk about Exploratory testing, is not intent to be adhoc.

Again. 1. Do your homework. Learn as much as possible prior start. 2. Follow a strategy. 3. keep track of all your test cases. 4. Be a good observer, keep clecklist This class is focused on systematic techniques. A good equilibrium could be 95% systematic and 5% exploratory. Out there are some techniques as scratch and sniff kind of exploratory testing, and is there =)!.

[week 07 Class 13 Thursday]

Syllabus revieew 4 primeros testing back testing process allolt time reqs-base testing feature interact exploratory Structure base (data) design expr testing tools

9/30/10 2:52 PM

a) Unit/Component white/black b) Integration Strategy 1. incremental 1.1. top down 1.2. Buton up 1.3. threads 2. Big Bang (all things thogyeter) c) System testing ---1Driver 1.1stub 1.1.1 actual componnt * incremental isolate the problem Integration problems are Design problems :-0

Integration Testing cubre -> SW SW, SW HW, component component, System system (enterprice producto)

NON-functional testing (QoS)

performance stress security usability 1. Performance testing Obj. verify that system meets its performance req for specified loud conditions including stress and volume scenarios * ENTRY CRITERIA 1. quantitative and measurable performance requirements. 2. reasonably stable system 3. test environments 4. EXAMPLE Airline reservation system {web-based} - excellence response -> time cuales condiciones? Normales que es normal? 200 customers on the system at the web site 2. Stress Testing Obj. Verify the behaivor of the systems meets its reqs when its resources are saturated ans pushed beyond limits

Next class Strategie to stress testing (it is gonna be ad-hoc contrary to all systematic) PAPER Cloud testing PAPER Architecture

[week 07 Class 13 Thursday]

LAST CLASE - working on NON-functional (QOS)

10/5/10 2:51 PM

Performance Obj: verify system mmets it performance conditions especifics Stress testng Obj: verify behavior when resurces saturated

1. Aproach To do Strees Testing systematically

1. identify the stress ponts (potential bottleness) work with developers and architets this is white box testing 2. Develop a strategy to strees the system at the stress point. Use tools to simulate load generator tools 3. Verify intended stress si actually generated Measurement tools (make a loop between 2 and 3) 4. Observe behavior. Verifi req are met under stress

Remember our analogies and their stress points: Human Body (heart), car (wheels), bridge (some point), computer system (cpu, network, memory); in general, the places that could fail under strees! Domain: embedded Systems Other Example: Phone System - metwork saturated, but not blocking 911

2. volume Testing Def. verify the behavior of the system meets its reqs when the system is subjected to a large volume of activity over and extended period of time. (memory leak after a period of time software stop running) (y2K type problems) (limits that will be reach, like integer limits in a counter) How to do volume testing? Stadistical analysys tools phone system is 24/7 embedded system (like cars) are not 24/7 because battery limits

3. Paper: Service Oriented Architecrture testing: Asurvey - SOA apps are important functional and non-functional reqs (Perforance, stress, volume). - SOA unique features that add complexity to testing burden include: from intrisect distributed to ownership shared among different stakeholders. TESTING LEVELS - unit testing LEVEL, a single service like {end user}- >{blackbox test} like [service developer] ->[white box | black box] = component testing [BPEL = control flow graph] -> structural testing - integration testing = multiple services - regression testing =

The rest of the paper is about SLA ?? And also talk about interesting complications

4. PAPER: Cloud9: A software Testing Service Cloud9 is a tool. - "testing as a service - cloud based testing" - develop software and upload to be tested by someone into the cloud ! - Clood made quality certification. [UL] - symbolic execution into the cloud = explore execution paths you requiere CPU a lot and Higher memory. But that what the Cloud Provide!

! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !

End of The First Midterm