Professional Documents
Culture Documents
Introduction to Measure
Introduction to Measure
Define Opportunities Measure Performance Analyze Opportunity Improve Performance Control Performance Has fact-based decision making been consistently applied? Are the teams assumptions continuously validated? Is business risk being actively managed? Is leadership informed & on board with the teams findings & conclusions? Can the likelihood of success be improved by revisiting previous conclusions or analysis? 1.0 Define Opportunities 2.0 Measure Performance 3.0 Analyze Opportunity 4.0 Improve Performance 5.0 Control Performance
Is the team operating within the context of the project charter? project scope goal statement project plan
What Is A Measure?
A Measure Is A Description Of One Characteristic Of An Object Or Activity
For a process: Time required to complete process steps Timely execution of process Number of errors in different process steps Percent yield of a process For a product: Errors in a product Product arrives/shipped on time Number of products shipped per month
Collect facts about a problem or opportunity and voice of the customer information Establish a baseline performance for Project Y to understand how well we meet customer expectations Identify the root cause of a problem and find the key to solving the problem Evaluate competing solutions based on their impact on performance; degree and direction of change; to compare process performance before and after the solution is implemented
Improve
Control
Quantify the change in process performance to ensure improvement gains are sustained
Supplier
Input
Process
Output
C O C
Start
Stop
Customer
CTQs
Suppliers
Inputs
Process
Outputs Customers
Input and process measures quantify some aspect of an input to the process or the performance at one or more process steps. They are often referred to as independent variables.
Output measures typically quantify product or service characteristics or process outcomes. They are often referred to as dependent variables.
Subprocess L2
x = cycle time
Y Credit
Y = Cycle Time
X Application
X Decision
X
Notification
Quality Measurement
Two Aspects Of Measuring Performance
Effectiveness:
Billing accuracy
Choice of data display and analysis tools Amount of data required: continuous data often requires a smaller sample size than discrete data Information about current and historical process performance
Types Of Data
Data Type Is An Important Consideration
Continuous Data Can be broken down into increments Infinite number of possible values
Counts
Examples
Number of incomplete applications Percent of responding with a 5 on survey Number of Green Belts trained
Examples
Cycle time (measured in days, hours, minutes, etc.) Weight (measured in tons, pounds, etc.)
Discrete
Ordered Categories (Limited options, i.e., 1-15)
Technically Discrete, but can often be analyzed as Continuous Ordered Categories (Many options, i.e., 1-100) Count Data (Limited possibilities) Count Data (Many possibilities)
Continuous
Cycle Time
Binary (Y/N)
Discrete Y Measure
% within specifications Number late Yes/no questions
Continuous Y Measure
Actual times for each unit Actual time deviated from target Rating 1-100
Number lost
The More Continuous We Can Make The Data, The More It Will Tell Us About Our Process
Write and pilot operational definitions Develop and pilot data collection forms and procedures Establish a sampling plan
Train data collectors Pilot process and make adjustments Collect data Monitor data accuracy and consistency
Data Collection Is The First Step To Understanding The Variation The Customer Feels
What To Collect?
Operational Definitions & Procedures
Start:
Stop: Instructions
Analysis Plan
Frequency
Ti me
Measurement
Audit Plan
State the purpose of the data collection Identify what data is required
Segmenting by external factors will help us identify the drivers of variation in the process
Possible categories:
Factor
What type When
Example
Complaints, defects, problems Year, month, week, day
Where
Who
Tip: Begin with factors outside the process box - often these are factors that were not considered when the process was first designed
Identify the factors for segmentation before you start collecting data Make sure the segmentation factors can be measured reliably Record the segmentation factors for each Y data point collected Segmentation factors are typically easy to collect, so collect more segmentation factors rather than less
For your team project: Brainstorm a list of segmentation factors for your project Remember to also segment on unlikely parameters Report out to group
Clearly specify variables to be collected: Operational definitions for all metrics Specific descriptions of how to take the measurement Specify the details of the data collection process: How to collect the data How to record the data The period of time for data collection The sampling plan to be followed
Operational Definitions
Defining The Measure
Definition
An operational definition is a clear, concise description of a measurement and the process by which it is to be collected.
To remove ambiguity Everyone has a consistent understanding To provide a clear way to measure the characteristic Identifies what to measure Identifies how to measure it Makes sure that no matter who does the measuring, the results are consistent
Always Pilot Your Operational Definitions
Purpose
What: Must have specific and concrete criteria How: Must have a method to measure criteria Must be useful to both you and the customer (the wing-to-wing concept)
Example: Loan Application Cycle Time What: Loan application cycle time is the number of hours from receipt of a loan application, to successful notification of decision for the loan application
How:
The clock starts when the computer attaches the time of application receipt at data entry
The clock stops when the phone caller notes time completed application decision notification in desk log
of
Measure one scale or level smaller than what your customer measures For Example: If your customer measures cycle time in days, your scale of scrutiny would be hours If your customer measures cycle time in hours, your scale of scrutiny would be in minutes Scale of scrutiny may expose larger true variation
Who All
Time
Desired Outcomes Practice applying operational definitions Collect data on the number of defects in a package of M&Ms
Partner Preparation
Find a partner for the exercise. Determine timing for each activity below. Read the background information. Develop an operational definition for one of the defect types found in an M&M, either, 1) chips and cracks, or 2) unclear and illegible M. The definition should include: What How Importance to customer
Partners
With your partner apply your operational definition to your package of M&Ms. Use the form on the following page to record the total number of M&Ms you inspect and the number of defective M&Ms . Note: If an M&M has one or more chips/cracks, classify the M&Ms as defective. Brainstorm the challenges of developing an operational definition for this exercise, and how these challenges may impact your own project work. Choose a spokesperson to report out on your operational definition, the challenges you experienced, and how these may impact your project work in the future.
Partners
Close Exercise
Partners
Business Need
Clear and legible M on the M&Ms
Project Y
M M
Subprocesses 1 2 k
A defective M&M is . . . 1. Any M&M with a chip or crack. 2. Any M&M with an unclear or illegible M. The two defect types should be measured separately.
Operational Definition:
# Of Pieces Inspected
Sampling
Objectives
Sampling
Basic Definitions And Symbols
Population (N): The entire set of objects or activities for a process : the mean (arithmetic average) calculated for a population : the standard deviation calculated for a population Sample (n): a group that is a part or subset of a population x: the mean (arithmetic average) of a sample s: the standard deviation of a sample
Sampling
Sampling is the process of: Collecting only a portion of the data that is available or could be available, and drawing conclusions about the total population (statistical inference)
Population
x x x x x x x x x x x x x
N = 5,000 What is the average resolution time?
Sample
x x x x x x
n = 100 From the sample, we infer that the average resolution time (x) is 1.2 days
When To Sample?
When to sample Collecting all the data is impractical or too costly Data collection can be a destructive process When measuring a high-volume process When not to sample A subset of data may not accurately depict the process, leading to a wrong conclusion (every unit is unique e.g., structured deals)
Statistically Sound Conclusions Can Often Be Drawn From A Subset Of The Total Available Data
Representative Sample: All parts of the target population are represented (i.e., selected for measurement) equally The customers view is captured How to guarantee a representative sample: Designing the sampling strategy Understand special characteristics of the population before sampling
Bias occurs when systematic differences are introduced into the sample as a result of the selection process Not representative of the population Will lead to incorrect conclusions about the population
Selection Bias
Convenience sampling: the ones I can reach Systematic sampling: at noon every day
Environmental Bias
Non-Response Bias
Initiated by respondents: only a subset of the population responds to survey (typically the 1s and 5s)
Process Data
Population Data
Population Approach Make probability statements about the population from the sample I have 95% confidence that the mean of the population is between1.5 and 2.5 seconds Use sample size formula Process Approach Assess the stability of the population over time Are the shifts, trends, or cycles occurring? Do I take a special or common cause variation approach to process improvement? Use rational subgrouping
Population N
Sample n
Description
X X X
Population
Segments Large Medium Units LLLLL
Sample
Description
L MMMM SS
Small
Randomly sample within a stratified category or group Sample sizes for each group are generally proportional to the relative size of the group
Process
X X X X X X X X X
Sample
Description
X X
Hour 1
Process Hour 2
Hour 3
Sample
Subgroup of samples
X X X X X X X X X X X X X X X X X X X X X X X X
Key points to consider Monitor process frequently enough to catch it going from good to bad Better to collect several small samples over different times then one large sample at a single point in time Unstable process more frequently Stable process less frequently Rapid cycle process time more frequently Long cycle process time less frequently
You are interested in estimating the proportion of callers (within 1%) who experience first call resolution. A customer survey will be used to gather the data. A random sample will be used to select potential survey respondents. 1. Are there any potential problems with the approach described? 2. What other approaches might be used in this sampling situation? 3. What other information is required? You are interested in improving billing accuracy and have decided to collect a subgroup sample of 30 bills processed from 4 to 5 p.m., every day for the next 4 days. 1. What sampling scheme(s) is planned? 2. Is there a potential to introduce bias using the plan described? If yes, how would the bias be introduced? 3. What other approaches might be used in this sampling situation?
1.
2. 3. D
An improvement team is interested in improving billing accuracy. They decided to sample and pull every 20th bill processed over the next 30 days 1. What sampling scheme(s) is planned?
2. Is there a potential to introduce bias using the plan described? If yes, how
would the bias be introduced? 3. What other approaches might be used in this sampling situation? .
Sample size (n) depends on three things Level of confidence required for the result, How confident I am that the result represents the true population Level of confidence increases as sample size increases Precision or accuracy () required in the result, The error bars or uncertainty in my result Precision increases as sample size increases Standard deviation of the population (s), How much variation is in the total data population? As standard deviation increases, a larger sample size is needed to obtain reliable results
n=
(1.96s )2
n=
(1.96s )2
Calculate sample size (n) based on: Precision () 95% confidence Level (1.96) Standard deviation (s)
Sample
N Values
X-
X
X+
1. Calculate sample size (n) 2. If n/N > .05 OR If n > N 3. Calculate n finite nfinite= n/1+ n/N
n = 1.96
( )2 P(1-P)
n=
(1.96
P (1-P)
Calculate sample size (n) based on: Precision () 95% confidence Level (2) Estimated proportion defective (P)
Sample
Conclusion: n Values I know with 95% confidence that the population proportion defective is P +
Recalculate n* based on the calculated P. If the new required sample size (n*) is more than the number of samples taken, take (n*-n) samples and recalculate P base on the full sample size. If it is not practical to take more samples, then use the actual n and P to recalculate the actual precision ()
The formulas give an approximate sample size Dont forget these important factors! Is the population homogeneous? If not, you will need to segment before sampling What is the opportunity segment for bias? Plan ahead to make sure your data is representative of the true population
What Is The Impact On The Customer If Your Sample Size Is Not Representative Of The Process?
Long Term
Short Term
Repeatability
Random Bias
Reproducibility
Accuracy Stability Linearity
Address Measurement System Variation Before Collecting Data To Analyze Process Variation
True Process
True Process
MSA is a set of methods for estimating the current amount of variation in the measurement process
Data is only as good as the process that measures it Identifies how much variation is present in the measurement process Understanding measurement variation is necessary for identifying true process variation and maximizing true Y improvements Without MSA, you run the risk of making decisions based on an inaccurate picture of your process MSA helps direct efforts aimed at decreasing measurement variation Excessive measurement variation distorts our understanding of what the customer feels
How much variation is caused by the measurement system? Determine which MSA is appropriate based on the type of data collected Determine which aspects of measurement are most relevant for the MSA study (accuracy, repeatability, reproducibility, stability, linearity) Measure units repeatedly. How items are measured depends on the aspect being quantified 2. Quantify the measurement process variation How much error or uncertainty is allowable for this data? 3. Determine if the measurement process must be improved
What are the sources of measurement error? 4. Determine how the measurement process will be improved
How can the error sources be eliminated or minimized Determine how the measurement process will be improved Audit the measure process to ensure accurate and consistent measurement
The type of Measurement System Analysis conducted depends on the type of data: When using continuous data, MSA is conducted through a Gage R&R (repeatability and reproducibility) study When using discrete data (discrete, count or ordered categories), MSA is conducted through a DDA (discrete data analysis) study
Accuracy the differences between observed average measurement and a standard Repeatability variation when one person repeatedly measures the same unit with the same measuring equipment Reproducibility variation when two or more people measure the same unit with the same measuring equipment Stability variation obtained when the same person measures the same unit with the same equipment over an extended period of time Linearity the consistency of the measurement system across the entire range of the measurement system
Standard Value
Observed Average
Validating accuracy involves repeated measurement of something with a known value (master/standard) The difference between the average of repeated measurements of the same master/standard and the true value of the master/standard represents the amount of inaccuracy or bias in the measurement system Service application: Validating the judgement of the person making the measurement against an agreedupon master/standard
Data Summary
Word Committee Battallion Asbestos Seperate Flamboyant Abbacus Catagory Lieutenant Occassionally Liquefy
Spell Checker 1 C C C I I I I C I C
Standard Dictionary C I C I C I I C I C
C = Spell Checker identified the word as correctly spelled I = Spell Checker identified the word as incorrectly spelled = Differences between Spell Checker and Standard Dictionary
Step 4: Quantify The Measurement Process Variation
Spell Checker 1
Correct Incorrect 1 Correct 4
Standard Dictionary
5 5 5 1 0
Incorrect
7 6 5 4 X=5.063
Standard Deal #
The difference between the average of the 20 deals measured by the panel of experts (the standard) and the average of the normal measurement process represents the bias, or inaccuracy, of the measurement process. Thus bias is Xstandard Xnormal , or 5.5 5.1 = 0.4. This means the measurement process has a bias or inaccuracy of 0.4 of a day in measuring cycle time.
The variation when one person repeatedly measures the same unit with the same measuring equipment
Repeatabilit y
(Minimum Variation)
Continuous Discrete
Calculate variation in terms of measurement units (standard deviation, span, etc.) Count number of times the same result is achieved for a given point (% correct)
Validating repeatability involves repeated measurement of the same item by one person with the same measurement device The difference between the first time an item is measured and the second time represents the error of the measurement process Therefore repeatability is the ability of the person or measurement device to consistently repeat measurements for the same items
C = Spell Checker identified the word as correctly spelled I = Spell Checker identified the word as incorrectly spelled = Differences between Spell Checks
1st 2nd 5.4 5.5 4.7 4.6 5.5 5.5 6.3 6.3 3.9 3.9 4.7 4.5 4.8 4.8 5.5 5.5 4.6 4.6 4.7 4.7 6.2 6.2 5.0 4.8 3.9 3.9 6.9 6.8 4.4 4.5 4.0 4.0 4.8 4.8 7.1 7.2 3.7 3.7 5.2 5.4
Actual Deal
Variation
Using either Minitab or Excel we can calculate the percent of the Total Variation contributed by the measurement process in terms of repeatability (see partial printout above). In this case we see that the repeatability of the measurement process is 99.57%, which means that nearly all of the observed variation is coming from the process, not the measurement system.
The variation when two or more people measure the same unit with the same measuring equipment Data Collector 1
Continuous
Discrete
Validating reproducibility involves repeated measurement of the same item by two people using the same measurement device The difference between the two measures represents the ability of the measurement process to be reproducible Therefore reproducibility is the ability of the measurement process to consistently reproduce measurements for the same items across people
Word Committee Battallion Asbestos Seperate Flamboyant Abbacus Catagory Lieutenant Occassionally Liquefy
C I
Spell Checker 1 C C C I I I I C I C
Spell Checker 2 C I C I C I I C C C
= Spell Checker identified the word as correctly spelled = Spell Checker identified the word as incorrectly spelled
Correct Correct
Spell Checker 2 5 4
Incorrect
12 6
Incorrect
0 1 5
43 5
10
Using either Minitab or Excel we can calculate the percent of the Total Variation contributed by the measurement process in terms of reproducibility and repeatability (see partial printout below). In this case we see that the reproducibility and repeatability of the measurement process is 98.52%, which means that nearly all of the observed variation is coming from the process, not the measurement system.
% Contribution 1.48 0.23 1.25 98.52 100.00 Variation due to measurement system
The variation obtained when the same person measures the same unit with the same equipment over an extended period of time
Stability Time 1 Time 2
Continuous Discrete
Calculate variation change in measurement over time Calculate change over time (% correct or consistent)
Linearity is the consistency of the measurement system across the entire range of the measurement scale
Continuous:
The endpoints of a pressure gage are typically not as accurate as the center of the gages range Assessing items for defects is easy in very obvious cases, but can be very difficult in borderline or less clear cases. Operators may have very consistent judgement performance at the beginning of a shift, but poor consistency before breaks or near the end of a work shift
Discrete:
Conducting An MSA
Step 5: Determine If The Measurement System Must Be Improved
Examine context of business environment, process, and customer How critical is the measurement? What are the risks of making an error? Review results of MSA study Typical Gage R&R specs: % < 30% of total process variation <10% of specification window
Conducting An MSA
Step 6: Determine How The Measurement Process Will Be Improved
Identify factors that could cause measurement process variation (measurement error) Reduce the impact of those factors
Conducting An MSA
Step 7: Audit The Measure Process To Ensure Accurate And Consistent Measurements
Determine how often (frequency of audits) Determine what to audit Procedures/documentation up-to-date Procedures/documentation used Quantify MSA
Measurement Systems
Summary
Measurement error is always present in your total observed variation Minimize the measurement process variation Use MSA to identify the amount of process variation Measurement error is always a bigger deal than you think Understand how measurement error impacts your customer
Make Sure Your MSA Is Examining The Actual Measurement System Itself
Who All
Time 1 min
Develop an MSA for your project Y data 1. Determine which MSA is appropriate for your data 2. Determine which aspect of measurement are relevant (accuracy, repeatability, etc.) 3. Develop the plan for how you will collect the data 4. List factors that might cause the measurement of an item to vary and how you would reduce the impact of those factors OR
15 min 13 min
How would you run a MSA on the following conditions: 1. Collecting dates from insurance files 2. Errors on billing statements 3. The number of customers contacted/converted by a broker 4. Call-center: categorizing call types
All
34 min
Communicate the what and why to the data collectors and process participants Train everyone who will be collecting data Pilot the data collection process and adjust as needed Confirm understanding of operational definitions Make data collection procedures error-proof Be there in the beginning to oversee data collection
Check to make sure the measurement system is stable Check to make sure the measurement procedures remain consistent (over time, and from data collector to data collector) Check to see if the data look reasonable
2.1 Determine what to measure Understand the role that data plays in process improvement Understand the cause and effect relationships that occur inside the team's process Determine the indicators needed to evaluate current process performance 2.2 Manage measurement Understand different types of data and how each type can provide the team with different insights and knowledge of a process Develop operational definitions and data collection plans that build validity and consistency in the data which the team gathers 2.3 Understand variation Understand the concept of variation and how a process can be evaluated by assessing its variation over time Plot and calculate the variation of the team's business process Gain hands-on experience with the use of the statistical software package MINITAB
ASU Lean Six Sigma Green Belt DMAIC
Understand the various calculations associated with determining process sigma Calculate the sigma performance of the team's process Understand the difference between First Pass Yield and Rolled Throughput Yield
Understand the different uses of our measurement systems Understand the language of measurement Understand how to conduct a measurement system analysis Understand how to interpret the results of a MSA study Understand MSA in administrative processes