What is the Correct Statistical Analysis? It Depends on your Measurement Scale.

When choosing the type of question that is best suited to your research question, one of the first steps is to stop and consider the type of data produced by each possible question type. In other words, plan your analysis before you collect your data. How is this done? One easy way is to create a three column Question x Analysis table in MS Word or Excel: Column 1: Column 2: Column 3: Enter the question type and text Enter the possible answer, scales and examples of the answer format: Open-ended text questions, dichotomous, multiple choice, rank order, scaled, or constant sum (ratio scale). Enter the type of analysis supported by the answer scale (percentages, means and standard deviations, cross tabulations, and statistical tests).

TYPES OF QUESTIONS:
More than 100 question types are available in Qualtrics, including questions that give answers in the format of openended text questions, dichotomous choices, rank order, pick-bucket-rank, multiple choice, multiple choice matrix, Likert or Semantic differential scales, constant sum, conjoint, side by side, etc. Each question has an associated measurement scale, that is either nominal (text, categorical), ordinal (rank order or ordered categories), interval (scaled without an absolute zero point), or ratio (scaled with an absolute zero point). These scales are often referred to as either discrete (nominal and poorly ordered scales), or continuous (well ordered, interval, or ratio). Each question, based on the associated answer scales, is appropriate for completing multiple types of analysis. These analyses include the computation of percentages and percentages by percentile groups, frequency counts, means and standard deviations, cross tabulations, and statistical tests (chi-square, t-test, ANOVA, regression, multivariate analyses, etc.).

MATCHING QUESTIONS WITH DATA TO DETERMINE TYPE OF ANALYSIS

In the following table we find two columns representing two families of statistical analysis: non-parametric statistics and parametric statistics. Generally, parametric analysis is preferred, but requires stronger data (interval, ratio scaled). Lets examine the table. Suppose you have a single variable that you want to analyze (What is your favorite sport team?). This variable is nominal, because the measure has no scale a team is just selected (respondents select a box indicating their favorite baseball team). In the table, we would find the row for one discrete variable and the column for non-parametric data. We read that the correct statistical analysis is percentages. We can report the percent reporting each team as their favorite.

November 1, 2011 Version, from PC-MDS documentation of MDPREF.

Qualtrics Statistical Tool Selection | 1

If instead, we had asked respondents to estimate the probability of winning the championship (0 100%), then this would have resulted in parametric data (ratio scale). The correct analysis for this data type would include computation of means (average probability of each team winning), and variance (distribution of scores for each team). Can you identify the type of data required to do a cross-tabulation analysis? How about a t-test? And finally, can you locate the multiple regression analysis?

A GUIDE TO THE SELECTION OF STATISTICAL TOOLS

Number of Variables Class of Statistics in the Analysis Non-Parametric Parametric and Type of Data (Nominal; Ordinal) (Interval; Ratio)

One 1. Discrete a. Percentage (Impossible) Variable 2. Continuous a. Median, Mode a. Mean b. Quartile Range b. Std. Deviation

Two 1. Both a. Chi Square a. Correspondence Variables Discrete b. Phi Coefficient analysis c. Contingency Coefficient d. Tetrachoric Corr. (2 rows x 2 columns) e. Yules Q (2 x 2) f. Lambda 2. One Discrete and the other Continuous a. Student t (if dependent variable is dichotomous) b. One-way ANOVA (if dependent variable is multichotomous) 3. Both a. Spearman Rank a. Pearson Correlation Continuous Correlation b. Simple Regression b. Kendalls R c. Eta Curvilinear c. Gamma Correlation

Qualtrics Statistical Tool Selection | 2

Three Variables 1. Three a. Cross Tabulation a. Multidimensioanl Scaling Discrete b. Correspondence Analysis (Multiple) 2. Two Discrete a. ANOVA (2 Way) One Continuous 3. One Discrete a. Comparison of a. Analysis of Two Continuous proportions Covariance b. Comparison of two correlations (If dependent variable multichotomy) c. Multiple Discriminant Analysis 4. Three a. Kendalls W a. First-Order Continuous Partial Correlation b. Multiple Regression c. Multiple Correlation

Four 1. K Discrete a. Cross a. Quasi-Metric Multidimen Tabulation sional Scaling b. Non-Metric MDS c. CHAID 2. One Discrete a. Prohibit Analysis K Continuous (For normal probabiltiy distribution) b. Logit Analysis (For Logistic Porbability Distribution) c. Multiple Discriminant Analysis 3. K Continuous Variables a. Second-Order Partial Correlation
b. Multiple Regression c. Multiple Correlation d. Factor Analysis e. Canonical Correlation f. Fully Metric Multidimensional Scaling

4. One Continuous a. CART K Discrete b. Multiple Regression (Dummy)

Qualtrics Statistical Tool Selection | 3