Research Methods 2013 Study Notes

Session 1 Research Methods - Compare, Replicate, Generalize Five Methods for Knowledge Acquisition The adherence method o Simply accept the torah The intuitive Method o i.e. Canadians do not feel cold in Israel. The Authoritative Method o Listen to the Rabbi The Rational Method The Empirical Method o Objective, Empirical, Solvable

Scientific Process Problem definition Predictions Data collection Data analysis and interpretations Empirical Generalizations

Scientific Models Deductive: General -> Specific Inductive: Specific -> General (scientific model)

The Principals of the Scientific Method Empiricism o Experiment on data not logic Testable, public Deterministic o Some kind of rule exists in the environment Parsimony o Passable research, picking experiments with least number of variables Generality o Be able to induce general conclusions.

Applied vs. Basic Research Applied o Examines subjects to their natural environment, many variables Basic o Test theoretical issues, more control over the research environment (for example using animals) 1

Session 2 Operational Definition How the test was conducted, all the details (i.e. it was 20 questions Multiple Choice, they did it during the day, had an hour)

Variables Dependent influenced variable Independent influencing variable Operated Independent Variables variables that may be manipulated and used to randomly assign subjects to experimental groups (eg. Type of music in store, brand name) Attribution Independent Variables difficult to manipulate and to be used for random assigning (eg. Gender, income, IQ)

Session 3 *Measurement Scales (N.O.I.R) (VERY IMPORTANT)

Nominal Scale (tags) o 1 Male, 2 Female numbers do not have meaning Ordinal Scale (rankings) o 1 = private, 2 colonel numbers have ranks o Or rank 1,2,3 for Heineken, Carlsberg, Corona Interval Scale (fixed intervals) o How much do you like Goldstar, rank either 1,2,3,4,5,6,7 (needs at least 5 rankings) Ratio Scale o Meaningful zero; zero doesnt exist or doesnt have (ie, how many times a month do you drink beer)

Session 4 Research Designs Correlational Research o Relationships between variables E.g. product satisfaction is related to increased consumption When the Independent Variable (IV) is an attributed IV the research is correlational. Experimental Research o Examines causality between variables E.g. product satisfaction leads to increased consumption When the IV is an operationalized IV the research is experimental. When can we establish Causality? o Correlation = relationship between two variables o Causality = one variable affects another variable o Correlation causality o Conditions necessary for establishing causality Correlation between the variables Temporal antecedence the independent variable must come before the dependent variable No third factor driving both

Session 5 Measurement Characteristics o Reliability How stable/accurate is the measurement tool Measured Score = real score + random error o Validity Does the measurement reflect what we want to measure Errors o Systematic Error - Error same magnitude and same direction (say always -5 pounds), does not affect reliability (affects accuracy) o Random Errors - vary errors one time to another, error is different every time I walk on the scale, its random not systematic. Reduces reliability and accuracy. Subjects mood, degree of alertness. o We are not worried about systematic errors, we are always worried about Random Errors.

Reliability o Long-Term Stability - Test-retest reliability o Compatibility - between parallel versions o Between/Within Judges Consistency between different judges degree of comparability between judgments of different judges Consistency within judges - degree of comparability between judgments of the same judge o Internal Consistency - among the measure index index/split test. Cronbachs Alpha o Measures the reliability as internal consistency this is a measure for the correlation between different test items (questions) in the same measurement tool (questionnaire) o o o
( )

Alpha range from 0-1, closer to 1 stronger reliability. (0.7 strong) Researcher's ask you the same question many times check to see if the test is reliable or not. Often people will not read the questions fully, often they will sometimes give a score slightly higher or lower than the true value. When we average out all these numbers, the random error is eliminated.

Session 6 Construct Validity o Examines whether we measure what we want to measure, is the match between the theoretical concept and the operational measure. i.e. Does heartbeat rate measure theoretical variable of anxiety? Problems with Construct Validity o Lack of exhaustion The measure does not measure all the variance in the theoretical concept, only part of it o Lack of exclusiveness The measure is related to other theoretical concepts as well. Conditions for Construct Validity o Reliability If measurement is valid must be reliable, doesnt measure random errors. o Content Validity Ask an expert if the test accurately represents, measurement of content is clearly defined. o Convergence Validity The existence of a high correlation between the score of the measurement tool and the score of another measure that examines the same concept and that has already been validated. o Discriminant Validity The existence of a low correlation between the score of the measurement tool with scores of other valid measures that examine other (but similar) concepts. Predictive Validity o A measure for the future rate of the phenomenon i.e. to what extent does the psychometric test predict success at the university? Sensitivity of Measurement Tools o Refers to the extent to which the measurement tool is sensitive to changes in the level of theoretical variable. o Over-sensitive Detects frequent changes in response to irrelevant information. o Under-sensitive A tool that misses important information that originate from the manipulation of the IV. Ceiling & Floor Effects additionally to sensitivity, a measurement tool should detect a range of values and avoid the ceiling and floor effect. o Ceiling effect The measure cannot express variance in high values o Floor effect The index cannot express variance low values. 5

Session 7 Important Notes, I recommend to read the class slides Survey A questionnaire to collect data, to predict behavior based on current positions. Questionnaire convenient way to collect data when the phenomena cannot be observed directly, attitudes, emotions, etc. o Method for Administering Questionnaires Mail (nowadays no used much), phone, web, fill in the presence of a researcher, Face-to-Face interview. Survey/Questionnaire o Advantages Suitable for variety of topics, easy to administer to a broad population, low costs and little time o Disadvantages May be susceptible to selection bias, i.e. people who are willing to participate in the survey may also be more friendly. o Stages in Design 1) Prepare list of relevant variables (Exhaustion Principal do we have all variables we need) 2) Draft an initial version (consult with experts) 3) Pre-test the questionnaire on a small sample (Pilot) 4) Draft the final version

Types of Questions o Closed vs. open questions Closed offer respondents several possible answers to choose Open do not provide guidance for answering, respond how see fit o Direct vs. indirect (projection) questions Direct: You smoke ganja? Indirect: Do your friends smoke ganja? o Conditioning vs. conditional questions Follow up question depends on response to previous answer If you are married answer question 1, otherwise move to question 2. Question 1 - what year did you get married? Possible Problems in Formulating OPEN Questions (VERY IMPORTANT) o Double-negative formulation Use positive formation. i.e. To what extent do you not oppose gay marriage? o Vague formation Clarity concepts. i.e. To what extent do you like hedonic products, products that are used for pleasure? o Multi-Aspect Formulation Split into multiple questions. i.e. To what extent do you agree with the governments economic and security actions. Could support the economic and not the security. o Biased Formulation Presenting a certain view as the normative view, the use of emotionally charged terms. i.e. To what extent do you agree with the conservative view that couples should not live together before marriage? Possible Problems in Formulating CLOSED Questions (VERY IMPORTANT) Exhaustive Principal Include all possible responses, if you are unsure youve exhausted the world of content, add the option other. Exclusion Principle Avoid overlapping between different options. Relevance Principle Ensure that all answers are relevant to the question wording. Balance Principle When using a rating scale present an equal number of positive and negative possibilities (Love it* Like it*, Quite like it*, Hate it**) Numerical/Verbal/Mixed Rating o Numerical How old are you? How many cigarettes do you smoke a day? 1) Up to 10, 2) 11-20, 3) +21 o Verbal Who are your three favorite election candidates? o Mixed Have you ever used light drugs? 1) No 2) Yes. Please elaborate. _____________ 7

Absolute/Relative Rating o Absolute when ranking options separately and independently of other categories. Rate the extent to which each of the following 10 qualities are important to leadership, on a 5 point scale. You can give a 5 rating for all qualities o Relative when choosing from a series of given answers and/or rating them in a particular order Rand the following 10 qualities from 1 to 10. Each quality gets a different rating. Even/Odd Scale o Even Scale 123456 Advantage People do not pay much attention when they fill out a survey, therefore giving many neutral answers. Disadvantage Does now allow neutral response, therefore Exhaustion Principal is not provided. Professor has preference to the even scale o Odd Scale 12345 Advantage - The existence of a neutral answer subject is not forced to take a stand. Disadvantage May create a bias not to take a clear stand Indices: Likert Scale o Series of arguments with equal weight I love Bamba Osem 1 2 3 4 5 I buy Bamba Osem 1 2 3 4 5 Osem is my favorite Bamba 1 2 3 4 5 Indices: Semantic Differential o Series of bipolar sequence of opposite attributes. Tasty 123456 Tasteless Fresh 123456 Old Sweet 123456 Bitter Expensive 123456 Cheap

Additional Biases o Biases originating from the subject Lack of motivation/interest/fatigue Social Desirability tendency to agree with norms and social conventions Halo Effect Response to a dominant and prominent item may affect responses to following items. i.e. rate the educational system in terms of: efficiency, organization, investment in developing new programs etc.

Biases originating from the relationship between the interviewer and the subject Subject is not comfortable exposing him/herself due to interviewer characteristics (i.e. woman/familiar person) The interviewer direct the subjects answers, relevant in face-to-face interviews (the interviewer fills out the questionnaire)

Session 8 Basic Concepts o Population: a group of items to which we want to generalize the research results o Sample: a subset of the population, on which the research is conducted o Sampling Frame: a subset of the population from which we sample the individuals for the sample.

Two Main Sampling Methods o Probability Sampling Random sampling each element in the population has equal chance of being sampled The sample represents the population Simple Random Sampling o Each item in the population has the same chances of being included in the sample. Systematic Sampling o Not used anymore o Carried out through random decisions on the first unit and then 9

systematically skipping listed items by a predetermined number. i.e. questionnaires by phone. Stratified Sampling o Most effective form of random sampling o Ensure all groups are represented, done by dividing the population into groups/layers and randomly sampling from each group/layer. Cluster Sampling o Used when it is possible to identify population units that represent the rest of the population. o Subjects are not sampled individually but as a group. i.e. sampling specific year from each IDC school to represent the entire IDC student population.

Non-probability sampling Non-random sampling different elements in the population may have different chances of being selected We use non-probabilistic sampling when it is difficult to define the population, locate subjects, or get subjects cooperation Methods of non-probabilistic sampling Snowball sampling o We use existing participants to get information on other subjects (referrals). Volunteer sampling o When there is a problem of cooperation on behalf of the participants, we advertise and seek volunteers. Quota sampling o When there is no record of the entire population we set a quota of subjects from each subset of the population and fill it. o I need twins, I will keep looking until I find 20 pairs. Convenience sampling o Based on unregistered call for subjects to whom we have easy access (close people, friends) o Good for pilot research Sampling Methods: Summary Method Advantage Disadvantage Probabilistic Good representation Difficult to preform Non-Probabilistic Convenience and ease Problematic representation

Sampling Error 10

o o o o

Chance our sample does not represent the target population Larger the sample, smaller sampling error Recommendation of at least 30 subjects per experimental condition (IV level)

External Validity o The ability to generalize the results from the study sample to the target population. Does not refer to the measurement tool, refers to the study in general. To what extent can I generalize the details of this study to the target population. o Problems that may damage external validity Non-representative sample (increase sample size) Specific time of the study (preform study at varying times)

Session 9
Internal Validity o Extent which alternative explanations for the casual relationship between the study variables can be ruled out Experimental Control o Includes techniques for increasing internal validity. o Experimental control attempts to gain control over The Independent Varible Extraneous Variables (Very Important) Additional variables operating in the experimental environment accept the IV. For example, an experiment that examines the effect of Vitamin A on vision. o External Variable: The group that received the vitamin was tested in the morning and the group that did not received the vitamin was tested in the afternoon o Random Variable Different subjects have different visual ability.

Common Extraneous Variables 11

o o o o o o o o

There are seven common extraneous variables that affect internal validity: Selection the experimental groups are different in the DV a-priori. History - another event occurred during the study that might have affected the results. Testing - If there are multiple measurements, they can influence each other. Maturation - a process of change/development/maturation of the participants (particularly relevant in longitudinal studies). Mortality selective dropout. Instrumentation - if there are some measurements the instruments may change between measurements. Statistical regression when participants are selected to experimental groups based on pre-tests These problems threaten only studies that attempt to establish causal inference

EXAMPLE - an experiment examined the effect of advertisement with a woman vs. man on willingness to buy the iPhone: o Selection Women watched the ad with the man and men watched the ad with the woman (women may like iPhones more). o History Between viewing the ad with the woman and the ad with the menthe iPhones antenna problem was made public. o Testing The ad with the woman caused an elated mood that influenced the effect of ad with the man. o Maturation Subjects lost their patience between viewing the ad with the woman and watching the ad with the man. o Mortality Some participants decided to leave the study after watching the ad with the woman but before watching the ad with the man. o Instrumentation The GSR device malfunctioned unbeknownst to the experimenter and gave lower scores in the second measurement. o Statistical regression The researchers recruited subjects who indicated in a pretest that they love iPhones

Experimental control 12

Methods for overcoming or controlling the impact of extraneous variables and thus contribute to increasing internal validity o Three primary methods Elimination Identify and eliminate the influence from the experiment o i.e. A room in which the woman was presented was more noisy than the room which the man in the ad was presented. o Eliminate source of noise o Problem: gender, prior knowledge, and certain variables cannot be eliminated. Equal Conditions Hold everything constant o i.e. to ensure that the chairs in both rooms are equally as comfortable for the participants. o Problem: cannot create equal conditions with any variable, such as personality type. Balancing Balancing the influence when we cannot eliminate or equalize conditions. o Examples: Selection: Women watched the ad with the man and the men watched the ad with the woman. Solution: each experiment group includes 50:50 men and women. Maturation: Participants lost their patience between viewing the different ads Solution: half the subjects will first watch the ad with the man, and the half will watch the ad with the woman. Randomization o Process of random assignments of subjects to the experimental groups. o Enables balancing effects of extraneous variables o Law of Central Limit Larger samples we take, more likely the influence of extraneous variables will be balanced. (Min 30 subjects each experiment group)

13

Session 10 Basic Symbols and Notations

------

Types of Experimental Designs True/Pure experimental designs o We can draw casual conclusions o Key Principles Operationalizing the IV and creating experimental group (EG) receives experimental treatment (X) Controlling the experiment by using a control group (CG) Random assignments of participants to the experimental and control groups (R)

14

Three true/pure experimental designs After only with a control group design i.e. How does Macys ad affects willingness to purchase there? o Sample 60 random consumers R o Randomly assigned to either CG or EG o Only EG exposed to the ad o Effects of Ad measured by difference between CG and EG in willingness to purchase ------Two Groups ------------------------------------------------------o R X 01 Experimental Group (EG) o R 02 Control Group (CG) o -------Manipulation effect (O1-O2) -------------------------

Before-After with a control group design i.e. o Sample 60 random consumers R o Randomly assigned to either CG or EG o All subjects will rate their willingness to buy at Macys Measurement 1 (01) o Only subjects in the experimental group (EG) expose to the ad. o All subjects will rate their willingness to buy at Macys Measurement 2 (02) o Effect measured by difference between the EG and CG in the change in willingness to purchase ------Two Groups ------------------------------------------------------o R 01 X O2 (EG) o R 03 O4 (CG) o -------Manipulation effect (O1-O2)-(O3-O4)------------------

Solomons Four-Group Array This array examines the influence of: o The experimental treatment (the manipulation) o The measurement This array is not often used b/c expensive and time consuming ------FOUR Groups ------------------------------------------------------o R O1 X 02 (EG) o R O3 04 (CG) o R X 05 (EG) o R 06 (CG) 15

False experimental desists o We cannot draw casual conclusions (not recommended) o Validity is difficult to assess, however, they are widely used o Three false experimental designs Case Analysis For example o Sample 30 consumers o All subjects exposed to the Ad (X) o All subjects rate willingness to purchase at Macys (O) o Effect measured by willingness to purchase after watching the Ad o Problem Unclear whether willingness to purchase was affected by the manipulation Manipulation Effect: O 1 GROUP -------------- X O Before After without a control group o Sample 60 consumers o All subjects rate willingness to purchase, Measurement 1 (O1) o All subjects will be exposed to the ad o All subjects rate willingness to purchase, Measurement 2 (O2) o Effects measured by differences in the two measurements o Manipulation Effect: (O1-O2) 1 GROUP -------------- O1 X 02 EG After only with a control group without randomization For example o Sample 60 consumers o Only subjects in the EG will be exposed to the ad o Rate willingness to buy at Macys o Effects measured by differences in willingness to purchase between EG and CG o Problem No random assignment susceptible to effects of exogenous variables -------- 2 GROUP ------------------------------o X O1 EG Manipulation Effect: O1-02 o O2 CG

Session 11 Class notes which are a summary of the entire year, highly recommended 16