Objectives

Topics 1
Research Design

Topic 2
Designs for Nursing Research

Topic 3
Concepts Relevant to Design

Topic 4
Design Validity

Topic 5
Good Study Design

Topic 6
Modeling study design

References

TOPIC 4:
Design Validity

Validity of a study simply answers this question: Are the results a reflection of reality rather than being effects of extraneous variables?
Extraneous variables include:

1. Antecedent variables - characteristics the participants brought into the study
2. Intervening variables - factors that occur during the study and could influence the results. This includes factors in the environment and also lack of control in the design and implementation of the study.

Do the researchers make a persuasive argument and competent test of the study propositions/ relationships? Have the researchers considered threats to validity and implemented mechanisms to control these threats? Judgment of a study’s validity is a major basis for deciding whether the study findings are useful for patient care.

Each of the four types of validity is concerned with threats to the study design, but each provides a focus on a different aspect of the study design. Multiple threats to each type of validity can occur, and the most common threats are described here.

• Statistical conclusion validity is concerned with conclusions and results drawn from statistical analysis procedures. Are the conclusions real or false? This concept can be expressed with the following terms that you will hear repeatedly throughout the course:
  • Type I Error – A relationship between variables or differences between groups is found, but no relationships or differences exist. In other words, the researchers find results that do not exist.
  • Type II Error – No relationships or differences are found, but relationships or differences exist. In this instance, the researchers fail to find differences or relationships that actually exist. If no results are found, researchers have mechanisms for exploring the findings further to ensure that potentially important findings are not missed.
  • Threats
    • Low statistical power
    • Unreliable measures (operationally defined variables)
    • Unreliable implementation of treatment – Delivery treatment (intervention) differs between subjects.
• Internal validity is very similar to design validity. Are the findings reflective of reality (what the researchers are studying), or are the findings the result of uncontrolled extraneous variables?
  • Threats
    • Effects of repeated testing (pretest and post-test designs)
    • Selection – Subjects who agree to participate in a study may be very different from those who refuse participation.
    • Mortality/ attrition – Subjects withdraw from the study before completion and may be very different from the subjects who complete the study.
• Construct validity asks if the conceptual and operational definitions of the variables are congruent? In other words, are they measuring what they think they are measuring?
  • Threats
    • Mono-operation bias – One method of measurement is used to measure a variable, such as a single questionnaire.
    • Mono-method bias – More than one method of measurement is used, but they are all the same. For example, the researcher decides to measure stress with more than one instrument, but they are all paper and pencil measures when physiologic measures also could be used.
    • Hypothesis-guessing within experimental conditions – Subjects within the study attempt to guess the researcher’s hypothesis.
• External validity asks if the study results can be applied to anyone other than the study sample. Application beyond the study sample is called generalizability.
  • Threats
    • Interaction of selection and treatment – This addresses the idea that people who participate in a study may be very different from those who do not participate. Thus, the results obtained from the group that participates may not be valid in relation to non-participants.
    • Interaction of setting and treatment – This takes the idea of differences in who participates and who does not and applies it to the settings in which studies are conducted,
    • Interaction of history and treatment –Occurrences during the conduct of the study (the history during that time) may have influenced the treatment. Therefore, the treatment may not yield the same results at a different time.