KCTC Student Survey Reliability and Validity
The details below briefly explain validity and reliability assurances of the Kansas Communities That Care (KCTC) Student Survey.
Validity
Validity is concerned with the accuracy of measurement. There are several types of validity. Three important types are internal, external, and content.
Internal Validity
Internal validity asks whether the survey questions measure what is intended to be measured and if results are accurate or can be evaluated against a desired outcome. There are three validity checks built into the survey to identify false reporting, including responses that are not honest, or patterns that demonstrate lack of attention to the survey questions.
Evidence of false reporting includes:
- Students responded they were not honest at all when asked, “How honest were you in filling out this survey?”
- Students indicated they had used a non‐existent drug.
- Students reported unrealistically frequent use of marijuana, LSD or other psychedelics, cocaine or crack, and inhalants. An algorithm for this strategy represents the respondent’s past 30-day use for each substance by the midpoint of each indicated response range. If the sum of the midpoints of the indicated response range exceeds 120, the response is considered invalid.
- Students who do not answer any of the questions involved in determining the validity of the survey according to the first three checks above, the survey is considered invalid.
How Districts Can
Encourage Response Accuracy
- Stress the value and importance of the survey to building principals, teachers, and students.
- Assure students the survey is voluntary, anonymous, and confidential. Students need to know that no one will see their responses, and responses cannot be traced back to any individual student.
Based on these validity checks, if there is evidence of false reporting in a survey, the entire survey is determined to be invalid and eliminated from the data set. Each year, approximately 4‐7% of surveys are found to be invalid and are not included in analyses.
External Validity
External validity refers to the degree to which a sample is representative of or can be generalized to the larger population. A representative sample is a group chosen from a larger population that adequately replicates or looks like the population.
The figure below shows how the percentage of students by grade participating in the KCTC Student Survey sample looks very much like actual distribution of enrollment by grade in the population reported by the Kansas State Department of Education. For example, in the distribution of surveys in the KCTC sample, 28% are from 6th grade, which is similar to the actual distribution of 6th graders enrolled (25%). While there is some fluctuation by grade, overall, by grade the sample (KCTC) looks similar to the population it is trying to represent (KSDE enrollment). This is important as it shows that the sample is well distributed and not heavily skewed in one direction or another.
The KCTC uses a census approach, meaning results reflect only those students who participate in the survey. Therefore, participation rates per grade, building, and district must be high for data to be reflective of the grouping it represents. For further comparison, the gray bars for 10th and 12th grades represent the distribution by grade of students who participated in the Youth Risk Behavior Survey (YRBS) that is administered statewide every other year through a random sample. This not only shows that the KCTC sample looks very much like the KSDE population by grade, indicating that it is representative and has good external validity, but that it is also similar to other surveys that have a random sampling process. Good external validity is also demonstrated for gender, ethnicity, and population density comparisons in the KCTC sample.
The KCTC Student Survey has been administered annually free of charge to Kansas school districts since 1995. One strength of this longevity is the ability to look at trends in student behavior and attitudes over time. Most fluctuations in data that bring about questions regarding validity are the result of inconsistent or low participation.
Content Validity
Content or construct validity refers to the appropriateness of the content of an instrument. In other words, do the questions accurately assess what you want to know? The Communities That Care Survey was developed by prevention science researchers at the University of Washington (Hawkins, Catalano, & Miller, 1992). Content validity has been statistically proven through factor analysis of each question’s contribution to the risk and protective factor scale it represents (Arthur, et al., 2002; Glaser et al., 2005).
Reliability
Reliability is concerned with the consistency of the measurement over time; the degree to which the questions used in a survey elicit the same type of information each time they are used under the same conditions. The KCTC has proven to be very reliable over time.
- Results obtained by the KCTC are similar to results obtained when compared with other student surveys measuring substance use and with trends reported at the national level (Monitoring the Future, 2021).
- Risk and protective factors are correlated with problem behaviors in the expected direction (Hawkins et al, 1992).
- Measures were shown to be equally reliable across females and males and five racial/ethnic groups (Glaser, et al., 2005).
- Additionally, higher levels of risk factors reported by students in a school were associated with lower average reading and math achievement test scores in Kansas (Chaney & Mann, 2020).
- The CTC survey has been widely recognized as a valid and reliable instrument. The National Center on Safe Supportive Learning Environments listed CTC in the School Climate Survey Compendium of valid and reliable surveys, and the Collaborative for Academic, Social and Emotional Learning includes CTC in its guidance document.
- The RAND Corporation Assessment finder tool also endorses the Communities That Care Survey.
- In 2019 the Kansas School Mental Health Advisory Council and the Kansas State Board of Education encouraged Kansas school districts to administer the KCTC Student Survey as the result of recommendations made by the Kansas Blue Ribbon Task Force on Bullying.
References
- Arthur, M. W., Hawkins, J. D., Pollard, J., Catalano, R. F., and Baglioni Jr., J. (2002). Measuring risk and protective factors for substance use, delinquency and other adolescent problem behaviors: The Communities That Care Youth Survey, Evaluation Review, 26: 575-601.
- Chaney, L., & Mann, C. (2021). Relationships between student climate types, school demographics, and academic achievement in Kansas. Girard, KS: Learning Tree Institute at Greenbush with Kansas MTSS and Alignment.
- Glaser, R. R., Van Horn, M. L., Arthur, M. W., Hawkins, J. D. & Catalano, R. F. (2005). Measurement Properties of the Communities That Care Youth Survey across Demographic Groups Journal of Quantitative Criminology, Vol. 21(1), 73-102.
- Hawkins, J. D., Catalano, R. F., and Miller, J. Y. (1992). Risk and protective factors for alcohol and other drug problems in adolescence and early adulthood: implications for substance-abuse prevention. Psychol. Bulletin, 112: 64–105.
- Monitoring the Future, National Survey Results on Drug Use. Bethesda, Md: National Institute on Drug Abuse, U.S. Dept. of Health and Human Services, National Institutes of Health, 2021. Print. https://nida.nih.gov/research-topics/trends-statistics/monitoring-future
More Info
If you have questions about the survey, contact the KCTC Team at Greenbush at [email protected], or call 620-724-6281 ext. 366.