KCTC Student Survey Reliability and Validity

The details below briefly explain validity and reliability assurances of the Kansas Communities That Care (KCTC) Student Survey.


Validity is concerned with the accuracy of measurement. There are several types of validity. Three important types are internal, external, and content.

Internal Validity
Internal validity asks whether the survey questions measure what is intended to be measured and if results are accurate or can be evaluated against a desired outcome. There are three validity checks built into the survey to identify false reporting, including responses that are not honest, or patterns that demonstrate lack of attention to the survey questions.

Evidence of false reporting includes:

  • Students responded they were not honest at all when asked, “How honest were you in filling out this survey?”
  • Students indicated they had used a non‐existent drug.
  • Students reported unrealistically frequent use of marijuana, LSD or other psychedelics, cocaine or crack, and inhalants. An algorithm for this strategy represents the respondent’s past 30-day use for each substance by the midpoint of each indicated response range. If the sum of the midpoints of the indicated response range exceeds 120, the response is considered invalid.
  • Students who do not answer any of the questions involved in determining the validity of the survey according to the first three checks above, the survey is considered invalid.

How Districts Can
Encourage Response Accuracy

  • Stress the value and importance of the survey to building principals, teachers, and students.
  • Assure students the survey is voluntary, anonymous, and confidential. Students need to know that no one will see their responses, and responses cannot be traced back to any individual student.

Based on these validity checks, if there is evidence of false reporting in a survey, the entire survey is determined to be invalid and eliminated from the data set. Each year, approximately 4‐7% of surveys are found to be invalid and are not included in analyses.

External Validity

External validity refers to the degree to which a sample is representative of or can be generalized to the larger population. A representative sample is a group chosen from a larger population that adequately replicates or looks like the population.

The figure below shows how the percentage of students by grade participating in the KCTC Student Survey sample looks very much like actual distribution of enrollment by grade in the population reported by the Kansas State Department of Education. For example, in the distribution of surveys in the KCTC sample, 28% are from 6th grade, which is similar to the actual distribution of 6th graders enrolled (25%). While there is some fluctuation by grade, overall, by grade the sample (KCTC) looks similar to the population it is trying to represent (KSDE enrollment). This is important as it shows that the sample is well distributed and not heavily skewed in one direction or another.

The KCTC uses a census approach, meaning results reflect only those students who participate in the survey. Therefore, participation rates per grade, building, and district must be high for data to be reflective of the grouping it represents. For further comparison, the gray bars for 10th and 12th grades represent the distribution by grade of students who participated in the Youth Risk Behavior Survey (YRBS) that is administered statewide every other year through a random sample. This not only shows that the KCTC sample looks very much like the KSDE population by grade, indicating that it is representative and has good external validity, but that it is also similar to other surveys that have a random sampling process. Good external validity is also demonstrated for gender, ethnicity, and population density comparisons in the KCTC sample.

The KCTC Student Survey has been administered annually free of charge to Kansas school districts since 1995. One strength of this longevity is the ability to look at trends in student behavior and attitudes over time. Most fluctuations in data that bring about questions regarding validity are the result of inconsistent or low participation.

Content Validity
Content or construct validity refers to the appropriateness of the content of an instrument. In other words, do the questions accurately assess what you want to know? The Communities That Care Survey was developed by prevention science researchers at the University of Washington (Hawkins, Catalano, & Miller, 1992). Content validity has been statistically proven through factor analysis of each question’s contribution to the risk and protective factor scale it represents (Arthur, et al., 2002; Glaser et al., 2005).


Reliability is concerned with the consistency of the measurement over time; the degree to which the questions used in a survey elicit the same type of information each time they are used under the same conditions. The KCTC has proven to be very reliable over time.


More Info

If you have questions about the survey, contact the KCTC Team at Greenbush at [email protected], or call 620-724-6281 ext. 366.