(2026)
Field Trial Sample Size for Scales (FiTS3): Requirements for Questionnaire Scale Analysis
R&D - FITS3 - Final Report v3.pdf332.14 KB
Besides assessing student achievement, IEA studies collect valuable information through context questionnaires.
These questionnaires are evaluated during a field trial (FT), intended to test the functioning of context scales
in each country and to provide empirical evidence to revise questionnaire material. However, the usefulness of
the FT depends on the quality of the evidence it can provide, and a key issue in this regard is the sample size
per country.
While previous studies have explored the topic of minimum sample size for scale construction and evaluation,
such studies are theoretical or done with simulated data. Whether these findings apply to real data and,
particularly, IEA studies remains unclear. To investigate this matter, we used existing student and teacher
questionnaire data from TIMSS 2019 and ICILS 2018, with available data for up to 65 educational systems and
70 scales.
We analyzed how different sample size minimums influence the distribution and precision of the indicators
obtained from a confirmatory factor analysis with real data scenarios. We focused on how different goodness-offit
indices (CFI, TLI, RMSEA, SRMR) change between sample size minimums and how this may impact scale
adequacy decisions based on common rule-of-thumb thresholds.
Our analysis confirms that bias decreases as sample size increases and supports the N ≥ 200 guideline, where
bias remains minimal. However, our findings also indicate that samples of N ≥ 100 observations may still
provide reliable evidence, with only a slightly higher bias compared to having 200 observations. Therefore,
while N ≥ 200 remains the preferred threshold, sample sizes in the range of 100-200 observations may be a
practical and valid alternative in certain contexts.
These questionnaires are evaluated during a field trial (FT), intended to test the functioning of context scales
in each country and to provide empirical evidence to revise questionnaire material. However, the usefulness of
the FT depends on the quality of the evidence it can provide, and a key issue in this regard is the sample size
per country.
While previous studies have explored the topic of minimum sample size for scale construction and evaluation,
such studies are theoretical or done with simulated data. Whether these findings apply to real data and,
particularly, IEA studies remains unclear. To investigate this matter, we used existing student and teacher
questionnaire data from TIMSS 2019 and ICILS 2018, with available data for up to 65 educational systems and
70 scales.
We analyzed how different sample size minimums influence the distribution and precision of the indicators
obtained from a confirmatory factor analysis with real data scenarios. We focused on how different goodness-offit
indices (CFI, TLI, RMSEA, SRMR) change between sample size minimums and how this may impact scale
adequacy decisions based on common rule-of-thumb thresholds.
Our analysis confirms that bias decreases as sample size increases and supports the N ≥ 200 guideline, where
bias remains minimal. However, our findings also indicate that samples of N ≥ 100 observations may still
provide reliable evidence, with only a slightly higher bias compared to having 200 observations. Therefore,
while N ≥ 200 remains the preferred threshold, sample sizes in the range of 100-200 observations may be a
practical and valid alternative in certain contexts.