In advance of the IEA International Research Conference (IEA IRC), IEA offered two-day, on-site workshops covering specialized topics related to large-scale assessment from 23–24 June 2025.
The workshops provided a stimulating and practical learning environment for all those who wish to improve their understanding of, and gain practice in working with data from large-scale international assessments such as those conducted by IEA.
Much comparative educational research poses questions about causal effects, but cannot employ experimental techniques to investigate them. However, during the last couple of decades analytical techniques have been developed within statistics, econometrics, and other disciplines, which support causal inference from observational data. The workshop presents examples of so-called quasi-experimental designs in the context of educational research for the identification of causal effects: instrumental variables, regression discontinuity, difference-in-differences, and fixed effects. These include approaches such as exploiting random variation due to class size or age of entry policies in schools, country-level longitudinal analyses (e.g., TIMSS 2011, 2015, 2019), comparisons between educational stages (e.g., grade 4, grade 8), within-student-between-grade designs (e.g., math, science), among other strategies.
In this workshop, causal effects will be defined using the potential outcomes framework from the Rubin causal model (RCM). Key differences between experimental and observational designs will be discussed to explain issues such as selection into treatments, omitted variable bias, and reverse causality. Thereafter, we will introduce different methods that address these issues in a non-technical way. During the discussion of these methods, we will review some studies that have employed the data from international assessments such as PIRLS and TIMSS and share R code examples replicating their results to illustrate how these methods can be applied to real data. Several example studies will be discussed in groups, giving participants practice in identifying identification strategies as well as discussing the potential and limitations of each method in the context of international large-scale assessments.
After the workshop, participants will be able to:
- Understand what a causal effect is;
- Be able to identify issues related to the identification of causal effects with observational data;
- Understand limitations of regression analysis;
- Know designs and strategies to identify effects with observational data from large-scale assessments.
Methods
The workshop will be a mixture of lectures, group discussions, and hands-on examples, to ensure participants gain both theoretical background and real-life examples applicable to educational research.
Prerequisites
Participants should have solid knowledge of regression analysis. Being familiar with the use of syntax code in a statistical software such as R or Stata is recommended.
This workshop introduces the basics of Item Response Theory (IRT) and the latent regression model, also known as the population or conditioning model. These concepts are essential for generating achievement scores in international large-scale assessments, especially when matrix-sampling booklet designs are used, as seen in TIMSS, PIRLS, ICILS, and ICCS.
The workshop will focus on practical applications, helping participants understand assessment design and IRT. It will provide hands-on training in using R packages like mirt and TAM to fit item response and latent regression models. Participants will learn to estimate item and person parameters, and compute plausible values.
The workshop is structured into four main parts:
- Assessment designs: This section will provide an overview of the matrix-sampling booklet design used in major ILSAs, where each subject receives a subset of the item pool.
- IRT Models: Participants will be introduced to the basic concepts of Item Response Theory, including the estimation of item and person parameters.
- Population modeling and proficiency estimation: This section will focus on the procedures for implementing population models and generating plausible values through multiple imputations.
- Application of plausible values in analysis: This section will demonstrate how using plausible values can reduce bias in secondary analyses.
Methods
The workshop will include a mix of lectures, demonstrations, and hands-on examples to ensure participants learn how to apply IRT and population modeling in large-scale assessments. This will enable them to conduct item calibration, estimate person abilities, and evaluate item fairness and test quality. The examples will primarily focus on assessment and background data from the ICILS 2023 study.
Prerequisites
The workshop is suitable for anyone interested in learning about the principles of assessment design used in large-scale assessments and the psychometric methods currently employed to analyze these data. To participate effectively, attendees should have a basic understanding of R. Although it is not a requirement, it is an asset for following the examples and hands-on activities.
This workshop will present variables derived from event data for a publicly released ICILS computational thinking module, as available in the ICILS 2023 International Database (IDB).
The focus will be on the structure and format of these derived variables, including the rationale behind data logging and the aggregation of events during task completion. Participants will explore several analysis examples demonstrating how the variables can be used to identify patterns in students’ engagement, problem-solving approaches, and response strategies.
The workshop will conclude with an introduction to an ICILS process data package, which includes timestamped event data from multiple annotated tasks within a second publicly released computational thinking module.