Pre-Conference Workshops

In advance of the main IRC meeting, IEA offered two-day, on-site workshops on specialized topics related to large-scale assessment from 14-15 November 2021

The workshops provided a stimulating and practical learning environment for all who wished to improve their understanding of and gain practice in working with data from large-scale international assessments such as those conducted by IEA. Each workshop varied in its focus and level; the specific topics are described in more detail below. 

IEA conducts regular workshops at international research conferences and in collaboration with the IERI Institute (www.ierinstitute.org). For more information, consult our training page.

 
 
1. Statistical Analysis of IEA Data Made Easy: Using the IEA IDB Analyzer (Basic)
Yasin Afana (Language of Instruction in Arabic)

Data from large-scale assessments are collected from complex samples and use specific methodology to examine the achievement of students. This is why base modules of statistical software packages cannot be used to derive correct statements on the state of education in the assessed populations. With the IDB Analyzer, IEA has developed an easy-to-use software that works as an add-on for SPSS, SAS and R. In this workshop we will work with the SPSS version, and will use TIMSS 2019 data for all presentations and practical examples. The goals of the workshop are to familiarize participants with the statistical complexities involved in working with international large-scale assessment databases, and familiarize participants with the data structure of TIMSS. Participants will learn how to run statistical analysis with TIMSS data, and interpret the results correctly. The workshop will balance presentations and hands-on practical assignments.

After the workshop, participants will be able to:

  • Understand the complexities of large-scale assessment data
  • Merge different data files
  • Combining variables, creating and validating indices
  • Reproduce results shown in the exhibits of the TIMSS 2019 International Report
  • Estimate percentages, means, correlation and regression coefficients and their standard errors given the complex designs

About the Instructor:

Yasin Afana works for more than two decades in the field of educational measurement and quantitative research, and about 16 years with IEA. Dr Afana has instructed multiple courses on the IEA data statistical analysis. He just recently obtained his professional doctorate degree in educational research from the University of Leicester in UK. His main research interest is in school value-added and improvement, as well as in research design and methodology.

 
 
2. Analyzing IEA data with R (basic)
Umut Atasever and Diego Cortes

In order to conduct targeted research using existing education data, researchers must have basic knowledge of statistical software in order to perform preliminary analyses, subset data to the target population of interest, and carry out primary analysis. A number of such software applications exist, but many are commercial while still requiring a steep learning curve for basic functionality. A free software application growing in use among the research and academic community is called R and has an active user base willing to discuss and exchange ideas, as well as offer solutions to basic and difficult programming questions.

R is one of the most important tools for statisticians presently used. It is an open-source project software environment and programming language with a wide variety of options that is highly extensible. It is not only used and supported by a broad and very diverse research community, but also by companies like Microsoft, Google, RStudio, and Data Camp that are actively promoting R as a lead platform.

This hands-on workshop is targeted to individuals who want to learn the basics of R and become familiar with this free and flexible software application. It will introduce participants to the differences between R and other statistical software such as SPSS. Participants will become familiar with installing R “packages,” the software’s use of functions to perform analyses, and data set manipulation. Furthermore, participants will be introduced to the recently implemented IEA IDB Analyzer integration with R to perform analyses with large-scale assessment data.

After the workshop, participants will be able to:

  • Understand the differences between R and SPSS
  • Install packages in R
  • Import data into R
  • Perform subsetting, filtering, and recoding with data
  • Understand different methods for storing information on missingness in data with R
  • Use functions to perform descriptive and inferential analyses
  • Use R in conjunction with the IEA IDB Analyzer
  • Understand the troubleshooting resources available for R users

Programming knowledge learned during this course aims to create a foundation for researchers in their use of R and R resources.

Prerequisites

Participants should have knowledge of basic descriptive and inferential analyses (e.g., frequencies, correlation, regression, significance testing) and some basic knowledge of large-scale assessment (e.g., weighting, plausible values) to understand the integration between R and the IEA IDB Analyzer.

About the Instructors:

Umut Atasever and Diego Cortes both work at the IEA Sampling Unit with major responsibilities in ICCS and TIMSS. They are part of the team who developed the R syntax for the IDB Analyzer software and have multiple years of experience analyzing data from IEA studies with R. Both support the Research and Analysis unit on a regular basis and pursue projects with particular focus on the design and methodology of educational surveys.

Diego focuses on different aspects of educational policy. Further, being responsible for sampling in ICCS, he is particularly interested in the design and methodology of educational surveys.

 
 
3. Methods for Causal Inference with Data from International Assessments
Rolf Strietholt and Andrés Strello
Rolf Strietholt and Andrés Strello

Much comparative educational research poses questions about causal effects but cannot employ experimental techniques to investigate them. However, during the last couple of decades analytical techniques have been developed within statistics, econometrics, and other disciplines, which support causal inference from cross-sectional and longitudinal data. The workshop presents designs that incorporate so-called fixed effects for the identification of causal effect: trend analyses (e.g., TIMSS 2015, 2019), comparisons between educational stages (e.g., grade 4, grade 8), within-student-between-grade designs (e.g., math, science), and school fixed effects.

In this workshop, the Rubin causal model (RCM) will be used as a framework of potential outcomes to define causal effects. Key differences between experimental and observational designs will be discussed to explain issues such as selection into treatments, omitted variable bias, and reversed causality. Thereafter, we will introduce different methods that address these issues in a non-technical way. We will review some studies that have employed the data from international assessments such as PIRLS and TIMSS to illustrate the methods. In a hands-on session, we will use data from IEA-studies for practical and interactive activities. 

After the workshop, participants will be able to:

  • Understand what a causal effect is;
  • Be able to identify issues related to the identification of causal effects with observational data;
  • Understand limitations of regression analysis; and
  • Know designs that incorporate fixed effects for countries, schools, students, years, and educational stages.

The workshop will be a mixture of lectures and hands-on training, to ensure participants gain both sound knowledge and practical expertise.

Prerequisites:

Participants should have solid knowledge of regression analysis and be familiar with the use of syntax code in some statistical software such as R, SPSS, or Stata.

About the Instructors:

Rolf Strietholt is the Co-Head of the IEA Research and Analysis Unit. His main research interests include school and educational effectiveness research and research methodology.

Andrés Strello is a researcher at the IEA Research and Analysis Unit. He is interested in educational inequality, comparative education, differentiation, and quantitative methods.