Pre-Conference Workshops

In advance of the main IRC meeting, IEA offers two-day, on-site workshops on specialized topics related to large-scale assessment. The workshops are offered in parallel from 14-15 November 2021.

The aim of the workshops is to provide a stimulating and practical learning environment for all those wishing to improve their understanding of and gain practice in, working with data from large-scale international assessments such as those conducted by IEA. Each workshop varies in its focus and level; the specific topics and prerequisites are described in more detail below. Lectures and discussions will be conducted in English.

The workshop participation fee is 200 euros. Attendance is limited to 25 participants per workshop. Participants must register before 19 October 2021 to secure a place (registration is on a first-come, first-served basis). Please note that in-person participation is required to attend the on-site workshops.

Registration for the workshops is open, to register, please go to the registration system. If new to the system, delegates are prompted to submit an email address and choose a password. Once in the system, all delegates can register.

IEA conducts regular workshops at international research conferences and in collaboration with the IERI Institute (www.ierinstitute.org). For more information, consult our training page.

To register, please go to the registration system. If new to the system, delegates are prompted to submit an email address and choose a password. Once in the system, all delegates can register.

 
 
1. Analyzing IEA data with R (basic)
Justin Wild and Diego Cortes
Justin Wild and Diego Cortes

In order to conduct targeted research using existing education data, researchers must have basic knowledge of statistical software in order to perform preliminary analyses, subset data to the target population of interest, and carry out primary analysis. A number of such software applications exist, but many are commercial while still requiring a steep learning curve for basic functionality. A free software application growing in use among the research and academic community is called R and has an active user base willing to discuss and exchange ideas, as well as offer solutions to basic and difficult programming questions.

R is one of the most important tools for statisticians presently used. It is an open-source project software environment and programming language with a wide variety of options that is highly extensible. It is not only used and supported by a broad and very diverse research community, but also by companies like Microsoft, Google, RStudio, and Data Camp that are actively promoting R as a lead platform.

This hands-on workshop is targeted to individuals who want to learn the basics of R and become familiar with this free and flexible software application. It will introduce participants to the differences between R and other statistical software such as SPSS. Participants will become familiar with installing R “packages,” the software’s use of functions to perform analyses, and data set manipulation. Furthermore, participants will be introduced to the recently implemented IEA IDB Analyzer integration with R to perform analyses with large-scale assessment data.

After the workshop, participants will be able to:

  • Understand the differences between R and SPSS
  • Install packages in R
  • Import data into R
  • Perform subsetting, filtering, and recoding with data
  • Understand different methods for storing information on missingness in data with R
  • Use functions to perform descriptive and inferential analyses
  • Use R in conjunction with the IEA IDB Analyzer
  • Understand the troubleshooting resources available for R users

Programming knowledge learned during this course aims to create a foundation for researchers in their use of R and R resources.

Prerequisites

Participants should have knowledge of basic descriptive and inferential analyses (e.g., frequencies, correlation, regression, significance testing) and some basic knowledge of large-scale assessment (e.g., weighting, plausible values) to understand the integration between R and the IEA IDB Analyzer.

About the Instructors:

Justin Wild and Diego Cortes both work at the IEA Research and Analysis Unit. They are part of the team who developed the R syntax for the IDB Analyzer software, and have multiple years of experience analyzing data from IEA studies with R.

Justin uses data from educational large-scale assessments to investigate issues of inequality and inequity. He is particularly interested in latent constructs and comparative analysis.

Diego focuses on different aspects of educational policy. Further, being responsible for sampling in ICCS, he is particularly interested in the design and methodology of educational surveys.

 
 
2. Multilevel modeling with IEA data (advanced)
Agnes Stancel-Piątak and Emilie Franck
Agnes Stancel-Piątak and Emilie Franck

Multilevel modeling (MLM) reflects the hierarchical structure of education systems and the clustered structure of the data providing insight into associations at different levels of the system. Usually, the results are less biased than results obtained with single-level models. This workshop is designed to introduce participants to the basic theory and application of MLM, focusing especially on those features that are particular to large-scale assessment data (such as weighting and scaling). Participants will gain basic practical experience of the application of two-level models to large-scale assessment data with the software MPlus.

The workshop begins with a methodological introduction to MLM and its underlying assumptions. Participants will receive practical training in how to use MPlus software for multilevel analysis of data from a selected IEA study, and learn more about methods for model selection and hypothesis testing. The course considers methodological concepts related to the complex study and sampling design of large-scale assessments in general, and the course instructors will provide comprehensive advice on selecting the most appropriate MLM approach for data analysis.

The following topics will be covered:

  • Methodological foundations of MLM;
  • Two-level random coefficient models with L1 and L2 predictors;
  • Short introduction to MPlus;
  • Calculating the compositional effect;
  • Centering approaches;
  • Application of MLM to large-scale assessment data (incorporating weighting and plausible values); and
  • Hypothesis testing and model selection.

The workshop will be a mixture of lectures and hands-on training, to ensure participants gain both sound knowledge and practical expertise. The models will be presented and workshop participants will then practice their implementation via a series of practical exercises using MPlus.

Prerequisites:

This workshop is aimed at individuals who already possess a working knowledge of large-scale assessment and a solid knowledge of intermediate statistics. Although no previous experience of MPlus is required, familiarity with syntax-based statistical software is beneficial. Participants must bring their own PC-compatible laptops with SPSS software (or similar alternative software that can be used for data preparation) preinstalled. A trial version on the MPlus software will be made available and used during the workshop.

The workshop addresses researchers who have a solid understanding of the structure and challenges of large-scale assessment data, elementary statistical methods, and the IDB Analyzer. Basic familiarity with SPSS is expected.

About the Instructors:

Agnes Stancel-Piątak is a Senior Research Analyst at the IEA Research and Analysis Unit. Her scientific work focuses on social justice, educational effectiveness, test construction, and complex methods of data analysis.

Emilie Franck is a PhD candidate within the EU founded project OCCAM (Outcomes and Causal Inference in International Comparative Assessments). She is based at IEA’s Research and Analysis Unit in Hamburg. Her main research interests relate to educational effectiveness, equity in education and equal educational opportunities.

 
 
3. Methods for causal inference with data from international assessments
Rolf Strietholt and Andrés Strello
Rolf Strietholt and Andrés Strello

Much comparative educational research poses questions about causal effects but cannot employ experimental techniques to investigate them. However, during the last couple of decades analytical techniques have been developed within statistics, econometrics, and other disciplines, which support causal inference from cross-sectional and longitudinal data. The workshop presents designs that incorporate so-called fixed effects for the identification of causal effect: trend analyses (e.g., TIMSS 2015, 2019), comparisons between educational stages (e.g., grade 4, grade 8), within-student-between-grade designs (e.g., math, science), and school fixed effects.

In this workshop, the Rubin causal model (RCM) will be used as a framework of potential outcomes to define causal effects. Key differences between experimental and observational designs will be discussed to explain issues such as selection into treatments, omitted variable bias, and reversed causality. Thereafter, we will introduce different methods that address these issues in a non-technical way. We will review some studies that have employed the data from international assessments such as PIRLS and TIMSS to illustrate the methods. In a hands-on session, we will use data from IEA-studies for practical and interactive activities. 

After the workshop, participants will be able to:

  • Understand what a causal effect is;
  • Be able to identify issues related to the identification of causal effects with observational data;
  • Understand limitations of regression analysis; and
  • Know designs that incorporate fixed effects for countries, schools, students, years, and educational stages.

The workshop will be a mixture of lectures and hands-on training, to ensure participants gain both sound knowledge and practical expertise.

Prerequisites:

Participants should have solid knowledge of regression analysis and be familiar with the use of syntax code in some statistical software such as R, SPSS, or Stata.

About the Instructors:

Rolf Strietholt is the Co-Head of the IEA Research and Analysis Unit. His main research interests include school and educational effectiveness research and research methodology.

Andrés Strello is a PhD candidate within the EU founded project OCCAM (Outcomes and Causal Inference in International Comparative Assessments). He is based at TU Dortmund University. Andrés is interested in comparative education, differentiation, heterogeneity, and inequality.

 
 
4. Statistical analysis of IEA data made easy: using the IEA IDB Analyzer (basic)
Yasin Afana (Language of instruction in Arabic)

Data from large-scale assessments are collected from complex samples and use specific methodology to examine the achievement of students. This is why base modules of statistical software packages cannot be used to derive correct statements on the state of education in the assessed populations. With the IDB Analyzer, IEA has developed an easy-to-use software that works as an add-on for SPSS, SAS and R. In this workshop we will work with the SPSS version, and will use TIMSS 2019 data for all presentations and practical examples. The goals of the workshop are to familiarize participants with the statistical complexities involved in working with international large-scale assessment databases, and familiarize participants with the data structure of TIMSS. Participants will learn how to run statistical analysis with TIMSS data, and interpret the results correctly. The workshop will balance presentations and hands-on practical assignments.

After the workshop, participants will be able to:

  • Understand the complexities of large-scale assessment data
  • Merge different data files
  • Combining variables, creating and validating indices
  • Reproduce results shown in the exhibits of the TIMSS 2019 International Report
  • Estimate percentages, means, correlation and regression coefficients and their standard errors given the complex designs

About the Instructor:

Yasin Afana works for more than two decades in the field of educational measurement and quantitative research, and about 16 years with IEA. Dr Afana has instructed multiple courses on the IEA data statistical analysis. He just recently obtained his professional doctorate degree in educational research from the University of Leicester in UK. His main research interest is in school value-added and improvement, as well as in research design and methodology.