Pre-Conference Workshops

26–27 June 2017, Prague, Czech Republic

In conjunction with the IEA IRC-2017, the IEA organized four optional workshops in specialized topics related to large-scale assessment. These two-day workshops were offered in parallel (26–27 June 2017), as part of the pre-conference activities.

The aim of the workshops was to provide a stimulating and practical learning environment for all those wishing to improve their understanding of, and gain practice in, working with data from large-scale international assessments such as those conducted by IEA. Each workshop varied in its focus and level; the specific topics and prerequisites are described in more detail below. Lectures and discussions were conducted in English.


Workshop 1: Added value. How IEA study participation can meet specific national research interests

Dr Sabine Meinck

Participants will learn more about the complexities surrounding the statistical analysis of IEA data. They will discover how they can reproduce the statistics presented in the international report, and will then focus on conducting further in-depth analysis concentrating on their own national research interests. Using the IDB Analyzer, the instructor will demonstrate how to conduct basic inferential statistics, such as estimating population means, percentages, correlation and OLS regression coefficients, and statistical significance testing for dependent and independent samples.

Workshop participants will be introduced to the many ways that IEA study data can be used in national analyses, extending study participation to accommodate specific national research interests. Topics that will be covered include:

  • tailoring sample designs;
  • extending questionnaires;
  • regional modules;
  • possibilities and limits of using additional national context data for analysis;
  • benchmark participation; and
  • statistical analysis with the IDB analyzer

Lectures on methodology will lay solid foundations for the practical application of statistical analysis. After a comprehensive introduction to the IEA IDB Analyzer, its application will be demonstrated using practical example analyses. Participants will receive course materials with step-by-step guidance on how to perform the various statistical analyses. They will have time to conduct their own example analysis, and develop and answer personal research questions. The workshop will provide an inspiring forum for the exchange of ideas on nationally-focused analyses, while the instructors will provide guidance and advice throughout on further possibilities or potential limitations. The workshop will showcase successful examples of tailoring participation in IEA studies for national needs, demonstrate procedures for implementation, and indicate potential constraints for consideration (such as methodological and financial constraints). Participants will thus learn how to conduct in-depth analyses of IEA data, with a specific focus on the possibilities and constraints for national analysis.

Prerequisites:

Participants should possess a working knowledge of basic and inferential statistics, and will need to bring their own laptop PC with Microsoft Office and SPSS 16.0 or higher preinstalled.

About the instructor:

Dr Sabine Meinck brings a wealth of experience to this workshop, being the Head of both the IEA’s Research and Analysis Unit and its Sampling Unit. For the last decade, she has been involved with sampling, weighting and variance estimation for all the IEA's large-scale assessments. Her main research interest lies with the methodological challenges of complex survey data.


Workshop 2: Multilevel modeling with IEA data

Dr Agnes Stancel-Piątak and Dr Leslie Rutkowski

This workshop will introduce participants to the basic theory and application of multilevel modeling (MLM), focusing especially on those features that are particular to large-scale assessment data (such as weighting and scaling). Participants will learn how to specify, estimate and interpret results of two- and three-level models using MPlus, and to formulate and test hypotheses for research and policy.

The following topics will be covered:

  • methodological foundations of MLM;
  • short introduction to MPlus;
  • calculating the compositional effect;
  • centering approaches; and
  • application of MLM to large-scale assessment data (incorporating weighting and plausible values).

The workshop will cover the following models:

  • Two-level random intercepts and random coefficients models with L1 and L2 predictors for: (1) students nested in schools and (2) student nested in classes.
  • Three-level random intercepts and slopes models with predictors at L1, L2, and L3.

The models will be presented, and workshop participants will then practice their implementation in a series of practical exercises using MPlus.

 Prerequisites:

This workshop is aimed at individuals who already possess a working knowledge of large-scale assessment and a solid knowledge of intermediate statistics. Although no previous experience of MPlus is required, familiarity with syntax based software is beneficial. Participants must bring their own PC-compatible laptops with SPSS software (or similar alternative software that can be used for data preparation) preinstalled. A trial version on the MPlus software will be made available and used during the workshop. The workshop will be a mixture of lectures and hands-on training, to ensure participants gain both sound knowledge and practical expertise.

About the instructors:

Dr Leslie Rutkowski is Professor of Educational Measurement at the Centre for Educational Measurement at the University of Oslo. She earned her PhD in educational psychology, specializing in statistics and measurement, from the University of Illinois at Urbana-Champaign. Leslie’s research is in the area of international large-scale assessment. Her interests include latent variables and examining methods for comparing heterogeneous populations in international surveys. In addition to a recently funded FINNUT grant to develop international assessment methods, Leslie was one of the editors of the Handbook of International Large-Scale Assessment (Rutkowski, von Davier, & Rutkowski, 2014).

Dr Agnes Stancel-Piątak is Deputy Head of the IEA’s Research and Analysis Unit and Sampling Unit. For over a decade she has been involved in the development, implementation and analysis of research projects in education including several large-scale assessments. Her methodological expertise encompasses applications of multilevel structural equation modeling, item response theory based modeling, and complex designs. Agnes’ research focuses on school effectiveness, teaching and learning, and social justice.


Workshop 3: Structural equation modeling using IEA data

Dr Deana Desa

This workshop is intended to provide a comprehensive overview of the basic theory and application of structural equation modeling (SEM) within the framework of IEA studies. Amongst other things, the workshop will cover how to use SEM models with sampling weights and plausible values, emphasizing the fundamentals of SEM and its underlying assumptions.

Topics covered during the workshop will include:

  • data preparation for SEM analysis;
  • latent variables and confirmatory factor analysis;
  • multivariate regression; and
  • multiple-group comparisons.

Participants will use MPlus or R software to perform SEM modeling, and focus on the interpretation of SEM models that are prepared for the analysis of complex data with latent variables. This workshop is ideal for students, researchers, or anyone else who wants to apply SEM in their analyses of IEA data.

The workshop will comprise of a mix of lectures, demonstrations, practical exercises and open discussion.

Prerequisites:

Participants should be familiar with basic statistical analyses, such as correlation, simple regression and analysis of variance, and knowledgeable about the features of IEA studies. Prior knowledge of SEM and the modeling software is not required. Participants will receive a set of presentations, sample data, and example SEM analyses. It would be helpful if participants could bring their own laptop; the operating system must be Windows 7 or a later version.

About the instructor:

Dr Deana Desa is a Research Analyst at the IEA, where she is involved in psychometric research and development of scaling complex data for IEA studies.


Workshop 4: Bayesian statistics with applications to IEA data

Dr David Kaplan

Bayesian statistics has long been overlooked in the quantitative methods training for education.  Typically, the only introduction that a student might have to Bayesian ideas is a brief overview of Bayes' theorem while studying probability in an introductory statistics class.  This is not surprising.  First, until recently, it was not feasible to conduct statistical modeling from a Bayesian perspective because of its complexity and lack of available software.  Second, Bayesian statistics represents a powerful alternative to frequentist (classical) statistics, and is therefore controversial. Recently, however, there has been great interest in the application of Bayesian statistical methods, mostly due to the availability of powerful (and free) statistical software tools that now make it possible to estimate simple or complex models from a Bayesian perspective.

This pre-conference workshop introduces practicing education scientists to the basic elements of Bayesian statistics and shows why the Bayesian perspective provides a powerful alternative to the frequentist perspective.  It is assumed that participants will have a background in basic statistical methods up to, and including, regression analysis. 

Topics to be covered include:

Day 1

  • The major differences between the Bayesian and frequentist paradigms of statistics, with particular focus on how uncertainty is characterized;
  • Bayes’ theorem;
  • Bayesian model building and model evaluation; and
  • Bayesian computation.

Day 2

  • Introduction to “rjags”;
  • An example and practice using data from PIRLS; and
  • Wrap-up: Relative advantages of the Bayesian perspective.

Suggested reading

van de Schoot, R., Kaplan, D., Denissen, J., Asndorpf, J. B., Neyer, F. J. & van Aken, M. A. G. (2013). A Gentle Introduction to Bayesian Analysis: Applications to Developmental Research. Child Development. DOI: 10.1111/cdev.12169 PDF

Kaplan, D. (2014). Bayesian Statistics for the Social Sciences. New York, USA: Guilford Press. (Order online)

Kaplan, D. & Depaoli, S. (2013). Bayesian statistical methods. In T. D. Little (ed.), Oxford Handbook of Quantitative Methods. (pp 407-437). Oxford, UK: Oxford University Press. PDF

About the instructor:

Dr David Kaplan is the Patricia Busk Professor of Quantitative Methods at the University of Wisconsin–Madison and an Honorary Research Fellow in the Department of Education at the University of Oxford. He is an elected member of the National Academy of Education and recipient of the Humboldt Research Award. Dr Kaplan’s methodological research focuses on Bayesian model averaging; objective versus subjective Bayesian modeling; and Bayesian approaches to problems in large-scale survey methodology. Dr Kaplan is actively involved in PISA, where he served on its Technical Advisory Group from 2005-2009 and its Questionnaire Expert Group from 2004-present; he served as the Chair of the Questionnaire Expert Group for PISA 2015 and remains a member of the Questionnaire Expert Group for PISA 2018. Dr Kaplan also sits on the Questionnaire Expert Group for the OECD's TALIS, and the Design and Analysis Committee and the Questionnaire Standing Committee for the National Assessment of Educational Progress (NAEP).

The IEA conducts regular workshops at international research conferences and in collaboration with the IERI Institute (www.ierinstitute.org). For more information, consult our training page.