In advance of the IEA International Research Conference (IEA IRC), IEA offers two-day, on-site workshops covering specialized topics related to large-scale assessment from 26–27 June 2023.
The workshops will provide a stimulating and practical learning environment for all those who wish to improve their understanding of and gain practice in working with data from large-scale international assessments such as those conducted by IEA. Each workshop varies in its focus and level; the specific topic of each workshop is described in more detail below.
Registration is now open via Conftool. The price of each two-day workshop is €150. Only one workshop can be chosen per delegate, since they are happening at the same time.
Following the initiative from our partner research associations WERA and EERA, IEA is waiving the participation fees for all Ukrainian researchers who wish to attend the IEA IRC 2023. Please contact irc@iea.nl to find out more.
The registration deadline for presenters and all participants has passed, and registration is now closed.
Over a 60-year history, modern ILSAs (international large-scale assessments) have become influential educational policy tools, moving beyond their historical role as descriptive “snap shots” of educational systems. As the number of ILSAs increase in both number of participants and subjects assessed, policymakers often ask how the resulting information can help them inform the policy process. One important way to meet such requests is to present policy-relevant information resulting from ILSAs in a brief and accessible format. This workshop will discuss the utility and limitations of ILSAs for informing policymakers and education practitioners. We will provide an overview of how policy briefs can be structured along with illustrative examples, specifically using briefs published under IEA’s Compass Briefs Series. Participants will be encouraged to start preparing their own policy brief and should leave the workshop with a well-developed outline that they can later develop into a publishable document that may be included in an IEA Compass Brief.
Contents
The workshop will focus on the studies PIRLS 2021 and REDS, and emphasize the following key topics:
- Challenges and solutions in the construction and development of educational briefs that utilize ILSA data (defining and addressing the audience, structure and length, tables and visual displays)
- General information about PIRLS and REDS: goals, purposes and intent, theoretical frameworks, target populations, achievement domains and background information collected
- Introduction on how to find, use, and interpret data from the studies
- Interpretation and discussion of findings, using terminology and wording that is accessible to the defined audience
Methods
Lectures and group work will alternate. Time will be provided for participants to discuss possible topics and outlines for briefs that would be relevant for their specific context. Participants will also get the opportunity to work with IEA data, either referring to readily available statistics or—for those familiar with it—using simple statistical analysis tools such IEA’s IDB Analyzer, for which aid by instructors can be provided. Participants will inspire each other by exchanging ideas on nationally focused analysis while being advised by instructors on possibilities and limitations.
Prerequisites and targeted audience
The workshop is suitable for those interested in informing policy and broader audiences on results retrieved from ILSA. While no specific requirements are needed; a working knowledge in basic statistics would be an asset.
About the instructors
David Rutkowski is a Professor of Research Methods at Indiana University (IU), and also worked as a researcher for IEA in Hamburg, Germany. David’s research focuses on educational measurement and policy. He has collaborated with or consulted for national and international organizations including the US State Department, USAID, UNESCO, World Bank, IEA and the OECD. He is currently the editor of the IEA policy brief series, co-editor of the journal Discourse, serves on the IEA publication editorial committee and is a board member of several academic journals. He also co-leads a project on improving assessment literacy among teachers.
Dr. Sabine Meinck is co-heading IEA’s Research and Analysis Unit and heading the Sampling Unit. She is responsible for the coordination and conduct of research and sampling-related activities within IEA. Apart from her research activities, one of her main interests relates to the dissemination of results of IEA studies to policy and practice.
In order to conduct targeted research using existing education data from international large-scale assessments (ILSA), researchers must have a good knowledge of statistical software in order to perform preliminary analyses, subset data to the target population of interest, and finally carry out statistical analysis, taking the specifics of ILSA data into account. A number of such software applications exist, but many are commercial while still requiring a steep learning curve for basic functionality. A free software application growing in use among the research and academic community is called R and has an active user base willing to discuss and exchange ideas, as well as offer solutions to basic and difficult programming questions. It has been integrated with the IEA IDB Analyzer, which is a freely available software for ILSA data analysis.
R is one of the most important tools for statisticians presently used. It is an open-source project software environment and programming language with a wide variety of options that is highly extensible. It is not only used and supported by a broad and very diverse research community, but also by companies like Microsoft, Google, RStudio, and Data Camp that are actively promoting R as a lead platform.
This hands-on workshop has two goals. Firstly, it is targeted to individuals who want to advance their data science skills using R and become familiar with R Studio – a software that makes R user-friendly. It will introduce participants on how to get their data into R, structure and transform it, analyze it, visualize the results, using the Tidyverse tools of data exploration. Secondly, participants will learn about the specifics of analyzing ILSA data, such as using plausible values and weights. We will use PIRLS 2021 data for our analysis examples. Participants will then be introduced to the IEA IDB Analyzer integration with R to perform analyses with large-scale assessment data. Also, we will introduce participants the R Markdown to turn their statistical results and graphs into documents, reports, presentations, and dashboards.
After the workshop, participants will be able to:
- Understand the use of R and R Studio
- Import data into R
- Perform subsetting, filtering, and recoding with data
- Use the R package Tidyverse to explore and visualize data
- Use functions to perform descriptive and inferential analyses
- Understand how to account for the specifics of ILSA data in statistical analysis
- Use R in conjunction with the IEA IDB Analyzer for research questions
- Compile the results and graphs to a report using R markdown for, e.g., writing a journal article
Programming knowledge learned during this course aims to create a foundation for researchers in their use of R and R resources.
Prerequisites
Participants should have basic knowledge of R as well as descriptive and inferential analyses (e.g., frequencies, correlation, regression, significance testing) to understand the integration between R and the IEA IDB Analyzer.
About the instructors
Umut Atasever works at the IEA Sampling Unit with major responsibilities in TIMSS, ICILS and ICCS. He focuses on different aspects of sampling methodology with studies on the validity of international assessments.
Diego Cortes is a research analysist at IEA Hamburg with experience in survey methodology and data analysis in the context of international large-scale assessments in the field of education. He specializes in generating probabilistic samples with a complex design and examining how survey-design features impact the inference one can reach about population parameters.
Much comparative educational research poses questions about causal effects, but cannot employ experimental techniques to investigate them. However, during the last couple of decades analytical techniques have been developed within statistics, econometrics, and other disciplines, which support causal inference from cross-sectional and longitudinal data. The workshop presents so-called quasi-experimental designs in the context of educational research for the identification of causal effects: instrumental variables, regression discontinuity, difference-in-differences, and fixed effects. These include approaches such as exploiting random variation due to class size or age of entry policies in schools, country-level longitudinal analyses (e.g., TIMSS 2011, 2015, 2019), comparisons between educational stages (e.g., grade 4, grade 8), within-student-between-grade designs (e.g., math, science), among other strategies.
In this workshop, causal effects will be defined using the potential outcomes framework from the Rubin causal model (RCM). Key differences between experimental and observational designs will be discussed to explain issues such as selection into treatments, omitted variable bias, and reverse causality. Thereafter, we will introduce different methods that address these issues in a non-technical way. We will review some studies that have employed the data from international assessments such as PIRLS and TIMSS to illustrate the methods. These studies will be discussed in groups, highlighting the authors’ identification strategy as well as discussing advantages and shortcomings of each method.
After the workshop, participants will be able to:
- Understand what a causal effect is;
- Be able to identify issues related to the identification of causal effects with observational data;
- Understand limitations of regression analysis;
- Know designs and strategies to identify effects with observational data from large-scale assessments.
Methods
The workshop will be a mixture of lectures, group discussions, and hands-on examples, to ensure participants gain both theoretical background and real-life examples applicable to educational research.
Prerequisites
Participants should have solid knowledge of regression analysis. Being familiar with the use of syntax code in some statistical software such as R or Stata is recommended.
About the instructors
Alec Kennedy is a researcher at the Research and Analysis Unit of the IEA. His interests are in education policy and quantitative research methodology.
Andrés Strello is a researcher at the Research and Analysis Unit of the IEA. He is interested in educational inequality, sociology of education, and comparative analyses.
Rolf Strietholt is the co-lead of the Research and Analysis Unit of the IEA. His main research interests include school and educational effectiveness research and research methodology.