Pre-Conference Workshops

24–25 June 2013, Singapore

In conjunction with the IRC-2013, the IEA Data Processing and Research Center (DPC) will conduct a series of free training workshops in specialized topics related to large-scale assessment. These workshops will be offered in parallel over two days (24–25 June 2013), as part of the pre-conference activities.

The aim of the workshops is to provide a supportive and practical learning environment for all those wishing to improve their understanding of, and gain practice in, working with data from large-scale international assessments such as those conducted by IEA. Each workshop varies in its focus and level; the specific topics and prerequisites are described in more detail below.

Attendance to the workshops is limited. Participants must register by 15 May 2013 via the online registration system to secure a spot (registration is on a first-come, first-served basis).

Plamen Mirazchiyski and Andrés Sandoval-Hernández

This workshop will provide an overview of the IEA databases currently available, with particular emphasis on the most recently released databases (TIMSS 2011, PIRLS 2011, ICCS 2009, and TEDS-M). As part of the workshop, participants will also be instructed on how to use the IDB Analyzer. The IDB Analyzer is an application developed by the DPC to facilitate access to and analysis of the large-scale assessment databases available from IEA. The IDB Analyzer creates SPSS code that can be used with SPSS to conduct statistical analyses, taking into account the complex sample structure of the databases.

The following topics will be covered during the workshop:

• overview of IEA databases: structure of the databases, accessing the data, reviewing the documentation
• sampling and test design—implications for analysis: general sample and test design, computing sampling and measurement variance, estimating statistical significance of results
• using the IDB Analyzer to combine datasets: merging data from students, schools, teachers, and parents; combining data from different countries; modifying and recoding data for analysis
• using the IDB Analyzer to conduct analysis: calculating means, standard deviations, and percentages; calculating correlations and regression coefficients; calculating percentages of students meeting achievement benchmarks; calculating differences between groups.

This workshop is aimed at individuals who have only limited or superficial familiarity with IEA databases and analysis procedures, and who want to become familiar with them. Participants are expected to bring their own laptop PC with Windows and SPSS installed (Windows and SPSS are required to use the IDB Analyzer). Free copies of the IDB Analyzer will be distributed at the workshop. Attendance is limited to 30 participants.

Leslie Rutkowski

This workshop will introduce participants to the basic theory and application of multilevel or hierarchical linear modeling (HLM). In the context of cross-country, school, and student achievement data, the rationale for HLM analysis will be introduced. Participants will learn how to prepare datasets and import them to the HLM 7 software (Raudenbush & Bryk, 2011). Using PIRLS 2011 data, participants will also formulate relevant hypotheses for policy research regarding the relationship of socioeconomic and immigrant status with reading achievement. Ample opportunity will be provided to work through several examples. Model specification and interpretation will be a central focus of this workshop.

An example of the type of analysis that will be presented is testing for the presence of an immigration effect. This test will evaluate whether there is a significant relationship between student immigrant status and academic performance within a two-level model (students in schools in a specific country). Various model specifications that consider some of the complexities associated with immigrant students will be considered. For example, we will differentiate between student first- and second-generation immigrant status. Further, we will examine the moderating effect of socioeconomic status on immigrant status and achievement. Fundamental statistics will be interpreted (i.e., slopes, variance accounted for, variance components), and the policy implications of each hypothesis test and resultant statistics will be discussed.

Next, increasingly complex hypotheses will be evaluated. For example, participants will test whether school-level contexts are related to achievement and whether student-level relationships are moderated by the school context. Finally, participants will extend the analysis to a three-level model framework, and determine whether the effect of immigrant status differs across education systems and how.

The following topics will be covered:

• theoretical background of multilevel models
• data preparation and importing data to HLM
• model specification, hypothesis testing, and interpretation of results
• two- and three-level HLM analysis.

Participants will learn to specify, estimate, and interpret results of two- and three-level models within the HLM software environment, as well as to formulate and test hypotheses with implications for research and policy. This workshop is aimed at individuals with a working knowledge of IEA databases and a solid knowledge of intermediate statistics. Participants will be required to bring their own PC-compatible laptops with SPSS installed. The HLM student version and PIRLS 2011 data will be made available and used during the workshop. The workshop will comprise lectures mixed with hands-on training. Attendance is limited to 25 participants.

Eugenio Gonzalez

This workshop will provide an overview of the principles surrounding the design of large-scale assessments, the item response theory models used to calibrate items, and the methodology used to assign proficiency estimates, also known as plausible values. Presentations will be mostly theoretical, with ample time given for discussion. Time at the end of the workshop will be spent running example analyses with software used in IEA studies.

The following topics will be covered during the workshop:

• overview of assessment design principles: advantages and disadvantages of current designs of large-scale assessments, consequences for analysis
• overview of principles of item response theory: advantages and disadvantages of different models, limitations of IRT models, using Parscale to compute item parameters and proficiency estimates
• overview of principles of population modeling and proficiency estimation: review of procedures and techniques for carrying out conditioning, multiple imputations in large-scale assessments, plausible values and why they are useful, using DESI to compute proficiency estimates and plausible values.

The general daily schedule will consist of presentations and limited hands-on practical assignments. Participants are expected to bring their own laptop PC with statistical software installed (SAS or SPSS). The trial version of the software used will be distributed at the workshop. Attendance is limited to 25 participants.

Sabine Meinck

The workshop introduces the sampling methodology applied in large-scale educational surveys, its background, and its significance. The concepts of cluster sampling, stratification, and bias and precision are defined and illustrated. The selection process is illustrated by means of examples. The necessity of using weights when analyzing data from complex sampling designs is demonstrated and the calculation of weights is explained. Other implications for data analysis, particularly the need for specific methods of variance estimation, are also examined. The workshop comprises lectures and hands-on trainings for the various subjects.

The following topics will be covered during the workshop:

• introduction to the concepts of statistical sampling, characteristics of complex samples, implications for analysis and interpretation
• sampling theory and underlying concepts: sampling methods, defining the target population, defining sample size, stratification, clustering, calculating coverage and participation rates, replacement of units
• computing sampling weights: calculating probabilities of selection, weighting components, post-stratification, rescaling of weights
• variance estimation in simple and complex samples: definition, working with simple random samples, calculating replicate weights, clustering effects and intra-class correlations, effect of stratification on sampling variance, design effect
• jackknife (JK1 and JK2) and balanced repeated replication techniques in large-sale assessments
• calculating statistical significance in large-scale assessments: evaluating differences between independent or dependent samples, selecting directionality and significance level, dummy coding and regression techniques.

The general daily schedule will consist of presentations and limited hands-on practical assignments. Participants are expected to bring their own laptop PC with statistical software installed (SAS or SPSS). Attendance is limited to 25 participants.