After the success of the first Beyond Results workshop held in 2020, a second edition was hosted from 30 September - 01 October 2021. The series is a collaboration between IEA and colleagues from the Leibniz Institute for Research and Information in Education (DIPF) and the Center for International Student Assessment (ZIB).
A sharper focus
This year's focus was on the theory-based construction of process indicators. The workshop discussed conceptual perspectives and empirical research on how to construct process indicators that are, in some way, anchored in theory. Topics included the design of cognitive assessments providing log data and strategies to validate the interpretations of process indicators.
International large-scale assessments (ILSA) have started to make log data available to the public by means of public-use files. These files contain events and specific event-related attributes. All events are stored together with a corresponding timestamp. Events reflect the time-takers interactions with the assessment system and internal changes in the platform's behavior. Log data promises to provide new insights into the work process and underlying cognitive and mega-cognitive, motivational, and affective constructs. To realize these promises, the link between construct, empirical evidence, and task situation must be clarified, as required by the Evidence-Centered Design (ECD) framework (one framework among others). To use log events in a meaningful way, a coherent reasoning process from raw log data to low-level features, process indicators, and finally, high-level (construct) interpretations are needed.
This reasoning from evidence may include data-driven and theory-driven elements to varying degrees. The data-driven construction of process indicators typically relies on data mining techniques to explore patterns that can be theoretically interpreted in terms of an attribute of the work process. The theory-driven construction of process indicators is already part of the assessment framework and is based on theoretical assumptions about low-level features, like actions and states, which represent the test-taker's information processing. The theory-driven approach also makes assumptions on how these low-level events can be aggregated. In any case, the intended uses and interpretations of process indicators require validation; that is, theoretical and/or empirical justification that the theoretical attributes of the work processes are valid.
Leading experts welcomed
Leading experts and researchers convened at the 2021 edition of the invitational workshop series as both attendees and contributors. Susan Embretson and Robert Mislevy delivered insightful keynote lectures, Matthias von Davier and Stephen Sirieci engaged in a lively, scientific debate moderated by Saskia Wools. Discussions took place about the consequences for construct validation and contrast irrelevant variance when considering data-driven approaches and theory-driven approaches. Workshop lectures considered topics from the perspective of various national and international studies, in which process data were collected in addition to response data. These lectures dealt with topics as diverse as log data and diagnostic processes, process indicators in ILSAs, the future development of process indicators, and process indicators and learning. Overall, a wide range of aspects on the validity of process indicators was presented and discussed over the two days by leading experts in the field.
The Beyond Results workshop series offers a format in which the latest research on the collection, processing analysis, and interpretation of process data can be presented and discussed yearly. IEA, DIPF, and ZIB are looking forward to hosting the following workshop together in 2022. All presentation abstracts and speaker biographies can be found on the Beyond Results website.