Process data constitute a highly contemporary area of interest and research. An online workshop "Beyond results: Paving the way for the use of process data" took place in cooperation with IEA, the Leibniz Institute for Research and Information in Education (DIPF) and the Center for International Student Assessment (ZIB) from 17 to 19 June. Days one and two followed a workshop format; day three featured a hands-on training using data from international large-scale assessments, including TIMSS, and DIPF's Ulf Kröhne went on to deal with indicators and state machines (LogFSM).
The event covered issues related to concepts and constructs, their valid measurement as well as more technical presentations on data management, documentation, extraction and privacy of log data. Frank Goldhammer (DIPF) introduced challenges, gaps and the motivation for the workshop, framed through an evidence-centered design perspective. Matthias von Davier (Boston College) followed with early insights from TIMSS 2019 from the study center perspective. Samuel Greiff (University of Luxembourg), Steven L. Wise (nwea), Johannes Naumann (University of Wuppertal) and Frauke Kreuter (University of Maryland) all illustrated various aspects of solution behavior, guessing, disengagement and related test and survey-taking aspects, for which process and paradata can provide a window into respondents’ minds.
A sizable part of the workshop focused on the current state of play regarding data management by IEA's Heiko Sibberns. Qiwei He (ETS) dealt with the challenges in logging documentation, and Hendrik Drachslet (DIPF) shared some of the lessons learned around privacy.
There was a remarkable heterogeneity in all areas, underlining why it is necessary to establish a common language and ideally integrated approaches. Participants noted the variety of computer-based assessment (CBA) systems and the log data formats, event types and attributes generated by them. Documentation also heavily depends on systems used and can range from only crude information to sophisticated approaches that offer tools for extraction and aggregation, including for secure materials. Data protection must be taken into account comprehensively, a further aspect of the conference examined through a learning analytics perspective. Extraction tools are essential and five of these were presented between Denis Reis Costa (CEMO) Ronny Scherer (CEMO), Jiangang Hao (ETS), Seskia Kespaik (French Ministry of Education), Carolin Hahnel (DIPF/ZIB), and Ulf Kröhne. All differing greatly, most extraction tools have been created for specific studies or formats and, accordingly, are not easily transferable. The different procedures and objectives presented were enriching for all participants and commonalities and possible harmonization became visible throughout the sessions.
The workshop was rounded off with a closing keynote by Stephen Sireci (University of Massachusetts Amherst), which summarized and linked the aspects dealt with, connected it with advances in psychological research from more than a century ago and offered perspectives for future research and suggested subsequent conferences to keep the momentum. A large number of participants were in favor of continuing as a community and the possibility is currently being discussed between the workshop organizers and their respective organizations.
Originally planned as an on-site conference at DIPF in Frankfurt, the event was converted to a virtual format due to the travel restrictions. With some smaller changes to the agenda, the event clearly demonstrated that such a conference can be held in the virtual space, sustain active participation and involve many perspectives and close to 100 attendees. Smaller group work was facilitated through virtual break-out rooms with moderators and rapporteurs. And the event even included a social event, and online performance of Frankfurt’s "Take Love Easy” jazz duo.
We would like to thank everyone who contributed to the success of the workshop, without whose stellar dedication and commitment the event would not have been possible. We have posted live recordings of the Beyond Results sessions that took place in June online so that they are available for all interested parties. They can be found at the below link.