IEA's considerable experience in the implementation of international large-scale studies and surveys of education means that we are able to offer tailored services, characterized by the highest standards of scientific research.
Our services encompass all phases of educational research projects. We can work with you on just one, a few, or all stages of your project, including study initiation, implementation, quality control, data analysis, and the dissemination of findings.
IEA's extensive experience in designing international, national, and regional cross-sectional studies, as well as cyclical studies directed toward collecting trend data across time, and longitudinal studies, means that we are well qualified to offer such services to others. Committed to building strong partnerships, we will work in close cooperation with you and any other stakeholders involved in the study you want to implement.
We are skilled in developing and implementing studies that have complex survey designs, and valued for our effective and efficient study frameworks, designed to address key research questions without compromising their intent and integrity. We have extensive experience with e-assessment, and students and other study respondents or participants may be offered the option of online assessments or questionnaires.
Our study design and management services include:
- Development of the procedures needed to conduct effective and efficient studies, particularly international studies, longitudinal studies, and multi-cohort designs;
- Time and resource management, tailored to your requirements;
- Coordination of an entire survey cycle using a study design and technical standards aimed at yielding high-quality data;
- Collaboration and communication with national project managers in the countries participating in one or more studies, with an emphasis on providing individual country support.
One of the most critical tasks during the early stages of study development is determination of sample design. The decisions made will determine the quality of study outcomes. IEA sampling experts regularly develop sample designs and select samples for international, national, and regional educational studies. Strict adherence to state-of-the-art methods in sampling provides a foundation for unbiased and precise population estimates.
During the last decade, we have been involved in selecting more than 300 samples for over 50 studies focused on differing educational contexts. In close collaboration with our clients, we propose and implement sampling procedures tailored to study objectives and target populations, while carefully balancing qualitative requirements and financial limitations.
Sampling services include, as a bundle or separately:
- Development of the sampling design;
- Determination of sampling size;
- Processing of sampling frames;
- Implementation and documentation of sampling.
Test sessions have to be administered in a standardized way across the participating countries, schools, or other educational institutions involved. Study manuals ensure that all involved are well informed about the correct methods and protocols.
We ensure that study manuals include sound information and advice on, among other matters, sampling, preparation of instruments, field operations, data collection, and data cleaning and verification.
Our services relating to designing and developing manuals and guidelines encompass:
- Manuals for the agents involved in international studies;
- Manuals for the coordinators within institutions;
- Handbooks and training papers for test administrators.
IEA has more than 15 years of experience in the development of software for educational research. Our software supports study administration at the organizer and participant level, accelerates data collection, and enables online-based tests and surveys. Software-user manuals can also be provided.
Our software development services include creating, adapting, and programming software for:
- Computer-based testing and surveying (including major international studies such as TALIS);
- Study preparation and administration;
- Data management;
- Data analysis;
- Design, translation, and adaptation of online questionnaires;
Collecting high-quality data requires high-quality instruments so the survey instruments that are selected are vital to a study's success. Instruments need to be free of errors, suitable in terms of language and layout for the intended respondents/participants, able to capture the required information, and correctly administered. IEA can help you develop and adapt test instruments and questionnaires for use in international, national, and regional surveys.
Development and Adaptation
IEA has developed a standardized process that provides quality control of instrument development and adaptation. The process also includes mechanisms governing instrument release, to ensure that respondents and participants receive the right instrument at the right time. This assurance is especially important in studies where instruments are administered according to a rotation design.
IEA can advise on developing study instruments for both paper (print)-based and computer-based assessment. Our considerable experience in implementing studies using both quantitative and/or qualitative methods means we can also help you to tailor your instruments appropriately.
Our services relating to developing and adapting study instruments include:
For paper-based assessments
- Editing the text and illustrative content of the study instruments, particularly in terms of checking instructions, filtering, spelling, and layout.
For computer-based assessments
- Checking all applications, particularly with regard to instructions, filtering, spelling, and accessibility;
- Programming of online questionnaires;
- Hosting the study instruments on IEA servers.
Careful translation of instrument items into the different languages within and across countries participating in a large-scale assessment study ensures the reliability of the instrument and the validity of the information it collects. Over the years, we have developed robust procedures for verifying the accuracy of translations and ensuring that each translated instrument collects data that can be compared across contexts and countries, despite their cultural and linguistic differences.
Our services relating to translation verification include:
- Overseeing the translation of instruments into different languages;
- Ensuring instrument content is sensitive to cultural differences;
- Verifying the accuracy, reliability, validity, and cross-context or cross-national comparability of each translated instrument.
Recruiting and retaining study participants on an institutional or individual level is a particular challenge for large-scale assessment studies. For many years, IEA has successfully recruited schools, students, and teachers in both general and vocational schools to participate in such studies. We can assist you with strategies to target and recruit study participants and ensure their sustained participation.
Our study recruitment services include:
- Securing the participation of individuals and institutions;
- Coordinating and managing longitudinal studies using our 'Status and contact management' system.
Test Administration (Only in Germany)
Our nationwide pool of more than 1000 experienced test administrators in Germany means we can easily conduct large-scale studies within the German national context.
We provide all test administrators with intensive training before their first deployment. This training focuses on the procedures used to administer a test. It also covers communication with study participants.
We also provide written manuals for all administrators. These include detailed descriptions of the procedures administrators must follow to ensure that test conditions are comparable across all organizations and individuals participating in the study. Establishing a uniform approach to test implementation is crucial to guarantee the quality of the study results.
Our test administration services will ensure consistency. We administer many forms of testing, including:
- Printed test materials and questionnaires;
- Computer-based tests and questionnaires, delivered online or offline;
- Group or individual testing;
- Tests for a wide range of age groups, from kindergarten children to adults;
- Independent surveys, conducted by direct mail or online.
Additional Services offered for Computer-based Assessments (Only in Germany)
IEA provides support on necessary technical equipment for computer-based assessments, this includes:
- Procurement (purchase or rent) of laptops/tablets;
- Configuration and preparation of the devices according to each study's specification, including functional tests;
- Just-in-time delivery of devices into the schools on the test days;
- Retrieval of study data from the devices after the study.
International large-scale assessments require standardized materials and operational procedures to ensure comparable data across the participating countries and systems. Quality control programs need to be established both nationally and internationally.
IEA has extensive experience in designing, implementing, and supporting international quality control processes for large-scale comparative studies in education. In cooperation with participating national centers, IEA recruits and trains 'international quality observers' from participating countries to monitor and document all data-collection activities. These observers pay particular attention to test and questionnaire administration. They also observe overall compliance with operational procedures.
Our quality control procedures ensure uniformity when administering tests, and include:
- Helping countries develop their national quality control programs during field trials and main data collection;
- Providing guidelines, manuals, instruments, and training directed toward standardized approaches to observing and ensuring the quality of data-collection processes.
Data Entry and Processing
IEA offers different options for data capture. We can tailor these methods to suit your research design and specific project needs. Manual, scanner-assisted, or combined methods are possible. We can provide advice on procedures to control the quality of data-entry and data-processing, ensuring close connections across the different steps of data collection in the field, data entry, and data processing.
Our services relating to data entry and processing include:
- Capturing data using scanner-assisted or manual methods, and transcription of open-ended replies;
- Checking the consistency and manually verifying implausible data;
- Undertaking formal and content-related validity checks of datasets;
- Compiling Stata-, SPSS-, or SAS-datasets, including full supporting documentation;
- Compiling code books;
- Digital archiving;
- Documentation of data-processing activities.
Survey instruments sometimes need to include open-response items that require respondents to write down an answer. To conduct a quantitative analysis of such open-responses, the information these answers contain must be transferred into numerical data. This process is known as coding or (sometimes) scoring.
IEA's 'CodingExpert' software facilitates accurate coding of data, offering a very high level of flexibility. Data may be collected in many formats: as scans, electronic responses, or as hard copy printouts. The software includes integrated control mechanisms that guarantee high-quality coding. We can also provide intensive training for coders and stringent assessment of their coding performance on a regular basis.
Our coding services include:
- Developing coding instructions and ensuring that all language versions of these instructions are clearly written and comparable in meaning;
- Coding open-ended responses;
- Coding job occupations according to international standards such as ISCO 08, ISCO 88, ISIC, ISCED, or KLDB, and assigning related social status indices (EGP, ISEI);
- Providing support during implementation of studies requiring electronic coding of data with 'CodingExpert.'
Data collected in international large-scale assessments of educational achievement originate from complex unit- and item-sample designs. The hierarchical structure of education systems, and the complex structure of educational processes require sound knowledge of statistical methods suitable for analyzing and interpreting the data obtained from these assessments. The international context of international comparative research means that cross-country comparability must be established to generate reliable results.
Our data analysis services include:
- Scaling of questionnaire and achievement test data, using, for example, Rasch or IRT scaling;
- Evaluating test consistency, the psychometric quality of the test items, and measurement invariance;
- Weighting of data, including nonresponse adjustment and scaling of weights;
- Variance estimation, using, for example, jackknife or balanced repeated replication;
- Univariate and multivariate analysis, including multilevel analysis and structural equation modeling;
- Development of simple descriptive and inferential statistics, for example, estimation of means, percentiles, benchmarks, correlation and regression coefficients, and the standard errors of these estimators.
IEA has a wealth of experience in interpreting and reporting the results of its large-scale studies for diverse audiences, such as researchers, practitioners, policymakers, and the media. Careful consideration not only of the types of information most useful to these various groups but also of the best way of presenting that information to each one ensures effective dissemination and use of study findings.
Our services relating to reporting and disseminating study findings include:
- Identifying the audiences interested in the study findings;
- Identifying which types of information each of these audiences is most likely to be interested in;
- Producing publications, both print and electronic, and ranging from press releases to full reports.
Printing and Distribution
Over the years, we have acquired expertise in printing and transporting materials, both nationally and internationally, for use in our studies. Combining this know-how with our own in-house printer means we are able to provide the following services:
For paper-based assessments
- Preparation of print templates and printing of study instruments (black/white and colored, digital and offset printing);
- Packing and dispatch of a diverse range of additional supporting survey materials (such as stopwatches, CDs, posters, and MP3 players).
For computer-based assessments
- Safe dispatch of test laptops and tablet PCs, via courier service.
Any survey conducted in a school context has to be approved by one or more educational authorities. These authorities must also be assured that the study will include strict data-protection procedures. Within Germany, IEA has developed long-term contacts with the ministries of education and the data-protection commissioners of all 16 federal states. This cooperation has resulted in the joint development (with these agencies) and effective use of study approval and data-protection procedures.
Our services relating to study approval and data protection include:
- Preparing the documents needed for the approval process;
- Communicating with the relevant authorities;
- Finding solutions for any conflicts between research requirements and data-protection requirements.