INTERNATIONAL LARGE-SCALE ASSESSMENTS IN EDUCATION:
A BRIEF GUIDE
COMPASS
BRIEFS IN EDUCATION
NUMBER 10 SEPTEMBER 2020
SUMMARY
Internaonal large-scale assessments (ILSAs) are one
of the most important tools policymakers and other
educaonal stakeholders have to inform evidence-
based decision making for educaonal reform. Despite
this, and the widespread use of ILSA data, results are
somemes misunderstood or misinterpreted. Here, we
oer a brief guide to ILSAs and illuminate some of the
important dierences and commonalies within and
across studies, limitaons, and why they remain one
of our most signicant tools for educaon evaluaon
and reform. We focus on and compare the key studies,
approaches, and structure of our own organizaon,
the Internaonal Associaon for the Evaluaon of
Educaonal Achievement (IEA), with other ILSAs.
IMPLICATIONS
IEA conducted the rst ILSA study in 1958 but the mandates
for ILSAs have changed over their history and new mandates
are constantly developing. Most recently, focus has shied
to educaonal outcomes rather than inputs. ILSA results
should be understood in the context of their changing remit.
There are substanal dierences between organizaons
and approaches to assessments. From the review process to
decision-making and fees, fundamental dierences should
be accounted for when understanding results.
Key ILSAs do share a common methodology marked by high-
quality standards carefully dened to achieve each step in
the ILSA process and, further, the data are accompanied by
detailed supporng technical reports to assist in interpreng
and reporng results.
Limitaons of ILSAs need to be acknowledged when
understanding and reporng results. In parcular, care must
be taken when considering results from ILSAs as a holisc
quality measure of the educaon system.
Despite limitaons, the assessments are unique, monitoring
systems over me within a robust internaonal framework,
being largely independent of any single polical system,
and with data freely available to the public. When properly
understood and analyzed, the data from ILSAs provide
valuable opportunies to help inform policy decisions and
research.
Internaonal Associaon for the Evaluaon of Educaonal
Achievement (IEA), Amsterdam.
Website: www.iea.nl
Follow us:
@iea_educaon
IEAResearchInEducaon
IEA
2
The most well-known ILSAs are the core studies of the Internaonal Associaon for the Evaluaon of Educaonal
Achievement (IEA) and the Organisaon for Economic Co-operaon and Development (OECD):
INTRODUCTION
Internaonal large-scale assessments (ILSAs) of educaon
are empirical studies that assess educaonal abilies around
the world. The data are used in various ways to help inform
policymakers, educaonal researchers, and the general public.
However, despite the widespread use of ILSA data, how to
interpret and report results is oen misunderstood. Dierent
study results are somemes reported or, interpreted to mean
the same thing, yet there exists important dierences that need
to be accounted for when using results. As leaders in the eld of
internaonal large-scale assessment, our intenon for this brief
is to provide context for a beer understanding of ILSA results
and; how they should be interpreted and reported—we discuss
what ILSAs are, the history of their development, dierences
as well as commonalies in approaches, organizaon, and
methodology, and important limitaons—and to express why we
believe that ILSAs are unique, important, and relevant tools for
understanding educaonal systems and student achievement
around the world, and informing evidence-based change.
WHAT ARE ILSAs?
ILSAs assess student achievement in specic disciplines and
provide context for the results by collecng addional data
at the student level. Further contextual details at the teacher,
principal, and/or system levels may also be collected. To
provide stascally valid results, a representave sample of
schools (usually around 150 to 200 schools) are drawn from
each parcipang country or educaon system, and a group of
students are randomly drawn from within each of the sampled
schools, either by sampling enre classrooms or by sampling
students across classrooms.
BRIEF HISTORY
The rst ILSA, IEA’s Pilot Twelve-Country Study (Foshay et al.
1962), was launched by a group of researchers in 1958 (Husén
1983; IEA 2018). Scholars from various disciplines met at the
UNESCO Instute for Educaon in Hamburg (Germany) and
decided to launch a then exploratory study to test whether it
was possible to compare learning outcomes across a range of
dierent countries and cultures. They chose to assess student
achievement in mathemacs, assuming that it would be easiest
to translate into dierent languages and was thus, more likely
to result in valid comparisons across countries. Their aim
was to nd out what could be learned through internaonal
assessment, with the hope that countries could learn from
each other. As Torsten Husén phrased it: “In general terms,
internaonal studies such as this one can enable educaonalists
(and ulmately those responsible for educaonal planning and
policy making) to benet from the educaonal experiences
of other countries. It helps educaonalists to view their own
system of educaon more objecvely because for the rst me
many of the variables related to educaonal achievement had to
be quaned in a standardized way” (Husén 1967, pp. 13-14).
While the rst ILSAs were conducted by researchers with
quite minimal resourcing to sasfy academic interests in
invesgang educaon, by 1990 educaonal policymakers
had begun to realize that ILSAs could potenally provide useful
evidence-based data. However, their nancial support for ILSAs
demanded rapid outcomes. Where the rst academic study
reports were somemes launched up to eight years aer the
study was conducted (Anderson et al. 1989), this wider interest
led to a new pressure to publish results as soon as possible. An
addional consequence was an interest in measuring trends
in educaon systems, a challenge that IEA rose to meet with
TIMSS, rst conducted in 1995 and followed up by a second
Source: OECD 2019; IEA 2019
IEA Internaonal Computer and
Informaon Literacy Study (ICILS)
OECD Programme for Internaonal
Student Assessment (PISA)
IEA Trends in Internaonal Mathemacs
and Science Study (TIMSS)
IEA Progress in Internaonal
Reading Literacy Study (PIRLS)
IEA Internaonal Civic and
Cizenship Educaon Study (ICCS)
3
trend assessment four years later in 1999. The four year gap
was chosen because TIMSS assessed grade four and grade
eight students in 1995; in 1999, only grade eight students were
assessed, the aim being to compare two cohorts of grade eight
students’ results, and assess the same cohort of grade four
students four years later.
In the 1990s there was also an emerging internaonal interest
to have strong educaonal data to beer understand economic
growth. As such, the OECD, one of the leading internaonal
economic organizaons, placed a greater emphasis on
educaon and the measurement of educaonal outcomes, and
launched its own assessment study, PISA, in 2000. Previously
the OECD had only used other sources—including IEA studies
and their annual publicaon Educaon at a Glance—for their
educaonal work. However, the organizaon decided to focus
on the skills needed to operate in a modern economy rather
than on assessing what schools were teaching. Specically, PISA
claims to assess what the OECD believes 15-year-old students
enrolled in school should know in reading, mathemacs, and
science literacy; the assessment is conducted every three
years. Originally, the study focused on OECD countries, but an
increasing number of non-OECD countries now take part in the
assessment.
Toward the end of the 1990s and into the rst decade of
the 2000s, regional ILSAs were iniated including the LLECE
(Laboratoria Lanoamericano de Evaluación de la Calidad de la
Educaón) in Lan America, and PASEC (Programme d’Analyse
des Systèmes Educafs de la CONFEMEN) and SACMEQ
(The Southern and Eastern Africa Consorum for Monitoring
Educaonal Quality).
More recently, since 2015 when the Sustainable Development
Goals (SDGs) were declared by the UN, a new emphasis for
assessments has emerged. In contrast to the Millennium
Development Goals, the SDGs focus on educaonal outcomes
of educaon systems rather than on inputs like expenditure on
educaon. This results in all countries being urged to report on
the percentage of students reaching minimum prociency levels
and clearly constutes a new challenge for ILSAs. In this regard,
IEA is playing an acve role through the implementaon of
the Rosea Stone Project
1
, in collaboraon with the UNESCO
Instute for Stascs, LLECE, and PASEC. The objecve is to
develop a concordance table that translates scores resulng
from regional mathemacs and reading assessments with the
TIMSS and PIRLS scales.
DIFFERENT APPROACHES =
DIFFERENT ASSESSMENTS
Although the best known ILSAs feature a number of similaries,
there are also some substanal dierences that need to be
considered when comparing the results for dierent educaonal
systems. In the following table we have compiled some notable
dierences that test consumers need to be aware of when
comparing study results conducted by the IEA and OECD.
1. hps://www.iea.nl/studies/addionalstudies/rosea
ASPECT IEA STUDIES PISA
Study philosophy
Content selecon
Cohort selecon
Seeks to measure what is taught in schools
and the contexts of learning.
The curricula of the parcipang countries
are analyzed. Parcipang countries then
jointly develop the assessment framework
and test materials to ensure naonal
interests are acknowledged.
Samples are grade based (grades four,
eight, or twelve) to reect the structure of
curricula and to establish a direct link to
the subject teachers.
Seeks to measure selected acquired skills
of students towards the end of their
compulsory educaon.
OECD-selected experts determine the
skills that they think students should have
mastered for use later in life and assemble
the study instruments accordingly.
Samples are age based (15-year-olds for
PISA) to assess a generaon, independently
of their school pathways or grade
distribuon.
Table 1: Dierence in approaches between IEA studies and PISA
4
COMMON METHODOLOGIES
Despite dierences in organizaon and approach, ILSAs
share a quite similar set of procedures and methods for
implementaon that have been developed and rened over
the last 60 years, and have contributed to methodological
advances in regional ILSAs and naonal assessment programs
in many countries. Drawing from experse from around
the world, all major ILSAs mandate high-quality standards
carefully dened to achieve each step in the ILSA process
including: sampling rules, translaon processes, eld trial
procedures, and psychometric modelling. These methods
may appear quite complex to those unfamiliar with ILSAs,
but, for transparency, the data are accompanied by detailed
supporng technical reports and all ILSA data is made freely
available online for researchers to download. Further, both
IEA and OECD produce a large number of resources to help
promote the secondary analyses of the data. These secondary
analyses are possible and made rich by the presence of
contextual data. A recent example is ICILS where contextual
data helped to inform student digital literacy.
DIFFERENT ORGANIZATION
Beyond the studies themselves, the two organizaons—
IEA and OECD—dier in many regards (see Table 2).
Most fundamentally, their missions are not equivalent when it
comes to the relaonship between results and policymaking.
The scope of the OECD is larger than the eld of educaon,
originally grounded in the economic sector, and one OECD
mission is to give its members recommendaons in terms of
policies to be implemented. IEA, as an associaon grounded
in academic research in educaon, has no vocaon to dra
recommendaons to its members but rather, to provide
evidence on which each individual country can build adequate
policies regarding its own context. The following table
describes some of the organizaonal dierences between the
IEA and OECD.
ASPECT IEA OECD
Type of organizaon
Method of carrying
out ILSAs
Review process
Parcipaon
Decision making
Fees
Non-governmental associaon (with
naonal members).
Conducts studies cooperavely with partner
organizaons from its scienc network.
All study publicaons are rigorously
peer-reviewed.
Open to all countries. No requirement to
take part in studies.
Final content decided by the Naonal
Research Coordinators (NRC) of
parcipang countries, each of which has
an equal voice.
Each country makes an equal contribuon
to the internaonal coordinaon costs.
Governmental organizaon.
Iniates studies and tenders the studies
for each cycle. Separates out dierent
tasks (e.g., framework development,
sampling, test plaorm).
Wrien and reviewed in-house
and reviewed by board members of
parcipang countries.
The agship PISA study was originally
targeted to OECD member countries
and is now open to all economies.
The organizaon structure reects the
original centrality of the OECD membership
and vong is restricted to OECD members
and select partner economies.
Member countries pay dierent
contribuons based on their GDP.
Non-member parcipants pay a at fee.
Table 2: Organizaonal dierences between IEA and OECD
5
LIMITATIONS
There are some important limitaons to consider in
understanding and reporng ILSA results. Firstly, all ILSAs
are cross-seconal studies. This means that they measure
educaonal outcomes at one point in me for a specic
populaon. Further, although some ILSAs such as TIMSS, PIRLS,
and PISA report trends over me, they are not studies that follow
individual students. This makes it challenging, if not unfeasible,
to draw causal conclusions about student achievement and is a
clear limit of the data with many policymakers seeking specic
recommendaon for change to help improve their educaonal
system. Although some researchers seek to use advanced
models to establish causal relaonships, we personally
maintain that such models are based on strong assumpons
that are dicult, if not impossible, to achieve with ILSA data.
A second limitaon of ILSAs is that the studies are not
designed to measure individual students’ achievement nor
the results of individual schools but to reect the educaonal
results and relaonship with background informaon within
educaon systems. As such, the assessments are considered
low stakes for schools, teachers, and students. Nevertheless,
the low stakes aspect of the assessment has the advantage
of lowering tesng stress on students and schools. In fact,
ILSAs require a fairly short tesng me when compared to
high stakes assessments and are only administered to a small
representave poron of the populaon in the countries.
Finally, the content domains covered by ILSAs are not an
exhausve list of what is taught in schools. For example,
IEA’s TIMSS focuses on mathemacs and science curriculum
aainment at grades four and eight and, while crical subjects,
strong aainment in these subjects alone cannot be considered
to be a reliable measure of the overall health of an educaon
system. Some have cricized ILSAs as exerng undue inuence
on educaon policies, with naonal approaches perceived as
being replaced with a tendency to target naonal curricula
toward beer achievement in the subjects assessed by ILSAs.
Consequently, care must be taken when considering results
from ILSAs as a holisc quality measure of the educaon system,
rather the focus should be on the results as important indicators
of what students know and can do in specic subjects, and
how students naonally compare to their peers internaonally.
WHY ILSAs ARE IMPORTANT
Despite limitaons, ILSAs are vital tools for educaon
system improvement. The assessments are unique in
that the informaon provided by the data can be used to
monitor systems over me within a robust internaonal
framework, and they fall outside the governance of any
one country thus being largely independent of any single
polical system. Further, the data is available to the public
allowing researchers from around the world to explore the
data and make use of it for their own research quesons.
In many countries, research resulng from ILSAs has improved
our understanding of how educaonal systems operate,
informing policy decisions that are based on strong and reliable
evidence. Various impact studies have shown that ILSA results
have been used to support policymaking (e.g., Breakspear 2012;
Schwippert & Lenkeit 2012; Wagemaker 2013) and as reported
in the TIMSS and PIRLS Encyclopedias various educaonal
improvements have been smulated by evidence from ILSAs.
When properly understood and analyzed, the data from ILSAs
provide valuable opportunies to help inform policy decisions
as well as research into educaon system improvement.
6
Dr Dirk Hastedt is the Execuve Director of
IEA. He oversees IEA’s operaons, studies,
and services, and drives IEA’s overall strategic
vision. Moreover, he develops and maintains
strong relaonships with member countries,
researchers, policy makers, and other key
players in the educaon sector. Dr Hastedt
also serves as co-editor in chief of the IEA-
ETS Research Instute (IERI) journal Large-
scale Assessments in Educaon.
Dr Thierry Rocher was elected to the IEA Chair
posion at the 59th General Assembly meeng
in October 2018. Dr Rocher previously served
as a Standing Commiee member and General
Assembly representave for France. He is the
Head of the Oce for Student Assessment
(DEPP) at the Ministère de l’éducaon
naonale in France.
ABOUT THE AUTHORS
DR DIRK HASTEDT
DR THIERRY ROCHER
WHAT NEXT?
As ILSAs connuously modernize—most recently with a move
toward computer-based assessments—new methodological
opportunies and challenges await. This is why organizaons
conducng ILSAs are acvely promong research in the
eld of internaonal assessment. For example, IEA sponsors
academic journals, conferences, and themac reports for
outside researchers as well as employing its own research
team to help further developments in the eld of assessment.
More generally, in its renewed strategy, IEA is placing a
strong emphasis on research and innovaon, including,
for example, the promising topic of “process data”(i.e.,
digital traces le by students when passing an assessment).
Another focus of ILSA development is the exploraon of larger
and more complex dimensions, so-called “21st century skills.”
IEA has recently launched a curriculum study (21CS MAP) which
aims to map these skills. This fundamental study will be based
on what is taught in schools (the intended and implemented
curricula) before developing a concrete assessment program.
CONCLUSION
In a growing interconnected world, we believe ILSAs can
help us learn from others and, through comparison, beer
understand ourselves. IEA works diligently on assisng the
policy and research community by training researchers and
policymakers on how to interpret and analyze data, by wring
and commissioning in-depth reports into the study ndings,
and by publishing quarterly briefs that are intended to be
short digesble summaries of interesng study results. Such
acvies are undertaken in support of IEA’s mission “to beer
understand educaon pracces, processes, and policies in
order to improve the quality of teaching and learning within
and across systems of educaon.” All IEA studies and reports
follow the highest academic standards for social science
research, including complete transparency about the tesng
process and robust peer review for studies and reports.
7
Anderson, L.W., Ryan, D.W., & Shapiro, B.J. (1989). The IEA classroom environment study. Oxford, UK: Pergamon Press.
Breakspear, S. (2012). The Policy Impact of PISA: An Exploraon of the Normave Eects of Internaonal Benchmarking in School
System Performance. OECD Educaon Working Papers No. 71. Paris, France: OECD Publishing
Foshay, A.W., Thorndike, R.L., Hotyat, F., Pidgeon, D.A., & Walker, D.A. (1962). Educaonal achievements of thirteen-year-olds in
twelve countries: Results of an internaonal research project, 1959‚ 1961. Hamburg, Germany: UNESCO Instute for Educaon.
hps://unesdoc.unesco.org/ark:/48223/pf0000131437.
Husén, T. (1967). Internaonal study of achievement in mathemacs: a comparison of twelve countries. Vol. I–II. Stockholm, Sweden
and New York, NY: Almqvist & Wiksell and John Wiley.
Husén, T. (1983). An Incurable Academic: Memoirs of a Professor. Oxford, UK: Pergamon Press.
IEA. (2018). 60 years of IEA 1958–2018. Amsterdam, Netherlands: Author.
hps://www.iea.nl/index.php/publicaons/60-years-iea-1958-2018.
IEA. (2019). IEA studies [webpage]. hps://www.iea.nl/studies/ieastudies
Lockheed, M. (2011). Reecons on IEA from the perspecve of a World Bank ocial. In C. Papanastasiou, T. Plomp, & E.
Papanastasiou (Eds.), IEA 1958-2008: 50 years of experiences and memories, Vol 2. Nicosia, Cyprus: Cultural Center of the Kykkos
Monastery.
Lockheed, M., Prokic-Bruer, T. & Shadrova, A. (2015). The Experience of Middle-Income Countries Parcipang in PISA 2000-2015.
Paris, France: PISA/OECD Publishing.
OECD. (2019). PISA: Programme for Internaonal Student Assessment. hp://www.oecd.org/pisa/
Schwippert, K., & Lenkeit, J. (Eds.). (2012). Progress in reading literacy in naonal and internaonal context: The impact of PIRLS 2006
in 12 countries. Münster, Germany: Waxmann Verlag GmbH
Travers, K., & Westbury, I. (1989). The IEA Study of Mathemacs I: Analysis of the Mathemacs Curricula. Amsterdam, the Netherlands:
Internaonal Associaon for the Evaluaon of Educaonal Achievement (IEA).
Wagemaker, H. (2013). Internaonal large-scale assessments: From research to policy. In L. Rutkowski, M. von Davier, & D.
Rutkowski (Eds.), Handbook of internaonal large-scale assessment: Background, technical issues, and methods of data analysis, p. 11.
Florence, KY: Chapman and Hall CRC.
REFERENCES
ADDITIONAL RESOURCES
Hastedt, D. (2016, November 7). The history and development of internaonal assessments [Podcast]. FreshEd Podcast 49.
hp://www.freshedpodcast.com/dirkhastedt/.
Lockheed, M., & Wagemaker, H. (2013). Internaonal Large-Scale Assessments: Thermometers, whips or useful policy tools?
Research in Comparave and Internaonal Educaon, 8(3), 296–306.
Singer, J.D., Braun, H.I., & Chudowsky, N. (2018). Internaonal educaon assessments: Cauons, conundrums, and common sense.
Washington, DC: Naonal Academy of Educaon.
8
ABOUT IEA
The Internaonal Associaon for the Evaluaon of Educaonal Achievement,
known as IEA, is an independent, internaonal consorum of naonal
research instuons and governmental agencies, with headquarters in
Amsterdam. Its primary purpose is to conduct large-scale comparave
studies of educaonal achievement with the aim of gaining more in-depth
understanding of the eects of policies and pracces within and across
systems of educaon.
Thierry Rocher
IEA Chair
Dirk Hastedt
IEA Execuve Director
Andrea Neen
Director of IEA Amsterdam
Gina Lamprell
IEA Publicaons Ocer
Compass Editor
David Rutkowski
Indiana University
Please cite this publicaon as:
Rocher, T., & Hastedt, D. (2020, September). Internaonal large-scale assessments in educaon: a brief guide
IEA Compass: Briefs in Educaon No. 10. Amsterdam, The Netherlands: IEA
Copyright © 2020 Internaonal
Associaon for the Evaluaon of
Educaonal Achievement (IEA)
All rights reserved. No part of this
publicaon may be reproduced,
stored in a retrieval system or
transmied in any form or by any
means, electronic, electrostac,
magnec tape, mechanical,
photocopying, recording or
otherwise without permission in
wring from the copyright holder.
ISSN: 2589-70396
Copies of this publicaon can be
obtained from:
IEA Amsterdam
Keizersgracht 311
1016 EE Amsterdam
The Netherlands
By email: [email protected]
Website: www.iea.nl
Follow us:
@iea_educaon
IEAResearchInEducaon
IEA
