Learn About: Evaluating Performance | Common Core
Home > Evaluating performance > A guide to international assessments > Archived chat: International assessments and student achievement
| print Print

Archived chat: International assessments and student achievement

The Center for Public Education hosted a live online chat featuring Jim Hull, policy analyst for the Center for Public Education and author of the Center's guide to international assessments called More than a horse race: A guide to international tests of student achievement. Following is the transcript of the discussion.

Jim Hull writes:

Hello everyone and welcome to our discussion on international comparisons.

A parent from Oakhurst, NJ asks:

Don't other countries track their students much more aggressively? In other words, aren't assessments really comparing all U.S. students against only the best students of other nations?

Jim Hull writes:

While systems do vary between countries on how they educate their students, since the 1990's almost all countries have followed the U.S.lead in providing universal education. Furthermore, there is less tracking through the middle grades.

An individual from Macon, Georgia asks:

Does it really matter if we're number one?

Jim Hull writes:

Although we want our students to achieve at the higest levels, it is not so important to be number one as it is to be competitive...to make sure all of our students are competitive in the global economy. In fact, even though our students are not number 1 on any international test they are very close to the top and very competitive on reading, especially our 4th graders.

An individual from Washington, DC asks:

Given the huge differences between nations, what legitimate lessons can be gleaned from comparing achievement levels on international assessments?

Jim Hull writes:

The differencers between industruialized countries is not as large as numeric rankings suggest. What's important is to consider when looking at international results is not where a country ranks numerically, but whether the differences are statistically meaningful. All international assessments now report results based on statistical significance, which means differences in scores did not happen by chance. For example, the U.S. ranks 15 out of 27 in reading on the PISA assessment but only 3 scored significantly higher.

A school board member from Special School district of St. Louis County Missouri asks:

The US provides universal education, including students with cognitive and other disabilities. To what extent are such students included or excluded in the testing conducted in other countries?

Jim Hull writes:

As we said earlier, countries have moved toward universal education and that includes students with disabilities. Students with disabilities are included in each of the three major international tests except for the most severely cognitively disabled. Countries generally exclude less than 5 percent of their students. And test administrators have procedures for ensuring a represntative sample of the nation's student population is tested including students with special needs, and countries must report the number of students excluded and why they were excluded.

A reporter from Washington, DC asks:

The recent Skills Commision report ( http://www.skillscommission.org/executive.htm) suggested that U.S. schools could be improved to the point where most students graduate after 10th grade. The comission bases this belief largely on the example of European nations such as Denmark, where both curriculum and tests emphasize understanding and reasoning, ahead of memorization and multiple-choice testing. Could U.S. schools improve themselves by adopting some of these European methods, and is the Skills Commission suggestion that the U.S. could have most students graduate after 10th grade realistic, or even desirable? Thanks, Paul Basken/ Bloomberg News

Jim Hull writes:

It's true that American students do not perform as well on problem-solving, so yes, we can do better. Some European countries score higher on these measures. Researchers are looking into the practices in these countries and they should be instructive for ours. Keep in mind that other countries are learning about some of the things we do well here, too. One of the benefits of the international assessments if for countries to share best practices with each other.

The notion of graduating after 10th grade is not brand new, but it's not a universal practice even in Europe. There are a lot of considerations, especially at a time when economists and policy makers are calling for more years of education, not fewer. Also, to do something like this in the U.S. would require a major cultural shift, given the central place the high school has in the community. Nonetheless, some states have had some success with similar models---for example, the middle college approach in North Carolina and tech prep "2 plus 2" programs. These serve some students well.

An individual from Arlington, VA asks:

Does the prevailing emphasis in American education moving toward collaborative learning and teaching, how does this affect test performance, where standardized tests and answers are based seemingly more on rote memory than understanding?

Jim Hull writes:

Just because a test is standardized does not necessarily mean it is designed to measure only memorization. The international assessments, especially PISA, are designed to measure higher-level thinking and reasoning skills. On each of the assessments, at least half of the test items are open-ended, meaning they require the test-taker produce the response, not choose one as you in a multiple choice format. See for youself by visiting our analysis More than Horse Race: A Guide to International Assessments of Student Achievement and selecting the links to sample test items.

A school board member from Enter your location [city, state] here. asks:

How worried should I be when I hear that China and India are producing far more engineers than the U.S., given that there is also a significant difference in populations between the two countries and the U.S.?

Jim Hull writes:

Unfortunately, neither India nor China participate in any of the current international tests. However, both countries are so huge that they could educate only a small percentage of their population and still produce great numbers of engineers. We need to be aware of these possibilities because we are in a global economy. We do know what individual students need to be successful after high school so this is where, as educators, policy makers, and parents, need to concentrate our efforts--to make sure our students can compete internationally.

An individual from Fairfax, VA asks:

Isn't it true that the United States' high performing students are doing just as well when compared to student s in other countries?

Jim Hull writes:

It varies by test and by age and by subject. In reading among our 4th graders we have as many high-end achievers as other industrial nations. On the other hand, in math, our we have fewer 4th graders compared to Japan, England, and Russia performing at the top levels. It's also important to compare the proportion of students who are in the lowest achievement levels. Raising the performance of students at the low end needs to be a priority. This is relevant to achivement gaps based on socieconomic status, which almost every country experiences. The United States's gap is near the international average based on parent education level, occupation, and native language. Germany has the widest achievement gap while Canada, for example, with a very diverse population, has one of the smallest gaps.

A principal from asks:

This seems like the view from 30,000 feet. How is all of this relevant to the students in my elementary school?

Jim Hull writes:

While we tend to focus on the numbers, the tests also collect reams of data about the students taking the tests as well as their schools, their teachers and instruction. Educators can get the most useful insights by looking into this background data, which provides the context that necessary to make sound instructional decisions. It's also important to use the data as benchmarks for what schools are teaching internationally--for example, in the 1990s, researchers used TIMSS data to identify high-performing countries and then studied the math curriculum of those countries . They found that middle school students in these countries were being taught algebra which was not typically introduced until 9th grade in the U.S. Interstingly, as we've been enrolling more 8th graders in algebera, U.S. math scores are going up.

A reporter from Washington D.C. asks:

You mentioned the achievement of 4th graders in reading and math. Is there something significant about the performance of that grade opposed to others? Or was that grade just chosen as an example? For the purpose of international comparisons is it more beneficial to test students of one grade and not another? Is more (better) information gained from doing so?

Jim Hull writes:

Test administrators usually try to test a primary grade, and middle grade and a high school grade. 4th grade or age 9-10 is cognitiively a gateway into higher level skills--for example it is in 4th grade that students begin to use reading as a means of learning (as opposed to learning how to read). In other words, reading for comprehension plays a bigger part. Students make a similar cognitive leap around 8th grade. PISA tests 15 year olds because they are able to get students in almost every country toward the end of compulsory schooling.

A state school board association staff member from Vancouver, BC, Canada asks:

Places like Singapore and Wales are moving away from standardized testing because they don't believe they are getting good results in terms of student achievement. Other jurisdictions are going the other way. What is the evidence one way or the other?

Jim Hull writes:

If you're looking at standardized testing internationally it's a very mixed bag. Some high-performing countries, like Finland, don't test their students very often. Others like England have a long history of using high-stakes tests. The technology of writing standardized tests is comstantly improving so that the tests can better assess higher-level skills. Local jurisdictions, of course, are not limited to using test results to evaluate instruction or make policy but can use a range of performance data including student work and classroom observations.

A school board member from California asks:

Is it true that the U.S. tests all its students while other countries only test a few? Don't these tests compare apples to oranges?

Jim Hull writes:

This was true in early international assessments, but not today. Since the 1990s due to better sampling techniques and a move by more countries to universal education, the results represent the performance of the whole student population, including students who atttend, public, private, and vocationsl schools, students with special needs, and students who are not native speakers of their nation's language. It's true that the sampling is not perfect, but it is greatly improved and we can make fair comparisons.

A public information officer from Indiana asks:

Regarding the international comparison tests that - Are the tests and their results handled by an outside, impartial company? Or does each ountry supply their results? Is there anyway the data can be corrupted to favor one country over another?

Jim Hull writes:

The tests are developed by international researchers, testing experts, and subject matter experts from across the globe. All of the test items are screened for cultural, national or group bias as well as any problems with translation. The sponsors of the assessments are independent organizations funded by the countries that participate. The main ones are the Organization for Economic Cooperation and Development (OECD) in Paris and the International Association for the Evaluation of Educational Achievement. There are independent audits of each of each assessment and the scoring process is monitored.

Jim Hull writes:

Thank you for joining us today. The transcript from today's discussion will be posted on teh Center's web site www.centerforpubliceducation.org.

Center for Public Education writes:

Join us for our next chat on Friday, March 1 when we discuss the research on homework. Other topics planned for the future chats include dropout prevention and the value of pre-K education.

Home > Evaluating performance > A guide to international assessments > Archived chat: International assessments and student achievement