Poor ranking on international test misleading about US student performance, researcher finds

Jan 16, 2013 by Jonathan Rabinovitz
An accurate comparison of nations' test scores must include a look at the social class characteristics of the students who take the test in each country, says Stanford education Professor Martin Carnoy. Credit: Shutterstock

(Phys.org)—A comprehensive analysis of international tests by Stanford and the Economic Policy Institute shows that U.S. schools aren't being outpaced by international competition.

Socioeconomic inequality among U.S. skews international comparisons of scores, finds a new report released today by the Stanford Graduate School of Education and the Economic Policy Institute. When differences in countries' social class compositions are adequately taken into account, the performance of U.S. students in relation to students in other countries improves markedly.

The report, "What do international tests really show about U.S. student performance?", also details how errors in selecting sample populations of test-takers and arbitrary choices regarding test content contribute to results that appear to show U.S. students lagging.

In conducting the research, report co-authors Martin Carnoy, a professor of education at Stanford, and Richard Rothstein, a research associate at the Economic Policy Institute, examined adolescent reading and mathematics results from four test series over the last decade, sorting scores by social class for the Program on International Student Assessment (PISA), Trends in International Mathematics and Science Study (TIMSS), and two forms of the domestic National Assessment of Educational Progress (NAEP).

Based on their analysis, the co-authors found that average U.S. scores in reading and math on the PISA are low partly because a disproportionately greater share of U.S. students comes from disadvantaged social class groups, whose performance is relatively low in every country.

As part of the study, Carnoy and Rothstein calculated how international rankings on the most recent PISA might change if the United States had a social class composition similar to that of top-ranking nations: U.S. rankings would rise to fourth from 14th in reading and to 10th from 25th in math. The gap between U.S. students and those from the highest-achieving countries would be cut in half in reading and by at least a third in math.

"You can't compare nations' test scores without looking at the social class characteristics of students who take the test in different countries," said Carnoy. "Nations with more lower social class students will have lower overall scores, because these students don't perform as well academically, even in good schools. Policymakers should understand how our lower and higher social class students perform in comparison to similar students in other countries before recommending sweeping school reforms."

The report also found:

  • There is an achievement gap between more and less disadvantaged students in every country; surprisingly, that gap is smaller in the United States than in similar post-industrial countries, and not much larger than in the very highest scoring countries.
  • Achievement of U.S. disadvantaged students has been rising rapidly over time, while achievement of disadvantaged students in countries to which the United States is frequently unfavorably compared – Canada, Finland and Korea, for example – has been falling rapidly.
  • But the highest social class students in United States do worse than their peers in other nations, and this gap widened from 2000 to 2009 on the PISA.
  • U.S. PISA scores are depressed partly because of a sampling flaw resulting in a disproportionate number of students from high-poverty schools among the test-takers. About 40 percent of the PISA sample in the United States was drawn from schools where half or more of the students are eligible for the free lunch program, though only 23 percent of students nationwide attend such schools.

With each release of international test scores, many education leaders assert that American students are unprepared to compete in the new global economy, largely because of U.S. schools' shortcomings in educating disadvantaged students.

"Such conclusions are oversimplified, frequently exaggerated and misleading," said Rothstein, who is also senior fellow at the Chief Justice Earl Warren Institute of Law and Social Policy at the University of California – Berkeley School of Law. "They ignore the complexity of test results and may lead policymakers to pursue inappropriate and even harmful reforms."

Carnoy and Rothstein examined test results in detail from the and six other nations: three of the highest scorers (Canada, Finland and South Korea) and three economically comparable nations (France, Germany and the United Kingdom). In cases where U.S. states voluntarily participated in the TIMSS for 8th grade mathematics, the researchers compared trends in these states' scores with trends in 8th grade mathematics on the NAEP.

The researchers show that score trends on these different tests can be very inconsistent, suggesting need for greater caution in interpreting any single test. For example, declining trends in U.S. average PISA math scores do not track with trends in TIMSS and NAEP, which show substantial math improvements for all U.S. social classes.

Carnoy and Rothstein say that the differences in average scores on these tests reflect arbitrary decisions about content by the designers of the tests. For example, although it has been widely reported that U.S. 15-year-olds perform worse on average than students in Finland in mathematics, U.S. students perform better than students in Finland in algebra but worse in number properties (e.g., fractions). If algebra had greater weight in tests, and numbers less weight, test scores could show that U.S. overall performance was superior to that of Finland. 

The report comes as the administrators of TIMSS are preparing to make public more detailed data underlying 2011 test results, following last month's release of average national scores. PISA plans to release detailed data on its 2012 test in December 2013. Carnoy said that he and Rothstein will then be able to supplement their present report by comparing the most recent TIMSS and PISA results by across countries. He invites other researchers to conduct similar analyses, to see if their findings confirm those of the present report.

Explore further: Personalized advertising attracts more attention, makes the contents of ads easier to remember

More information: A copy of the report is available to download at www.epi.org/publication/us-stu… performance-testing/.

Related Stories

Inconsistent math curricula hurting US students, study finds

May 02, 2011

A new study finds important differences in math curricula across U.S. states and school districts. The findings, published in the May issue of the American Journal of Education, suggest that many students across the countr ...

Report: 8th-grade students still lag in science

May 10, 2012

Eighth-graders in the U.S. are doing better in science than they were two years ago, but seven out of 10 still are not considered proficient, the federal government said Thursday.

Worldwide study finds few gender differences in math abilities

Jan 05, 2010

Girls around the world are not worse at math than boys, even though boys are more confident in their math abilities, and girls from countries where gender equity is more prevalent are more likely to perform better on mathematics ...

Recommended for you

Why are UK teenagers skipping school?

Dec 18, 2014

Analysis of the results of a large-scale survey reveals the extent of truancy in English secondary schools and sheds light on the mental health of the country's teens.

Fewer lectures, more group work

Dec 18, 2014

Professor Cees van der Vleuten from Maastricht University is a Visiting Professor at Wits University who believes that learning should be student centred.

User comments : 32

Adjust slider to filter visible comments by rank

Display comments: newest first

antialias_physorg
4.5 / 5 (13) Jan 16, 2013
Jobs go to those who can do well. Not to those who can do well "if one were to take their socioeconomic status into account".

So upping testscores by 'adjusting for socioeconomic inequality' is just a way of fooling yourself.

Many years ago I spent 2 years in a US highschool. Almost straight A's, honors student...the whole shtick.
When I got back after those two years I was unable to reintegrate in the class I had left - I had lost a whole year's worth of education in those 2 years abroad compared to my former classmates.
dogbert
2.3 / 5 (12) Jan 16, 2013
Education standards were deliberately lowered in the 1960's and have steadily declined since.

Federal involvement in the education system is the problem. The single most effective method to raise American education standards would be to get the federal government out of education.
Claudius
3.9 / 5 (7) Jan 16, 2013
I have experience with the educational system in Finland. There were no multiple choice exams. Essays were the rule. Standards were very high. The teachers all had at least a master's degree in the subject they taught. There were no school sports, no coaches teaching mathematics. No school mascots.

I can't believe that American students (in the pubic schools), even when socioeconomic status is taken in consideration, can come even close.
EverythingsJustATheory
1 / 5 (1) Jan 16, 2013
The differences between honors and AP classes and the regular classes are extreme. GT and AP classes are extremely demanding, as much so as any university course I took at UVA (arguably top public school in the nation), although the curriculum is spread over 2 semesters instead of 1. Regular classes are a joke, even at a top 50 nationwide public high school like the one I attended.

I took regular english junior and senior year. On Monday the teacher would assign a 1.5 page (ds) essay on any topic, which I could shit out in 30 mins. On Friday when 2/3 of the class didn't have it done, she would let them work on it in class. I didn't mind, free homework time for more interesting topics, math and science.

However, given the fact I was playing tennis 2-3 hours a day in high school and knew I would never go into English or History, taking the regular courses was the correct choice for me.
EverythingsJustATheory
4.3 / 5 (12) Jan 16, 2013
Education standards were deliberately lowered in the 1960's and have steadily declined since.

Federal involvement in the education system is the problem. The single most effective method to raise American education standards would be to get the federal government out of education.


The problem with not having the federal government involved is that you get nutjobs (Texas, Kansas, etc...) that want to teach creationism in Biology class. Teaching creationism is fine in a religion class, not in a science class.
EverythingsJustATheory
1 / 5 (4) Jan 16, 2013
The biggest factor in the quality of education in the US declining is the quality of the teachers. And this mostly has to do with women finally being given a chance to do something other than be a teacher or a nurse.

All the women that are now top doctors, lawyers, scientists, etc.. used to be teachers 50 years ago. Now they have other employment opportunities that pay a lot better and are a lot more challenging/interesting.

One of the downsides out of the womens' movement, which was an extremely positive thing for society. But sometimes positive things can also have slight negative feedbacks.

The solution is to pay the teachers who have higher educational creditials (not necessarily the current ones) a lot more to entice higher quality teachers.

It'd be interesting to look at the correlation between the average teacher salary and performance between nations. I wish I had the time for such things.
Caliban
2.3 / 5 (3) Jan 16, 2013
All else aside, this study has identified a fundamental flaw in in the administration of this testing regime, namely, that the methodology must be uniformly implemented in order to guarantee the greatest level of validity.

In a comparative regime, it is necessary to compare apples to apples.

Obviously, now that this is understood, an ongoing analysis of the sampling methods has to be undertaken in order to avoid further "false negatives" --and not out of bias towards any one group, but simply out of basic fairness and objectivity towards all.

Doug_Huffman
1.8 / 5 (5) Jan 16, 2013
Charlotte Thomson Iserbyt documents The Deliberate Dumbing Down of America from John Dewey to Outcome Based Education. $40 at AmaXon or free with DuckDuckGo-fu.
dogbert
1.6 / 5 (17) Jan 16, 2013
EverythingsJustATheory,
The problem with not having the federal government involved is that you get nutjobs (Texas, Kansas, etc...) that want to teach creationism in Biology class. Teaching creationism is fine in a religion class, not in a science class.


Yes. You want preference for your religion and banishment for all other religions. Most religions would like a mandate for themselves and banishment for all others.

Our education system did just fine without the federal government and our education system was then the envy of the world. Now it is laughable -- all because of federal involvement.

There is not even a constitutional provision which gives the federal government the power to direct education. It is totally inappropriate and counter productive.
Claudius
3.6 / 5 (8) Jan 16, 2013
And this mostly has to do with women finally being given a chance to do something other than be a teacher or a nurse.


In Finland, the country that first gave women the vote, a large proportion of their highly skilled teachers (and other professionals) are women.

I am old enough to remember women in 1960's America that were very good teachers. I knew one personally that retired when the standards were lowered.

America has a lot to learn about education, and there are good models on how to do it. All it will take is the political will to do it, and we can have an exceptional system. It will mean giving up a lot of the "touchy-feely" "self-esteem" paradigm that has defined the decline of education in this country. Standards have to be much, much higher, for teachers as well as students, and poor students need to flunk out. Finland has trade schools for the non-academically inclined, starting around age 13.
FrankHerbert
2.8 / 5 (16) Jan 16, 2013
Dogbert always the coward with your christian propaganda.
PCB
3.7 / 5 (6) Jan 16, 2013
Also, US Teacher Unions heavily resist attempts at teacher quality reform.
Shootist
2 / 5 (12) Jan 16, 2013
Education standards were deliberately lowered in the 1960's and have steadily declined since.

Federal involvement in the education system is the problem. The single most effective method to raise American education standards would be to get the federal government out of education.


The problem with not having the federal government involved is that you get nutjobs (Texas, Kansas, etc...) that want to teach creationism in Biology class. Teaching creationism is fine in a religion class, not in a science class.


While it is true there are "nutjobs" who want to teach creationism, having the Federal Government involved in educations doesn't stop Texas, or where ever from teaching what ever they wish.

Your statement, however, suggests you are an elitist git.
EverythingsJustATheory
4 / 5 (8) Jan 16, 2013
dogbert,

Explain how the scientific method can be used to demonstrate repeatable experiements that back up the claims of creationism. When you have satisfied the above, then and only then, may creationism be taught in science classes.

I made no statements that the existance/non-existance of god should be discussed in science classes. Only things that have verifiable and comprehensive evidence of their validity should be taught as fact. Evolution comes to mind. Evolution makes no definitive statement about how life began, only how it has developed since its inception. Evolution is not a religon and is backed by every bit of evidence discovered so far, in other words, it is backed by the scientific method.
EverythingsJustATheory
3.4 / 5 (7) Jan 16, 2013
Education standards were deliberately lowered in the 1960's and have steadily declined since.

Federal involvement in the education system is the problem. The single most effective method to raise American education standards would be to get the federal government out of education.


The problem with not having the federal government involved is that you get nutjobs (Texas, Kansas, etc...) that want to teach creationism in Biology class. Teaching creationism is fine in a religion class, not in a science class.


While it is true there are "nutjobs" who want to teach creationism, having the Federal Government involved in educations doesn't stop Texas, or where ever from teaching what ever they wish.


While technically true, I'd imagine that federal funds may be withheld if a state decided to teach that the world is only 5,000 years old in a science class.
jerel
4 / 5 (8) Jan 16, 2013
I'd wager most of the commenters here are not from a "disadvantaged socioeconomic class". In schools in ghettos, barrios, and inner-cities (for example) it's all the teachers can do to keep order in the classroom. Peer pressure to not appear smart is enormous at an age where such things really matter. Violence in the student's lives outside the classroom is a constant threat and fact of everyday life. That coupled with the influx of children for whom English is not their first language, and they are not literate in their first language, and yet they must be taught, somehow, to be literate, puts enormous pressure on the teachers and on the other students. If you read the article carefully, it says that we actually serve that population better than the other countries to which we are so often compared. Nevertheless, if you are surveying mostly privileged students in one country, and mostly underprivileged or "disadvantaged" in another, how can you claim the comparison is fair?
dogbert
1.4 / 5 (12) Jan 16, 2013
EverythingsJustATheory,
I made no statements that the existance/non-existance of god should be discussed in science classes.


Sure you did. And you are as welcome to your religion as anyone else is to their religion. You are not welcome, however, to demand that your religion be taught to the exclusion of other religions.

Naturalism is OK to teach, just don't expect to exclude all other religions.
kochevnik
3.2 / 5 (9) Jan 16, 2013
@dogbert Naturalism is OK to teach, just don't expect to exclude all other religions.
Naturalism isn't a religion. It's a philosophy which denies the supernatural. You failed again
@shootster Your statement, however, suggests you are an elitist git.
Insisting upon naturalism makes one elitist? Fine by me because you are excluding anyone passing midschool
proudfoots
5 / 5 (2) Jan 16, 2013
I am not so sure you can just blame the education system or the teachers in any country. Here in Australia the younger generations are becoming more and more ignorant of education and really only care about what directly affects them. Family dynamics are changing and the learning that children used to get from their parents doesn't seem to be happening on the same level. What about the technology factor? Why would you need to know anything when everything you need to know is as simple as a google search?!

This is most certainly a complex problem that will not be solved by one solution alone.
ClydeG
1.8 / 5 (5) Jan 17, 2013
Pretty damning evidence against the corporate reform measures that do nothing to address the harmful affects of poverty on student intellectual growth and have irreparably harmed the efforts of American teachers to bring meaningful learning experiences to their students only to be ridiculed and demeaned by corporate press propagandists and their corporate reform cohorts and political tools....
brianweymes
3.8 / 5 (5) Jan 17, 2013
"Jobs go to those who can do well. Not to those who can do well "if one were to take their socioeconomic status into account"."

You missed the point of the article. It's saying that the students chosen for the tests were not a balanced sample of the population, but biased in favor of traditionally low-scoring students. Let me put this in another more obvious way: if American students scored low on an international Mathematics test one year, but half the students who took the test were African Americans, when African Americans represent only 12% of the American population, do you think that would really be an accurate reflection of the average American student?

None of you know what you're talking about.
obama_socks
2 / 5 (9) Jan 17, 2013
"Jobs go to those who can do well. Not to those who can do well "if one were to take their socioeconomic status into account"."

You missed the point of the article. It's saying that the students chosen for the tests were not a balanced sample of the population, but biased in favor of traditionally low-scoring students. Let me put this in another more obvious way: if American students scored low on an international Mathematics test one year, but half the students who took the test were African Americans, when African Americans represent only 12% of the American population, do you think that would really be an accurate reflection of the average American student?

None of you know what you're talking about.


What does the fact that they've come from the 12% of the population have to do with their ability to score high on tests? It has nothing to do with race or socioeconomic level, but it has everything to do with the QUALITY of the education they received prior to the testing.
obama_socks
2.1 / 5 (8) Jan 17, 2013
The problem with teachers in the U.S. is that they are, in large part, poorly trained in how to motivate students to study and learn the material they are being taught. Every child is different from everyone else. Each one is an individual with individual needs and abilities. Some need a great deal of attention while others may breeze through because learning is easy for them.
Montessori schools are far more successful at motivating kids to learn because the teachers have special training to spot the easily taught students and work with the students who require a bit more attention and a little more "push" to achieve the goals of the curriculum. If the nation's teachers were trained the Montessori method, it is almost guaranteed that students would have a better chance of scoring high on tests and learn the material required for superior grades.
But it is doubtful that the U.S. Dept of Education would agree to such training of public education teachers.
stefan_calisto
4 / 5 (4) Jan 17, 2013
Sorry Obama socks. I agree to a lesser extent, have those Montessori teachers teach at my school for a month and they would quit. The special training you say, well that special training is for kids with GT, not poverty. Socioeconomic low kids are very hard to understand and teach. You cannot say, well it is the teachers fault. A lot of our kids are physically hungry, literally. They drink our ranch dressing like no tomorrow. They are sleep deprived, nutrition is a farce. Parents are gone, fathers left when they knew the woman was prego. Most of our kids are bastard children. I hate seeing it, that is why I teach them. There is no one panacea, it is a multi systematic approach to learning about these kids. First is the complete lack of parents. We have kids in my school who are prego at seventh grade, yeah sure that girl will raise the child and sure the boy left very quickly. It is a cycle we cannot stop. Our kids have mental issues out the wazooo yet we do not have therapists.
stefan_calisto
5 / 5 (5) Jan 17, 2013
Also Obama socks, I have seen our best teachers fail with these kids. They cannot get past the rut of the life they live outside of school. To say it is the quality of teaching, well yes I agree to a certain extent. Grant it there are terrible, over the hill teachers collecting a pay check...it is just not that though. I invite you also into our special education dept. where there are a multitude of learning problems. Again I can go back to a parent who was a drug riddled addict who bore the child, I can say it is heredity. There are FAR too many reasons for low score outcomes. There is no one panacea. I would say in my educated self, it is a cultural problem.
VendicarD
3.7 / 5 (6) Jan 17, 2013
Mindless Claptrap from Obama-Sox.

"The problem with teachers in the U.S. is that they are, in large part, poorly trained in how to motivate students to study and learn the material they are being taught." - Sox

The failure of Americaa's Education system is a directresult of the failure of America's cultural failure.

A nation based on Money Grubbing, Gun Grubbing, and rabid self interest is a society that is incapable of survival.
VendicarD
4.3 / 5 (6) Jan 17, 2013
No they weren't.

You are lying.

"Education standards were deliberately lowered in the 1960's" - DogBerTard

Both teaching and education standards have been increased since then. Back in the 60's you didn't need to have a degree to teach public school children. Today you do.

What has changed since the 60's is that the nuclear family has been destroyed by Capitalism. Now both the mother and father are corporate slaves and the children are left to raise themselves with a little help from the school.

But schools are not intended to, designed to, or funded to replace parents.

Hence the failure of American society. Capitalism has destroyed America.

VendicarD
4.2 / 5 (5) Jan 17, 2013
There has been a constant push for teacher reform since the grades started to drop in the 1970's.

All such reforms have failed.

"Also, US Teacher Unions heavily resist attempts at teacher quality reform." - PCB

The reason is obvious. The problem is not teachers. The problem is the students.

When they graduate, American Students are experts at being Gun Grubbing, Money Grubbing, Crack heads.

This is what they learn from the people around them, and the corporate/capitalist media they consume.
exo ra luna
1 / 5 (2) Jan 18, 2013
I may not be as educated as you, but I would beg to differ on your 'reason'ing. Unless, of course, by 'cultural' you mean 'a national' problem.
xen_uno
2.6 / 5 (5) Jan 19, 2013
Ven - "Hence the failure of American society. Capitalism has destroyed America."

And the opposite (socialism) has helped how many countries? One only has to look at the UK to see how a formerly capitalist country can sink into a self loathing broken society with a dead economy and no future for most. People like Obama are accelerating the US's journey down that same path of destruction.

Educational standards are dropping because excuses for being a certain color and of being a member of a so called dis-advantaged socioeconomic group are all the rage now. No Child Left Behind was/is an awful piece of fed legislation that even prompted my sister to get out of teaching entirely. Til the Feds get out of K thru 12 education it's only going to get worse.
DailyReader
1 / 5 (1) Jan 19, 2013
It's not just the teachers.

Take the police for example.

In New Zealand a policeman does the same training as an FBI agent in USA.
DruidDrudge
1 / 5 (6) Jan 20, 2013
I look at our kids today and think they have it too easy. pretty sure my father thought the same of me.
I compare 15 year old kids with myself, I should be comparing them with when I was 15.
they dont look so bad now. when I was in grade 8, there were kids in my class that could not read.
they were farmers kids, their parents saw no reason they needed more.
truth is, each generation is brighter than the last
to wit, there are more physicists alive today than have ever lived.
look at technoogy and medicine, the advances are coming faster than we can comprehend.
only arrogance claims WE were the smarter ones.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.