Comments on PISA reporting

The following statements about the PISA were made in a BBC report [1]:

Seekers after educational excellence once used to head pilgrim-like towards Finland. This was the most quoted example of a high performing school system, even though in many ways it was a very distinctive and individual system. Scandinavia was the education world’s sensible successful neighbour.
But Finland has slipped downwards and the gloom has spread across Nordic countries, with Sweden among the biggest fallers. Norway and Denmark are absent from the top end of the tables. Their sluggish performances have been overtaken by countries such as Estonia, Poland and Ireland.

These statements were made based on PISA data. Anyone who quotes and therefore stands by these statements implicitly accepts the PISA data underpinning these statements. It would be strange for anyone to simultaneously denounce PISA data and embrace such statements based on PISA data.

There are also several validity issues with these statements. For example, Norway and Denmark didn’t just become absent from the top end recently, they have never been in the top 10 since PISA began in 2000. Rather than slipping or falling, Norway’s rankings have fluctuated up and down. The same can be said of the Science ranking of Denmark.

Math ranking Science ranking Reading ranking
Norway 2000 17 13 13
2003 22 28 12
2006 28 24 25
2009 21 24 12
2012 30 31 22
Denmark 2000 12 22 16
2003 15 31 19
2006 15 18 19
2009 19 26 24
2012 22 27 25

1

The notion that Norway and Denmark have been overtaken by Estonia and Ireland is also problematic.

For science scores, Estonia and Ireland were never behind Norway and Denmark to begin with and so cannot be said to have overtaken Norway and Denmark.

PISA science scores 2000 2003 2006 2009 2012
Norway 500 484 487 500 495
Denmark 481 475 496 499 498
Estonia 531 528 541
Poland 483 498 498 508 526
Ireland 513 505 508 508 522

2

For reading scores, with the exception of 2009, both Estonia and Ireland were already better than Norway and Denmark when they joined PISA in 2006 and 2000 respectively so once again overtaking is not the right word to describe their achievements.

PISA reading scores 2000 2003 2006 2009 2012
Norway 505 500 484 503 504
Denmark 497 492 494 495 496
Estonia 501 501 516
Poland 479 497 508 500 518
Ireland 527 515 517 496 523

3

For math scores, Estonia was never behind Norway and Denmark and so could not have overtaken them. The latest score difference between Ireland and Denmark is too fine to be useful for ascertaining anything. Ireland also didn’t just catch up with Norway, Ireland has always been better than Norway except for 2009.

PISA math scores 2000 2003 2006 2009 2012
Norway 499 495 490 498 489
Denmark 514 514 513 503 500
Estonia 515 512 521
Poland 470 490 495 495 518
Ireland 503 503 501 487 501

4

Thus, the claims made in those statements were more wrong than they were right and are therefore not worth quoting.

Finland
In the case of Finland, there is no evidence that its science scores have slipped over the years although its math and reading scores may have slipped by 3.2% and 4% respectively since it first participated in PISA rankings in 2000. It may be over presumptuous to associate this 3.2% or 4% dip in math and reading scores respectively as evidence that the Finnish education model is finished already. Given the many measurement issues with PISA, the 3.2% or 4% could simply be due to measurement issues. There is no strong enough evidence yet to condemn the Finnish education model.

2000 2003 2006 2009 2012 2012 / 2000
Finland math 536 544 548 541 519 -3.20%
Finland science 538 548 563 554 545 1.30%
Finland 546 543 547 536 524 -4.00%

5

Furthermore, Finland’s science score still ranks amongst the top 5 in the world, not a gloomy ranking at all.

2000 2003 2006 2009 2012
China Shanghai, China #N/A #N/A #N/A 575 580
Hong Kong, China #N/A #N/A 542 549 555
Singapore #N/A #N/A #N/A 542 551
Japan 550 548 531 539 547
Finland 538 548 563 554 545

Finland’s reading score is still ranked 6th in the world, again not a gloomy ranking.

2000 2003 2006 2009 2012
China Shanghai, China #N/A #N/A #N/A 556 570
Hong Kong, China #N/A #N/A 536 533 545
Singapore #N/A #N/A #N/A 526 542
Japan 522 498 498 520 538
South Korea 525 534 556 539 536
Finland 546 543 547 536 524

Finland’s math score is now ranked 12th, not as sterling as before but definitely not something to be ashamed of.

2000 2003 2006 2009 2012
China Shanghai, China #N/A #N/A #N/A 600 613
Singapore #N/A #N/A #N/A 562 573
Hong Kong, China #N/A #N/A 547 555 561
Taiwan #N/A #N/A 549 543 560
South Korea 547 542 547 546 554
Macau, China #N/A #N/A 525 525 538
Japan 557 534 523 529 536
Liechtenstein 514 536 525 536 535
Switzerland 529 527 530 534 531
Netherlands #N/A 538 531 526 523
Estonia #N/A #N/A 515 512 521
Finland 536 544 548 541 519

What is interesting to note is that by and large, Finland’s rankings have fallen over the years, not because it was overtaken by other countries but because of the gradual addition of East Asian nations into the PISA list of countries. East Asian nations have shown themselves to be quite good at topping PISA rankings and displacing other nations to lower ranks. The fact that Finland is still the best scoring non-East-Asian nation in science and reading is itself a very commendable achievement.

Culture
The following statements from the same report [1] are also problematic:

The runaway success story has been the achievement of a clutch of Asian education systems. But results saw the OECD’s Andreas Schleicher challenging any stereotypes about some places having an inherent “culture” of education. Results in Shanghai and Vietnam are much better than three years ago, he says, but the “culture” hasn’t changed. The improvements reflect a deliberate policy of ensuring that a high proportion of pupils will succeed.
This also applies in other parts of the world. Poland has been transformed into one of the best school performers in Europe and the OECD argues this reflects an active policy of change and not any inherent quality of its culture. The implication of this is that other countries could follow their example.

Improvement despite having kept culture constant is no proof that culture is therefore not responsible for the improvement. Just as a car rapidly increasing its speed from 0 to 100 km/hr while carrying the same engine doesn’t prove that the engine is not responsible for the rapid speed increase. Or a school with rapid improvement despite having the same principal and teachers is no proof that the principal and teachers hadn’t contributed to the improvement.

Holding Poland up as an example that rapid improvement can occur outside Asia is again no proof that Poland will eventually attain the high levels achieved by the East Asians. Just as a school that won the best improvement award is no proof that it will eventually become an RI.

Inaccurate labelling
Another report [2] by the same BBC author carries a picture with inaccurate labelling. The picture below suggests that the UK’s math PISA score is just average amongst nations that include Brazil and Peru. But ‘494’ is not the average score of all nations that participated in PISA but the average of OECD countries instead. The average of all nations’ PISA math scores should be ‘473’ instead. The British thus performed better than average of all nations by 21 points.

6

[1] http://www.bbc.co.uk/news/business-25205112, BBC Business news, Pisa tests: What do we know now?, Sean Coughlan, 4 Dec 2013

[2] http://www.bbc.co.uk/news/education-25187997, BBC Education & Family news, Pisa tests: UK stagnates as Shanghai tops league table, Sean Coughlan, 3 Dec 2013

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: