Opinion Change – Judgement and Decision Making


Down with My School
November 2, 2010, 5:25 am
Filed under: Uncategorized

Every year across the country, children in grades 3, 5, 7 and 9 are tested on their reading, writing, grammar and numeracy abilities in the National Assessment Program – Literacy and Numeracy (NAPLAN) tests. Problems have plagued the NAPLAN tests since they started in 2008. Every year cheating allegations are made (Chilcott & Schultz, 2010), states and territories fret over where they place overall in the nation, and this year teachers went on strike and threatened to do so again on the days of testing (Qld behind in literacy, 2010).

Issues with NAPLAN testing have only increased since the introduction of the My School website in January this year. The My School website, created by the Australian Curriculum, Assessment and Reporting Authority (ACARA), displays the NAPLAN test results from all schools across Australia, and associates each school with up to sixty statistically similar schools to compare academic performance (ACARA, 2010). The Index of Community Socio-Educational Advantage (ICSEA), a measure specifically created for this purpose, is employed on the My School website to compare statistically similar schools. A number of variables were considered when creating the website, including a school’s indigenous population, school’s location, and the Socio-Economic status of the school’s location (ACARA, 2010). However, these statistically similar schools can vary quite significantly in regards to student population. Not only do these schools vary on significant traits not included in the statistical analyses to determine similarity between schools, but an election promise made by the Labor party in the lead up to the 2010 election (Set a National standard, 2010) could see a performance-based pay scheme being introduced for teachers. This could lead to a number of problems, none of which should be occurring.

Therefore, this essay will propose that the My School website be taken offline permanently. Several problems in regards to the My School website will be discussed, and a number of judgement and decision making processes will be applied to these situations to illustrate the benefits of NAPLAN test results’ anonymity.

Judgement and decision making processes such as naïve empiricism, predicting from imperfect predictors, selective processing, the hindsight bias and more will be applied to issues regarding the criteria used to compare schools, the impact of cheating on the NAPLAN test results published on the My School website, and the election promise for the introduction of performance-based pay for teachers.

Naïve empiricism is the belief that interpretations can be applied to data once it has been viewed neutrally. Naïve empiricism also requires the use of safeguards to view the data neutrally and to apply interpretations, however this has not been done by the developers of the My School website. Firstly, the data has not been viewed neutrally, because it has not been viewed as a means of testing an individual’s performance across his or her schooling career (Educational groups slam, 2010). Therefore, even before interpretations have been applied, the neutrality of the data has been violated. This can affect the results seen on My School because whilst some individual’s are exempt from participating in the NAPLAN test (e.g., children in Special Education Units), the My School website publishes data for these students regardless, allocating them scores of zero (Experienced Senior Teacher, personal communication, 14 September, 2010). These outlier zero scores are then included with the rest of the grade’s data, reducing the average. As the NAPLAN test results published on the My School website are becoming a main indicator of a school’s success (Government accused, 2010), schools with poor performance could be impacted by a decrease in student enrolments (Hudson & McMahon, 2010).

This is also an example of the representativeness heuristic, the process of comparing and judging situations to establish whether they are typical of known occurrences. This is because the My School website does not make it known that the results of children in SEUs are exempt but their null results are included, yet a number of schools in Australia who have a SEU attached to the school would be affected by this, thus making it a typical occurrence.

Problems applying interpretations to the NAPLAN data for the My School website also exist because the variables used in the statistical analyses used to determine similar schools do not include all possible indicators. Whilst variables do include such things as student indigenous population and the Socio-Economic Status of the students, other relevant information such as the percentage of student population attached to a SEU and the percentage of students who’s first language was not English was not included. Also, the data used in to create the ICSEA was not taken from the student population data, but from the 2006 Australian Bureau of Statistics Census data for the school’s area (McAlpine, 2010). Whilst these issues are being investigated for proposed changes to the My School website (Changes proposed, 2010), this is currently an example of predicting from imperfect predictors. A submission to the Senate by the Independent Schools Queensland group agrees (Independent Schools Queensland, 2010), pointing out a number of examples where schools are not similar but deemed so statistically, and stating that the ICSEA does not include a number of complicated variables important in predicting performance.

Interpreting the data from the My School website poses problems as well, due to a lack of information. Wu (2010) discussed the numerous flaws in using NAPLAN test results, including the error variance associated with a test of only forty questions and a lack of causality in the data. Wu illustrated through statistical analyses that a child’s results on the NAPLAN tests could fluctuate approximately twelve percent above or below their average score, and that growth measured within-participants over time (one of a number of proposed changes to be published on the My School website) could fall within the error margin of the previous test. For a more accurate result, the child should be tested several times over the year and an average should be taken, therefore regressing to the mean.

Cheating allegations during the NAPLAN tests include low-achieving students being asked not to attend school on the days of testing (Experienced Senior Teacher, personal communication, 14 September, 2010), students given more time to complete tests and teachers giving students the answers to questions (Owens & Lim, 2010).

The Fundamental Cognitive Error (FCE), a person’s underestimation of how their thoughts and beliefs affect their ability to observe and judge fairly, naïve realism, which occurs when people avoid the truth by deceiving themselves, and selective processing, which includes dismissing evidence inconsistent with one’s belief, are all judgement and decision making processes that apply to the 51 reports of cheating associated with the NAPLAN tests (Owens & Lim, 2010). The principal at Coorparoo State School was stood down one week ago after it was revealed he gave 27 students extra time to complete the NAPLAN tests in May. The Queensland Education Department believes the test results from these children will not affect the data and will be published as per normal on the My School website (Chilcott, 2010). This is an example of the FCE, naïve realism and selective processing because the Education Department is not being fair in uploading these inaccurate results, they are deceiving themselves in thinking these results won’t affect the averages at the school and they are dismissing evidence that suggests these results scored by cheating will affect the grade’s average.

If cheating is already occurring without incentives, the incidence rate will increase significantly if Labor’s election proposal of performance-based pay for teachers is introduced. In their book, Levitt and Dubner (2005) investigated allegations of cheating before and after the introduction of high-stakes standardised tests as part of the ‘No Child Left Behind’ (NCLB) policy in the United States of America was introduced in 2001. They found that approximately five percent of teachers cheated somehow in high-stakes tests, and that a massive increase in cheating occurred when a financial incentive was introduced. Interestingly, Levitt and Dubner (2005) also revealed that younger teachers were more likely to be cheating teachers, which could become an issue in Australia as the mean age of teachers is decreasing rapidly.

The hindsight bias occurs when an individual attempts to ignore the outcome of an event, even when the already know what is going to happen based on previous experiences. The Australian Government would be ignoring the most likely outcome of performance-based pay for teachers if this scheme is introduced. The incentives to cheat would be greatly increased, with more funding going to schools with an increased student attendance and better NAPLAN test performance (Set a National standard, 2010), and teachers receiving bonuses when their class performs well on the NAPLAN tests. The Liberals agree with this thought, stating that nothing useful is measured on NAPLAN tests and that a performance-based pay for teachers will encourage cheating (King, 2010).

Another negative implication for performance-based pay is the funding for poorer schools. It is typically poorer schools who performance worse on the NAPLAN tests, so it would be these disadvantaged schools becoming more so due to insufficient funding.

This essay has discussed a number of points as to why the My School website should be taken offline permanently. Inaccurate interpretations being applied to data both before and after publication on the My School website are examples of naïve empiricism, the representativeness heuristic and predicting from imperfect predictors because these problems have occurred by ignoring the original intentions of the test then applying statistics not entirely relevant to the data to make comparisons between different schools. The publication of cheating students’ data on the My School website illustrates the Fundamental Cognitive Error, naïve realism and selective processing because the Education Department wrongly believes these data won’t affect grade averages. Finally, the hindsight bias was applied to the possibility of the introduction of performance-based pay for teachers and the negative consequences of such actions. These points have all illustrated why the My School website is inaccurate and ineffective when it comes to rating and comparing a school’s performance.

 

References

Australian Curriculum, Assessment, and Reporting Authority (ACARA). (2010). My School: About ICSEA [Fact Sheet]. Retrieved from http://www.myschool.edu.au/Resources/pdf/My%20School%20FACT%20SHEET%20ABOUT%20ICSEA%2020100120.pdf

Changes proposed for My School website [Online exclusive]. (2010, June 22). Covenant Christian School. Retrieved from http://www.covenant.nsw.edu.au/

Chilcott, T. (2010, October 2). Results at Coorparoo State School will stand in spite of cheating. The Courier Mail. Retrieved from http://www.couriermail.com.au/

Chilcott, T., & Schultz, A. (2010, October 1). Cheating claims against Coorparoo State School principal Greg Kretschmann confirmed. The Courier Mail. Retrieved from http://www.couriermail.com.au/

Educational groups slam My School website [Online exclusive]. (2010, January 29). The Sydney Morning Herald. Retrieved from http://www.smh.com.au/

Government accused over My School website [Online exclusive]. (2010, January 27). Cath News. Retrieved from http://www.cathnews.com/

Hudson, P., & McMahon, S. (2010, January 29). Knowledge is power for all parents as My School website launches. The Herald Sun. Retrieved from http://www.heraldsun.com.au/

Independent Schools Queensland. (2010). Submission to the Senate education, employment and workplace relations committee: Inquiry into the administration and reporting of NAPLAN testing. Retrieved from http://www.aisq.qld.edu.au/files/files/whatsnew/ISQ-SubmissionNAPLANSenateInquiry(June2010).pdf

King, M. (2010, October 5). Teachers, tests and cheating: Where do we draw the line? ABC News. Retrieved from http://www.abc.net.au/

Levitt, S. D., & Dubner, S. J. (2005). Freakonomics: A rogue economist explores the hidden side of everything. United States of America: William Morrow Ltd.

Owens, J., & Lim, N. (2010, October 2). Principal stood down over NAPLAN tests. The Australian. Retrieved from http://www.theaustralian.com.au/

McAlpine, J. (2010). Shortcuts on My School website worry principals [White paper]. Retrieved from New South Wales Secondary Principals’ Council http://www.nswspc.org.au/index.php?option=com_content&view=article&id=35:shortcuts-on-my-school-website&catid=2:news&Itemid=21

Qld behind in literacy, new figures show [Online exclusive]. (2010, September 10). The Sydney Morning Herald. Retrieved from http://news.smh.com.au/

Set a national standard, but it must pass the test [Online exclusive]. (2010, September 5). The Sydney Morning Herald. Retrieved from http://www.smh.com.au/

Wu, M. (2010). Interpreting NAPLAN results for the layperson. Retrieved from http://www.appa.asn.au/images/news/naplanforlayperson20091022.pdf

Advertisements

Leave a Comment so far
Leave a comment



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s



%d bloggers like this: