Accountability in Education (Excerpt From Knowing The Learner)
The Problems of Accountability, Teacher Evaluation, and Merit Pay
From Chapter V of Knowing The Learner
In a collective social enterprise such as publicly supported education, all participants — teachers, students, parents, supervisors and administrators, government officials, and providers of educational resources — should be accountable for their actions. But what forms precisely should that accountability take for each of these stakeholders? The public mind today is focused primarily on teacher accountability. At the time of the writing of this book, teachers are being held accountable for the performance of their students on high-stakes, norm-referenced tests. Hiding in the shadows behind this practice is the assumption that we can know the effects of teaching on students’ test performance. Hiding more deeply is an assumption of causality — the assumption that teachers’ actions or lack thereof are a cause of the level of student performance on these tests. It seems straightforward that there should be some relationship between teachers’ actions and students’ attainment. The whole educational enterprise is founded on this assumption. But what can we soundly infer about the actual relationship?
A large number of factors would have to be considered to build any such causal equation. To begin with, there are the numerous human attributes that need to be taken into account (including student intention, motivation, interest, previous learning, learning habits, and cultural factors such as level of competency in language in which test is administered, and socioeconomic well-being). These constitute competing explanations for level of success on assessments and tests. Some of these (e.g., interest) might themselves be considered worthy goals for learning, often more worthy than stated course content itself. Many teachers take on the challenge of developing these as capabilities, but, of course, HSNR tests neither measure nor take these capabilities into account. Consequently, the exclusive use of such instruments reduces the evaluation of teaching and learning to improperly defined content and format.
Then there are questions of the effects of poverty/wealth, cultural enrichment/ deprivation, and sense of security/insecurity on student performance. Might such effects have as great or greater an influence on student performance than the experience of spending a period of time with a particular teacher? What proportion of these factors can a teacher reasonably be held accountable for? How can we tease out the effects of such factors?
We must consider the attributes of teachers as well. How long have they been teaching? What preparation do they have? What resources do they have available? What support are they obtaining from colleagues and from the school community as a whole? These would all have to be entered into the equation. Valid and reliable assessment and research instruments would need to be developed for each of these attributes and then applied and integrated into interpretations of students’ performance. The Institute of Educational Sciences sets a premium on funding research that employs randomized control trials (RCT) as a basis for inferences of causality in educational settings. But RCT studies struggle persistently, and unsuccessfully, with the problem of over-determination — that is, situations “…where a number of factors are present, any one of which could have produced the same result that is being attributed to the intervention” (Cook, Scriven, Coryn, & Evergreen, 2010).
In actuality, no school has the resources to conduct studies to permit the inference of causality. It turns out that the attempt to tease out these attributes in order to establish causal effects of a teacher’s action is not a feasible enterprise. In fact, it constitutes a monstrous pseudoscience. Therefore, on what basis do states and schools make the inference that the teacher or the teacher’s method is responsible for students’ performance on a test? The answer is on no sound rational or empirical basis at all! There is no way that research to establish cause (as a feature of instruction) and effect (as the performance on a HSNR test) could be carried out other than in highly restricted artificial educational settings. The cost would be too great, the conditions could not be met, and the inferences could be easily challenged on the basis of the argument that the results could have been attained without the method of instruction used (Klees, 2016).
But all such attempts at relating student performance to teacher performance neglect the most fundamental questions of all. Do the assessment instruments provide information on the attainment of targeted learning outcomes, those to which teachers’ efforts should be directed and for which teachers might truly feel accountable? Do any of the assessments used in educational programs measure what is most important (e.g., core capabilities)? Are they accurate and dependable measures of the learning goals that constitute the program’s aim? The reader should not be surprised at this point to realize that, for the most part, none of these conditions are met. Again, we are led to the sad realization that those who engage in these virtually omnipresent processes of establishing teacher accountability are resting on the cultural belief that HSNR tests are the best available way to measure the value of instruction. Once the pseudo-scientific attribution of causality is discarded, the entire justification for attempts by school districts, states, and federal agencies to manipulate educational programs based on HSNR tests collapses, its lack of foundation evident and its potential for harm exposed. Any thought of basing teacher evaluation or teacher accountability on such instruments should be immediately and roundly dismissed. Naturally, any proposals for merit pay based on HSNR test results must quickly be allowed to fall by the wayside as well.
The strongest basis for sound educational decision-making that we have at present is the considered judgments of seasoned, dedicated teachers based on their knowledge of their students and these teachers’ facility in working with targeted learning goals. Is it wise to lose these treasures in deference to policies and authorities operating in ignorance at a hyper distance from the teaching learning process?
Systematic educational assessment and community-based educational evaluation such as we have proposed in the preceding chapters give a sound basis for accountability in the context of professional development and program improvement. The most productive next step that we can take at present towards teacher accountability is a purely educational one. It is part of the realization of teacher as learner. It can be accomplished in the context of helping teachers learn how to use assessment information to help students attain the practical learning outcomes for their curriculum. As education is a social enterprise, teachers who cannot learn to help students attain the targeted learning goals that make up the curriculum must inevitably chose to leave of their own volition in the face of public reports of assessment results. Accountability of teachers and students (indeed, all facets of educational programs from the student to the state) are critical. But, first, the instruments and procedures of accountability must themselves be made accountable. They must show their efficacy in supporting educational activities and events.
Thank you Hunter for this illuminating perspective on the issues we tackled in our discussion of accountability in education. Establishing direct cause is something that even ‘hard’ science finds problematical. What we can do, as you have pointed out, is to obtain information on how well discrete learning goals have been attained. If we do this before and after an instructional sequence or program is offered we will have evidence relevant to the question of whether the instruction has been successful. If so, well and good, we need not look further into the nature of the instruction ( Although we should remember that success in attaining the learning goals is only supporting evidence and not proof that the instruction was the cause of the attainment). If the learning goals have not been attained, this also does not pinpoint the cause of the failure. The instruction may have been carried out poorly, or it may be that the instruction was not appropriate for the targeted learning goals. It may also be that the assessment was not appropriate to the learning goals or not properly carried out. Unless all of these potential causes for assessment results are ‘evaluated’ we can not properly assign responsibility to the teacher for the lack of success. In any case assessment results associated with the aggregated test scores produced by high-stakes, norm-referenced tests, contain no information on the attainment of discrete learning goals, and so do not provide the information needed to evaluate or improve educational activities. This is why we say in the book that assessment and evaluation must first be made accountable before we try to make teachers accountable for assessment results.
The above two quotes illustrate the concept that most struck me the first time I read through KTL, relating to accountability. At the beginning of the section entitled “The Problems of Accountability, Teacher Evaluation, and Merit Pay,” Paul and Will refute the claim that there is causality in education, saying there are a number of factors situated between the teacher’s communication and and the student’s reception of the communication that may be as important or more important than the communication itself, giving examples of student intention, motivation, interest, etc. This is extremely radical! We live in a society that thrives on causality and accountability – how can we justify paying taxes to support a school system which cannot show us that is doing its function, educating its students? The problem is not that there is no way to hold educational institutions accountable, however, only that we are going about it in the wrong way. This is where discrete learning goals come in. Discrete learning goals are the kernel of truth from which we can base a claim of causality. If developed with an eye towards simplicity and assess-ability, some of the noise is removed from the data, some of the chaos of schooling. Once reoriented towards discrete learning goals, we can again hold schools and teachers accountable in good faith.