Paul Zachos
Forum Replies Created
-
Posts
-
Hi Patrick,
I like the way Monica selected portions of your review and commented on them. I will do the same. Also, I like the fact that you start with an expression of your feelings concerning your reading. It gives an added flavor to the review.
In general my reaction to the criticisms that you have leveled is that they are valid and point to substantial improvements that could be made in the book. These should be considered if anyone wishes to update the book or carry its points further
Let’s start with:
“I have no idea what the chart on page 17 was supposed to mean and it just confused me.”
This confusion is entirely my doing. I just wanted to give a flavor of what someone would find in Johnson’s Intentionality in Education. I must admit I have a tendency to place puzzles, anomalies and mysteries in the things that I write. My co-author Wil Doane is innocent regarding this particular example.
________________“What is the example in “Make it Stick”? I understand it has something to do with a hunter and a doctor, but don’t see how that applies to the discussion of skills and how they relate to concepts.”
I am afraid that the whole section on the distinction between concepts, skills and dispositions is too highly abbreviated. A book could be devoted to this distinction and its significance for curriculum. The references to Brown et al (“Make it Stick”) and to Inhelder and Piaget are meant to lead the reader to works that have addressed the point in greater depth. Waldorf Educators have a simple, concrete way to make this distinction; they talk about head, hand and heart, respectively. The important point is that if we concentrate on only one or two of these two types of capabilities, we are not educating the full person. In general ,our mainstream educational institutions focus on learning goals that target knowledge (concepts), less so skills and only very slightly, dispositions. This is particularly true regarding the assessment of these capabilities. Many new initiatives are appearing in the field of educationfocused on social emotional learning are targeting what we call dispositions.
_______________________________
“Why do we use learning goals instead of intended learning outcomes? This chapter specifically touched upon the importance of precise and consistent language.”‘Intended learning outcomes’ for me gets the idea across effectively. But it is too much of a mouthful and has a very technical feel to it. ‘As long as the idea of ILOs is grasped then ‘learning goals’ serves as a precise equivalent in ordinary and familiar language.
________________________
.
“Artistic aptitude is listed as an innate trait, which I wholeheartedly disagree with. Artistic aptitude is more of a learned skill, just as much as aptitude with typing or open mindedness. One could make an argument that there are no traits beyond the strictly physical. Is extroversion not a combination of valuing social interaction, habits of being talkative and open, etc… You mention that there are no hard and fast boundaries, but I would argue that there are no real boundaries at all. Looking over this I would not be able to define whether something was a trait or disposition.”I see enough problems with our presentation to not want to push the case further at this point. Thank you for the clarifications you have offered.
________________________
“The example with cubes and liquids helped me understand concepts, skills and dispositions more after initial confusion.”
Yay! Cubes & Liquids saves the day again!____________________________
“Instruction can be thought of the artistic function in education” – According to the earlier statement, wouldn’t that make an aptitude for teaching be a trait rather than a disposition?
More problems with our distinctions. Thank you for pointing them out.
___________________________________“Education is not what teachers do, it’s what the student learns” – I like this quote a whole lot, it’s one of those short phrases that really packs a punch and gives a great overview of what we are doing.
To be more precise I would say that education IS what teachers do, but only when they are acting to help students attain learning goals. ‘What the student learns’ is the outcome of education but only when educational activities have been successful. Thank you, Mauritz Johnson, for helping me to see how to think clearly about these things.
____________________________“I would reorder “The Contexts of Educational Activities” to Settings, Programs, Activities, Events since that more closely matches their relations. The current order seems mismatched”
I would like to learn more about why you find the original order mismatched.
____________________________
The Idea of organizing the review around Feelings, Questions and Problems, as you did has been very productive from my point of view. I would like to encourage its continuation.
Hi Patrick,
If more people approach Knowing the Learner with the seriousness, thoughtfulness and insight that you have contributed I believe that we will be building a strong foundation for constructive change in educational practice.
Your review has definitely led to improvements in how I think about and give expression to the underlying issues that we are concerned with. For example Your discussion on community illustrates a basic principle that we value but have only lightly mentioned in the book — the human system dynamics that operate in educational communities when assessment and evaluation are carried out. I have recently been learning more about this through the work of Patterson, Holladay and Eoyang
https://smile.amazon.com/Radical-Rules-Schools-Adaptive-Complex/dp/0615766269/ref=sr_1_2?crid=1AXSIL7BZVVIJ&keywords=radical+rules+for+schools&qid=1562013492&s=books&sprefix=radical+rules%2Caps%2C135&sr=1-2Patrick,
Thank you for your extensive follow up comment. I have found it valuable to review what you have written both to evaluate what we have presented but also to have a peek into how a person who is looking into our work for the first time might react.
The big message that I take from what you have said is that proposals like accomplished based grading are interesting but what do they actually achieve when put into effect? This is indeed the critical question to be posed once one the basic concept has been understood.
I look forward now to reading your comments on the second chapter.
Paul
Thank you, Patrick, for this sincere and thoughtful review of Chapter I.
Your review of the 1st chapter succeeds in bringing to the fore many of the most critical issues and persistent problems that dog the field of educational assessment and evaluation, precisely the issues and problems that Knowing the Learner (KTL) is directed to address. I believe that as you progress through the book you will be able to come up with sound and productive answers on your own, so for now I will concentrate on points you have discovered that we may not have addressed adequately or sufficiently in the text.
Accomplishment Based Grading, and its radical thesis concerning the divorce of grading from assessment dwells primarily, as you point out, in the realm of ‘can be’ rather than ‘is’ or ‘does’ although is viability can be demonstrated in practice in at least two cases — Robert Pavlica’s High School Science Research Program and our own course, EA&E online, which teaches the fundamentals of educational assessment and evaluation. The changes in thinking and practice that would be required to put accomplishment based grading into effect in conventional educational settings is formidable and that, I believe, is why we do not see more of it. However the critical point to keep in mind is that the existing grading systems are not actually productive in serving truly educational purposes and so turn out to be both ineffective and inhumane. Something else is needed. Accomplishment Based Grading points the way to the requirements for rational, efficient and humane practices in educational assessment and evaluation.
Regarding the assessment activity Cubes and Liquids, you pose the question of what happens if the student does not engage fully in the activity, or perhaps does not make any effort at all. This is as deep and far ranging a problem as one can imagine and is worth a chapter or a book in and of itself. However, if I am correct, it does not have to do specifically with Cubes and Liquids but is a question that can be posed regarding any task whatsoever assigned in an educational setting.
Regarding the application of the notion of the ‘Bell Curve ’ to understanding what is happening with teachers and students in the activity of Cubes and Liquids we must consider another central thesis of our work. This is our requirement that the use of information in educational programs be educationally productive. The nature of this requirement will be defined precisely in Chapter II. For now consider the following question — How does the application of the notion of the bell curve help us to achieve an educational purpose in the context of the Cubes and Liquids activity?
Your questions concerning the science research course, e.g. ‘How do we ensure that students are setting realistic goals that are both attainable and require effort?” and what if students are “consistently meeting goals but at the end of the semester had not completed the project due to scope of project, improper goal setting or other problems.” These would be legitimate concerns in almost any conventional instructional program. Significantly, and very much to the point, these are not problems in the science research course as designed and realized by Bob Pavlica. This is because of the intensive assessment and evaluation that characterizes the course. Student work is continually being ‘observed’ and what is observed is used as the basis for moving forward in a way that does not allow any student to ‘slip through the cracks’.
Returning to Accomplishment Based Grading, you say
“By adding some information on how this process improves student outcomes, I think it makes the information much more compelling for non-educators. At the same time, providing this information would allow educators interested in this change to more easily get parent buy-in.”This is precisely our intention, and so the fact that you need to point it out suggests that we did not present it as clearly as was needed. So how does accomplishment based grading relate to improving student outcomes? Built into accomplishment based grading is the idea that the justification for the assignment of any instructional activity is that engaging in that activity will lead to the attainment of intended learning outcomes. This is an essential feature underlying accomplishment based grading as it relates to the improvement of student outcomes.
But in the end this is, as Jean Piaget used to say, ‘an empirical question’. We believe it ‘can’ do so, but ‘will’ accomplishment based grading actually lead to greater effectiveness and humaneness in educational programs? How do we demonstrate that accomplishment based grading can help to ‘improve student outcomes’, and have its other intended effects. Later in Knowing the Learner we address how to lay the foundations for educational research that could begin to answer just such a question.
Cubes & Liquids (C&L) may be used by teachers to support their daily practice. It is also sufficiently refined that it may be used as a research instrument. For an example of the latter, we are now using C&L to develop concepts and practices relevant to validity and reliability of educational measures working at the level of practical learning goals and outcomes. Teachers may also be seen as engaged in research when they are systematically building or refining assessment instruments.
The questions associated with daily practice vs. research are distinct.
For daily practice the primary question is, “How well are my students performing with regard to some learning goal?”
For research the questions are more of the nature of, “How confident am I in the results of this assessment? How valid are they, how dependable? What is the likelihood that the judgment made (i.e. of level of attainment of the learning goal) is accurate?”
In our sister conversation [Interpreting Educational Data] a request was made to see the comments made by raters regarding their judgments of level of attainment of a learning goal. This is part of evaluating the accuracy of judgments. For research purposes then it makes sense for raters to be generous in their comments (i.e. in giving reasons for their judgments) and for all stakeholders in the process to be attentive to those reasons.
It can indeed be considered remarkable that five individuals would observe a complex human behavior (in this case a student’s attempt to accurately depict an experiment) and make the same judgment as to the student’s capabilities. Yet this reliability of judgement is exactly what every educational measurement aims for. The fact that there are only two levels of attainment on this learning goal makes it easier to achieve agreement than it would if there had been more choices for judgement. But still the probability of 5 such judgments coming to agreement entirely by chance is small and would be likely to happen less than 4% of the time. [The probability of selecting one of two items strictly by chance is .5, and of doing so again is .5 x .5. For five judgments it would be .5 x .5 x .5 x .5 x .5 = 0.03125.]
Perfect agreement provides evidence that our assessment instrument is working as we would like it to. Disagreement, as in the case of judgments regarding Roland Questevarn’s response, can provide an opportunity for improvement of some aspect of the assessment or evaluation processes. Discussion on improvements of this kind take place in the thread devoted to ‘Evaluating an Assessment Instrument’.
Attached find the reliability report enhanced with rater comments that was requested in the previous post.
These questions (and there are several) are all good introductions to major themes addressed in Knowing the Learner. I will give some general answers here, but also direct the reader to portions of the book where these issues are addressed more fully.
The reader begins by referring to the fact that when outcome information concerning distinct learning goals is mixed together, typically by aggregating performance scores to get a total score, that the meaning of the original goals is lost and one cannot use the aggregated score for an educational purpose. While this is a good general rule of thumb, it is necessary to recognize that there are instances where aggregation across discrete learning goals can be helpful. One of these is presented in the section of Knowing the Learner called ‘Is Aggregation Across Diverse Learning Goals Necessarily Inappropriate’ (pp. 45-50)
Regarding the question of grading, please consider that we have not chosen to oppose the idea of grading in educational programs. Rather we try to make explicit the conditions when grading can be and those were it is not educationally productive. The whole first chapter of Knowing the Learner can be considered a critique of the grading paradigm, and yet we conclude this chapter with a section called ‘The Redemption of Grading — Accomplishment Based Grading’.
