Tuesday, October 2, 2012

On stopwatches, Fairness & Testscores: We Trained Them Well

Student: "Can you just give me the 'book' answer?"
Me: "We don't use a book in this class"
Student: "You are exasperating."

The @40ishoracle and I have had an running theme for the last few weeks. My approach from the student perspective and hers from the teachers. We have decided to write duelling blogs on the topic of "We have trained them well." Read her reflection on teachers here.

Vignette 1: Missed Opportunities

I want to start with one of my favorite teachers. He is a strong presence in the classroom who creates an environment where students feel comfortable exploring complicated questions of faith, justice, and religion. He is typically recognized by excellent students as one of the strongest teachers they have had.  He sees his calling as one of planting seeds, equipping students with thoughts and skills that may be used years down the road.

During lunch one day, he described a day in his classroom. A student was sharing a reflection, showing a video. The teacher said the context, presentation, and subject matter were visually and emotionally moving. Yet as he looked at the classroom, his students were disengaged. They were not being rude. There was no doodling, or sleeping, or gaming, or surfing. Just a strong detachment from the moment and the opportunity they had been given to experience something profound. It was disappointing for this teacher who has had a rough time with freshman this year.

Vignette 2: Claim Analysis Project

Students use collaborative notetaking to capture ideas
from discussion  complete with comments
One of the primary skills in the Digital Citizenship class (#digcit) is the development of information analysis skills. As the media becomes more and more biased and social media allows us to live in echo chambers of our own making (see the original post on the PLNs or the #theatershooting analysis), it is important for students to be able to dissect news and opinion pieces for grains of truth and boulders of bias.

Assignment: Students took two articles from the first issue of the student newspaper: one news, one op-ed. The students were given five minutes to read one of the articles and reflect on the following:
  • do you believe the information given in the article?
  • what is it about the article or the context that makes you trust (or distrust) the information?
Students were then put into groups of three to share their reflections. It was at that moment that I began to notice that this lesson was not going to go as planned.

One group, the most vocal and active, sat in a straight line. They were discussing, but clearly not engaged in the assignment. It was a factual exchange of information that scratched the surface. More disturbing was the group of students who sat silently with their heads down staring blankly at the news-article. -- silence.

When I prompted them to interact, "The next step is to share what you three are discussing...you might want to, you know, discuss", one of the students looked at me and said, "I don't understand."

"Did you believe what is being stated in the article?" [Task Comprehension]
"I don't know"
"Ok. What is your gut feeling? Accurate or not?" [Start at the basics and build]
"I'm not sure what you mean. Do you think the article is accurate?"
"Why does that matter? I want to see how you processed the information." [Technically I should have gone with another question, but I was getting a that tickly feeling teachers sometimes get -- this could be important]
"I don't want to be wrong. It is not fair for you to ask us a question that we do not know the answer too. You already know the answer. You are just being mean."

There it was. 
And two other students nodded.

The lesson went...ok. By the end of class, students were able to identify the difference between a CLAIM being made and the DATA or ANALYSIS that might show that a claim is indeed true. But that look of helplessness stuck with me.

A Reflection



This year's Freshmen were in kindergarten one year after I left the public school system. While there were a number of reasons I left (one of which was a starry-eyed dream that I would be a stay-at-home Dad and read comicbooks to my baby all day long), part of it was a growing dissatisfaction with test-culture. I taught remedial English to students who had already failed the state exit exam and where preparing to take it a second, third, or fourth time.

Students in the #digcit class due a sorting activity to
identify best sources for  research
I remember long discussion with my wife (then in medical residency) about the dichotomy of this teaching assignment. If I could teach them to think and communicate effectively, to provide a base-level of understanding, they would have a good shot at passing the exam, but there were years of apathy and a ton of factors beyond my control. However, if I went straight for the test -- drilled vocabulary, taught a formulaic writing system that graders (or auto-graders) would be able to checklist through quickly, taught ways to game the natural flaws in a testing system -- I was pretty sure most of the kids would have a good shot at passing.

I tried it both ways. At the end of the "test prep" semester, I was told by the administration that they wanted me to have a training session with other teachers on how I had taught so many kids so well -- 80% pass rate. That was the writing on the wall for my tenure in public schools

Vignette 3: The Magic Formula

A student working on an essay assignment had gathered all of the information for the paper. As students began writing rough drafts, a number of hands went into the air:
"How many sentences have to be in a paragraph?"
"Does the thesis have to be at the end of the third paragraph?"
"I am writing my outline first. What do you want to read in paragraph two?"

The communication aspect of the essay was lost. This was not informative or persuasive. This was writing by formula. As the questions continued and the answers were clearly not "8 sentences" or "Yes", the students began sharing their experience in other classes and at other schools. They described teachers who set arbitrary minimums as a quick way to encourage depth and critical thinking or a formulaic structure as a surefire ways to pass the standardized test (cold chills went through my spine). What struck me was that students had internalized the rules without understanding the reasoning -- the goal was to think deeply, not to churn out eight complete phrases.

We have trained them well

The mantra for the last few weeks as @40ishoracle and I have been discussing with teachers, students, and our twitter PLNs is, "We have trained them well"
  • We have trained students that there is one correct answer
  • We have trained students that the one correct answer is known by the teacher and/or the test grader
  • We have trained them that there is (usually) a simple formula for determining that answer - a process to solve the problem, a structure for an essay, a list of terms to memorize.
We train them by showing them a video that walks through the steps of solving a problem but does not explain why those steps lead to the solution. We train them when they are taught that a science experiment is following the step-by-step guide of a demonstration. We train them with worksheets, one-answer textbooks questions, and tests...lots of tests.

At the point that parents are hiring tutors to help kids with the testing in Kindergarten, the time has come to revisit the system. One of the businessmen that we work with has recognized this need as well. He puts it succinctly: "We are reaping what we have sown". 15 years ago businesses were calling for graduates who could communicate effectively, work well with teams, and work independently without a heavy hand guiding them each step of the way. 15 years later, the American Associations of College and Universities points out that things have not changed all that much:

  1. The ability to work well in teams—especially with people different from yourself
  2. An understanding of science and technology and how these subjects are used in real-world settings
  3. The ability to write and speak well
  4. The ability to think clearly about complex problems
  5. The ability to analyze a problem to develop workable solutions
  6. An understanding of global context in which work is now done
  7. The ability to be creative and innovative in solving problems
  8. The ability to apply knowledge and skills in new settings
  9. The ability to understand numbers and statistics
  10. A strong sense of ethics and integrity 
What has changed is our approach to education. In pursuit of these goals (or in pursuit of unstated goals such as cost-savings or opening the educational marketplace), we have developed a system that substitutes opportunities for reflection and teamwork with additional assessments. These assessments to not promote problem analysis or creative solutions -- they look for formulaic methods that give a single scorable answer.

Just read number 8 and think about the student who called lesson was "unfair".

When Yong Zhao presented to the educators and technologists at the  ISTE12 Keynote, he noted that while America obsesses over global test scores, the rest of the world is busy working on creating innovated geniuses: problem solvers, creators, communicators -- those things that have traditionally kept the U.S ahead of the technology and industry game despite high labor costs, lower population numbers, and bad test scores (remember, the US has never excelled in standardized testing worldwide).

But there is hope:

  • Math classes that give the answers to the homework but look to the students' process to arrive at that answer.
  • #Flipclass and No-homework programs that de-emphasize content transfer and emphasize the rich interaction that are capable in a classroom the individualizes and provides opportunity for collaboration
  • The growing number of #digcit and #byot programs that emphasize processing and problem solving as well as information literacy and communications. As these systems replace push-button training classes, students are exposed to more opportunities to identify and solve problems rather than follow a pre-selected path to a single answer.
  • History and Social Studies classes that are rejecting textbook summaries of events in exchange for time spent interpreting and analyzing history through primary source and historical documents.
  • Schools with a mission to educate and form the whole person rather than force an unnatural separation by subject areas.
But these hopeful techniques and movements are a gamble. Because if "success" is designated by a score on a standardized test, rather than the development of creativity and innovation, then we run the risk of denigrating the best programs in favor of "test-prep" environments. Worse, we are creating a system that measures students, rewards teachers, and honors school systems for creating students that do not have the skills that we want in either our well-informed citizenry or our innovative workforce.

Its time for the student-based-objective to focus primarily on the student.
Its time to focus on the race and runners and not the stopwatch.
Its time to stop training and start teaching.


4 comments:

  1. Hi JD- I really enjoyed this post. I agree with everything you say here. I'd like to add my perspective and see what your thoughts are.

    I agree that our current method of using multiple-choice, high-stakes testing is, at best, very crude and does not measure meaningful student performance. I think this is because, in part, the items are written for the lower levels of Bloom's (typically) and that we are not actually measuring the performances of true interest (your point about how a math problem is solved, for example, versus just getting the correct final answer).

    One of the things that frustrates me about the AACU standards is that they are so vague. "The ability to understand" is not an objective, in my opinion, because there is no description of the performance learners must demonstrate in order for us to know that they understand. I am happy for there to be a whole range of possible ways for learners to demonstrate understanding; the breadth and/or depth of the range that is appropriate would depend on the objective, of course. But the performance would have to be observable...an indicator of understanding, if you will.

    One of the things that frustrates me, often, is that teachers and other education professionals tell me that the stuff they think is 'really' important can't be measured. And I certainly agree that the important stuff is NOT being measured by the current multiple-choice assessments. But for me, as an instructional designer, measuring the hard stuff is an exciting challenge. What are the observable indicators of complex thinking? Of problem solving? Of creativity? Of imagination? They may well be different in every context, but I need to figure them out (or again, what the range of acceptable possibilities are) so that I have a way of evaluating the effectiveness of my instruction.

    If I were Queen for a Day I would eliminate the multiple-choice, high-stakes testing and I would implement ongoing (as in daily) formative assessment. As an instructor, I would have continuous data about what kids can do so that I could adjust my approach on the fly. And if I did give summative evals (as they would likely be required) they would be performance-based and criterion-referenced. But, theoretically, I wouldn't need the summatives because I have the data from the ongoing formatives, so I know what individual kids can do.

    So for me, the evals are not about "judging" kids....they are about providing ongoing feedback for every individual kid to the teacher. I have found, over the years, that teachers are not able to report accurately on what kids can do without data. It's just too much to keep track of in their heads and their intuition has not been demonstrated to be reliable. I look at evaluation data as helpful tools for teachers. Again, I'm talking formative assessment, so teacher still have a chance to modify what they're doing.

    I think you and I would agree that the current method of testing isn't good because 1) it doesn't measure meaningful student performance; and 2) when the data are collected it's too late for the teacher to do anything about it. I would agree that we should bag it. That said, I'm not prepared to dispense with assessment altogether because I think it's too valuable, if done properly.

    Thoughts? And thanks!
    Karen
    www.karenmahon.com

    ReplyDelete
    Replies
    1. I think you might be my favorite test-designer of all time. :)

      Your comments made me think back to this summer when we were listening to a BBW12 presentation on Analytics and I came to the realization that we were putting a lot of eggs into a basket that was still being woven (http://geekreflection.blogspot.com/2012/07/educational-analytics-idea-that-might.html)

      I agree that the AACU guidelines are not testable as listed (nor should they be, they are desirable employee traits). And I will certainly work for you when you become Queen-for-a-day if you can figure out how to give usable data on-the-fly to teachers...I might even want to do some traditional professional development if that model becomes a reality.

      That said, I think we are approaching the crisis point. The ends-based measures of our current test-prep culture are now having real impact on the development of skills of students. Far from being a benign measure, they are now influencing the observed object in some kind of twisted educational uncertainty principal.

      To add to your to do list:
      1. Stop putting high stakes on low hanging bloom's flavored fruit
      2. Make assessment a direct line of communication between the student and the instructor with feedback to students that improves their skill development
      3. Create alternative forms of assessments of instructors that does not include using numbers that were never designed to assess instructors (when you type it out like that it is just so silly)
      4. As testing instruments and analytics matures, implement it as appropriate to improve education as we (as a society) want it to be -- stop using it as a weapon to push a party line. stop using it as a wedge to force open a new market

      Everyone, feel free to join in the discussion...

      Delete
  2. I'm reading this on the day that I just realized that this year's freshmen will be the first to take the 'new and improved' high stakes test in their junior year, spring of 2015. And they're the least inquiry-driven, knowledge-seeking class I've known. One friday min day I gave them a wordsearch. It was like a sedative. I think it was the first day they didn't really have to 'think' in class. Paper pacifier. I'm getting really, really scared.

    I've flipped my Alg 2 class. The majority of them are off the hook. Only a handful of freshmen. But my all freshmen Geometry Honors (which, we need to revisit how some of them ended up there) and half freshmen regular Geometry class. Yikes!

    On the one hand, I'm glad I didn't flip Geometry as the regular class in particular, would have been a struggle, I'm pretty sure. However, they need the time to do the rich activities that flipping would afford. And time to pick up some remediation on arithmetic, algebra, etc.

    When I had a conversation this morning with dept chair re: my epiphany about this year's freshmen and 'the test', and it led to the talk of integrated math vs. traditional alg-geom-alg2 paths. She said 'How can we do integrated without a book?'

    I really think the book is doing me in, particularly in Geom. I'm starting to have some VERY radical thoughts. I'd already started having them. But now...again, what do I have to lose? Hmmm.

    ReplyDelete
  3. I just want to say thanks for your useful info about stopwatches..

    ReplyDelete