Assessing 21st Century Assessment: A Meta with Peeps - Happy Easter

• Industry and business sends up the call for increased problem solving skills, collaborative ability, creativity. 
• College professors complain "kids these days cannot write!" but what they mean is that students have difficulty connecting primary assertions with supporting evidence in a coherent fashion (more of the critical-thinking thing).
• The schools mission demands we cultivate and nurture students who are loving, open to growth, religious, and committed to justice as well as intellectually competent -- these things are what we should write assessments to measure. 

Have you read the common core recently?

Having exact answers, and having absolute rights and wrongs, minimizes the necessity of thinking, and that pleases both students and teachers. For that reason, students and teachers alike prefer short-answer tests to essay tests; multiple-choice over blank short-answer tests; and true-false tests over multiple-choice.

 But short-answer tests are, to my way of thinking, useless as a measure of the student's understanding of a subject. They are merely a test of the efficiency of his ability to memorize.

- Isaac Asimov, "The Relativity of Wrong" 1989

(Full versions available online - did not link due to copyright)

Those of you who regularly read both this blog and @40ishoracle's know that, probably because we work about 15 feet away from eachother, we tend to talk about the same things. So it is probably not surprising that assessment has been the topic of choice for awhile.

Not safe for assessment?
marshmallowpeeps.com

my  #savethedinosaurs post became a little moot this week since NYC DOE backed off of its proposed list of 50 banned words. While I am hesitant to beat a dead tyrannosaurus (yeah, i had been saving that one), what I have been thinking is that this is an example where a lack of assessment goals (or possibly the wrong assessment goals) leads to some really awkward decisions about testing guidelines.

If the primary purpose of an assessment is to measure the ability to solve a certain level of equations, link a topic sentence to supporting sentences, and follow the basic rules of grammar, then it is easy to say that an assessment tool can be create which avoids uncomfortable topics like politics, evolution, and candy-peeps (mmmm...peeeepps).

Context: Computer Applications and HTML programming

At Brebeuf Jesuit, all freshman are required to take a digital citizenship course called "Computer Applications" (Course titles change about as fast as the Titanic could turn). One unit of this course is an introduction and overview of HTML programming. As the teachers discussed this unit in light of our new focus on "civilizing the digital natives", we began to unpack the assumptions in this topic.

• HTML programming gives students an opportunity to experience the rigor and close attention to detail that is necessary for the discipline (and useful across the curriculum).
• HTML programming, at a basic level, makes for more efficient blogs and other web communications that will become used more and more w/o relying solely on WYSIWYG editors. It is a communications imperative (and looks great on that intern job application).
• HTML programming gives students an idea about what it takes to build the webpages that they interact with everyday and this affective appreciation is helpful.

Looking at these objectives, a traditional assessment (multiple choice or matching to identify <tags>; a "revise the miswritten code" section; defining key terms like "attributes", etc) would not be able to successfully measure a student's ability beyond a basic understanding of the second objective.

Experience: Designing an Assessment Experience for HTML

Goals:

• Practical application of tags, attributes, and common processes such as links, formatting, and image control
• Use of critical thinking including disciplined attention to detail, consistent rechecking of method to results, interpretation of instructions
• Appreciation of the code that is working in the background of webpages

Assessment: 

Instagram of Old-School Programing

1. Students will complete a webpage based on instructions given in written from by the teacher. The written form will be a combination of text that should be replicated and highlighted instructions of how to format the text. for example:
   

   The Font of most of the text on this webpage is [Describe the color HERE]. Text should contrast with the background or it can be hard to read. [the word “Contrast” in the last sentence should be a link to this website: http://www.colorsontheweb.com/colorcontrasts.asp].

2. Student may use the web, notes, tutorials, etc. But the page must be created in NOTEPAD
3. Students will have one day to work on the test. At the end of the period students will have an opportunity to save progress to date and told that they are NOT to work on the exam at home (paper is collected) but that they may study and practice anything they would like.
4. Students turn in the completed (or nearly completed assessment) on the end of the second day

Reflection:  How well we have trained them not to think

"This was by far the hardest test we have had all year", remarked one student after the test.

As I have already been accused of being the hardest teacher in the school multiple times, I decided to pursue:

• "With a math test you are just following the steps until you get the answer and then you are done."
• "In English, even if you have to write, you just write about what you talked about in class. You don't have to constantly think about it."
• "This wasn't a test. You couldn't guess at anything. If it was wrong you had to figure it out."
• "I can't believe we have a test where you have to study in between the test. Heck, I had to study during the test!"

The students were thrown off for a variety of reasons, but primarily it fell into two categories:

 Outcome/Feedback Check: 

Web on left; Code on Right
Refresh to check

In designing a webpage, you typically (in notepad or in design-software) have your screen split to view the output on one half and the coding on the other. During the testing situation, the students did the same. They immediately knew if the coding was wrong because the output was incorrect. This would be the equivalent of completing a math problem, and immediately being told by the teacher it is wrong and should be corrected. Immediate feedback is disquieting for students used to the delayed feedback of the grade ("I totally bombed that test, but at least it is over").

Technology affords educators the opportunity to design assessments that are closer to the real world. There is seldom a final exam in a career-path job. Projects are assigned, drafted, submitted for feedback, returned, revised, given to colleagues. This process is essential to the collaboration and problem solving called for by the industrial complex, but is absent from traditional assessment.

 Assessment simulating Life, an open book:

In addition to the issue of immediate feedback, students had every resource available to them to complete the task. They had been given a week to work with simulations and had those same simulations available during the test and during the overnight study period. Deep down in dark places where excuses come from, each student knew that there was little reason to not be able to complete the exam perfectly. This created a lot of resentment and complaint. Indeed, the students who did the worst on the assessment instrument were not ones who did not know how to code; they were the ones who did not research in the intervening night, did not carefully check work, and did not have the care to complete the task.

Conversely, it also was fascinating to watch the thrilling smile creep across a student's face as she realized by poring over sample code and comparing it that her closing tag was in the wrong location. Like the feedback loop, there are few situations beyond traditional testing where you cannot refer to some other resource to help verify an answer or procedure. Knowing when you need to seek assistance may be more important than knowing the information initially.

Conclusions

Technology is only one of five 21st Century skills 

Our challenge than, as educators of the digital natives, is to take advantage of the technology and this critical moment in education to create a new kind of assessment. By clearly identifying the skills and knowledge that we expect students to have (OUTCOMES) and designing environments that demand access to that knowledge and use of those skills (ASSESSMENTS) then we are well on our way to transforming education.

The key becomes defining those outcomes, through discussion and collaboration (and maybe even student involvement - scandalous) in such a way that it acknowledges the broad strokes of 21st century skills and does not become mired in the details which, ultimately, can be looked up on Wikipedia.

Assessments grow out of those outcomes and, here again, we must be on guard to not sacrifice the big picture (career skills including flexibility and adaptablity; Learning skills including problem solving and systems thinking, etc.) for the easily scored details that should be the tools by which skills are demonstrated.

Because if they can think, let them eat Peeps.

#savethedinosaurs - What NYC DOE's attack on the Terrible Lizards reveals about Education

UPDATE: The New York DOE has backed off of its banned words list, relegating my #savethedinosaurs campaign to pithy quotes of the past. But feel free to read on for some nice hypothetical discussion. -- adding a few updates from social media comments too.

If you haven't followed, heard about, or laughed out loud at the latest educational gaff from a Department of Education, you can read about it in a CBS news report or any number of blogs like this one. In short, the New York City DOE has proposed a list of 50 words (and concepts) that should not be used on standardized test for fear of their potential to be upsetting or create an unpleasant testing environment.

I sat on this for a few days...ok, I tweeted about it, but i refrained from blogging for a few reasons. First, I was infinitely hopeful that NYC DOE would back off of the statement (They haven't. Originally, they were attempting the "but Florida does it too" defense that is so popular among my seven year old's Sunday school class). Second, there are much better general education writers out there (looking at you, Answer Sheet), and I wasn't sure if my voice had much to add to the discussion. Third, I was really struggling on the best format: I considered a McSweeney's style letter to the Dinosaur, a classic rant decrying the downfall of education due to big government short-sightedness, etc.

So why write?

1. I think #savethedinosaurs is a meme that really needs to take off
2. One thing that i haven't seen written yet is the intersection between this testing mandate and our current reliance on testing as an evaluation tool for schools, teachers, and students alike.

For lack of a witty subheading -- Seriously?

I am not going to include the full list in this post (the links above do a good job of listing), but I am willing to call out some things that I haven't seen mentioned yet (or at least not enough).

Extremely Sensitive: The initial claim is one of sensitivity -- not creating an unpleasant testing environment (Florida generally avoids the use of "hurricanes" for fear of PTSD triggers).Yet there are a number of items on the list that seem to be following this guideline in only the most general of terms:

• Birthday celebrations (and birthdays)
• Computers in the home (acceptable in a school or library setting)

While I understand that there are religions that do not celebrate another-year-around-the-sun days, I am hard-pressed to find evidence of a birthday-related question triggering a stress-event that skewed a test (in fact, the only mention of this situation during my search was in reference to the NYC Banned Word list). It is common in our society today to base policy on the hypothetical rather than the actual (closely related to Generalization from Fictional Evidence, but if you know what the actual fallacy is, please let me know in the comments). 

While I understand the need to be boy-scout-level prepared in as many situations as possible, we do ourselves a disservice to create general policy to cover every situation...better to create a general policy that, through care and sensitivity, can allow exceptions (the Jesuits refer to this hallmark of effective education as cura personalis -- an understanding of student context, caring adults who are able to be flexible in the moment, etc. (Note: this is not en vogue in education practice today).

Not-So-Hidden Agenda

• Bodily functions
• Celebrities
• Junk food

There is always a risk when we mix assessment instruments with social-norm change goals. Again, I have trouble visualizing the PTSD moment from being asked to answer a question about hotdogs and potato chips (or the ever-popular porkrinds). At the point that we move away from the sensitivity claim, it seems that we have a number of items that are taboo because in an ideal world, we wouldn't have to deal with them (celebrity culture, overweight children due to malnutrition and overeating, fart jokes, etc.).

This becomes a slippery slope. Have we as parents and/or educators abdicated our duty of what should and should not be open for analysis and discussion? Arguably some of these "social" issues would be better served by some analysis and critical thinking. 


Note: one discussion in the Teacher Resource Center, points out that it could be even more insidious -- how difficult does it become to criticize the government when you can't discuss natural disasters (#FEMAFail), Politics, or War. Far from creating the informed citizenry, this list specifically encourages us to avoid assessing the ability to critique the government.

Good Test Practice: Even some of the legitimate test-avoidant topics:

• In-depth discussions of sports that require prior knowledge

are not really there because of cultural awareness/sensitivity. They are there because calling on specialized knowledge in a general question is a bad idea. So is writing a multiple choice question with one answer that is particularly long or a fill-in-the-blank that has the answer embedded in the question. Rather than giving a comprehensive list of words.verboten, the NYC DOE would be better off making the statement "write a good test".

Aside: this particular topic led to an excellent dinner table conversation hypothesizing "bad questions" based on this concept. My favorite was the ten-year-old's analysis that a question referring to pre-cellphone responses (contacting someone, emergencies, etc) would be impossible for most kids to answer.

On Backward Design - The hidden curriculum message in banning the Dinosaurs

Early in my teaching career, there was a battle being waged over the amount of standardized content that was appropriate in a classroom. Should every English teacher set his own reading list? are there a few must-reads and some flexibility in the rest? Is each book and each unit designed as a whole? -- While this argument has been settled in some schools, it is one of those classic discussions that appears to cycle about every 3-5 years.

As Wiggins and McTighe's Understanding by Design (sorry, no KINDLE link) took off in schools, the system began to weigh in on how we shape assessments and understand what is taught in the classroom. It is a popular framework, but by no means universal. In way-to-brief summary:

1. Identify the outcomes for the individual learner.
2. Identify the evidence that would constitute acceptable demonstration (mastery, etc.) of those outcomes -- commonly, this is the assessment instrument
3. Design lessons, activities, and experiences that lead to effective demonstration.

• In its most negative interpretation, this is "teaching to the test".
• In its most positive reponse, it is retorted "of course. If the objectives are sounds and the assessment is good, why wouldn't you?

But what if the assessment is not good?

The corollary of this mindset is that there is little-to-no reason to teach content that is not going to be assessed, because it is extraneous to the outcomes being worked toward.

• Why teach a lesson on junk food and nutrition if it will not be assessed?
• Why address issues of social injustice if there will be no questions about poverty, homelessness, or even computers that are owned by a family instead of the library?
• Why teach evolution if it has been banned from the assessment?

Teachers have been told (particularly in NYC) in no uncertain terms that the results of these test-scores are direct reflections on their ability as teachers and that this ability will be publicly pronounced in the virtual town square for all to see and comment upon. At the point that I have been told, specifically, that this content will not be a part of high-stakes assessment, the only responsibility I have to teach this content is my prerogative as a good educator -- and the DOE and legislatures have done a pretty good job of telling me, again and again, that this is not enough.

And so we are now to prepare students for a test that does not encourage critical thinking, does not ask for analysis of topics which may be deemed controversial or uncomfortable, and which puts forth a multi-faceted social agenda.

This goes beyond critical correctness.

And the dinosaurs will not the only casualties.

--
As always, if you made it this far, I appreciate your thoughts, comments, likes, +1s and retweets. If you are so inclined to share, please include the #savethedinosaurs hashtag, because I think that would be cool :)