On Assessing Technology in Projects: Avoid the Blind Spot

Yesterday, Brebeuf Jesuit hosted its 1st annual Jingle Bell Tech Conference. Four schools from around Indianapolis came. It was a small gathering but had great content and discussion, which is about all you can hope for this close to Christmas. Special thanks to @40ishoracle who conceived of it, did the planning and ran the show and @tcockrum who once again played guide about the #flipclass.

The topics covered included Google Apps for Education, Adapting a classroom for BYOT, Flipping your classroom, and Assessment strategies. Any of these topics are worth a few blogs, but, as is normal, the best part of the conference occurred when teachers began to discuss and share, in this case, on the subject of assessments.

Assessing Technology skills is difficult on a number of levels. This is highlighted in an age where many schools are doing away with “tech” classes in favor of an integration strategy. Even schools that have a technology focused class have raised the expectations that all teachers will integrate technology and, presumably, evaluate it. Some of the difficulties:

• Students are more comfortable with technology so they are able to “dazzle” the teacher with effects that are not really all that difficult or representative of actual skills.
• Students are able to (as one teacher put it on our ever-present chalktalk poster boards) “pull one over” on the teacher by blaming technology for procrastination or failure to implement.
• Teachers feel unqualified to grade “tech” but obligated to because of the ever-present specter of expectation
• Students replace learning material through repetition or practice with technological shortcuts so that the "shiny pretty" obscures the assessment of learning objectives

This mix of vague expectation, separation of skill from learning objective, and general dis-ease with the topic leads to the creation of Evaluative Blind-Spots. for the purpose of this rant, Blind-Spots are evaluation categories that are vague, generalized, and often skew the overall grade positively without necessarily assessing any actual learning objective.

In our experience, they often use terms like “creativity” or “use of technology” or “overall”.

Not sure if something is a blind-spot? ask yourself these questions:

• Does the category come at the end of the rubric?
• What is the link between this category and the content/skills being evaluated overall?
• Do i feel qualified to evaluate this category?
• Am i willing to drop someone a letter grade because of this category? Two letter grades?
• On a five-point scale, can i visualize what a “1” would look like? a “5”?
• Have i ever assessed someone “below average” in this category? is it just free points?
• If i ignored this category, could i still assess the primary learning objectives?

If the questions above indicate that you a) don’t really care about this category or b) are not sure how to assess this category objectively (yes, i said it), then eliminate it as a part of the evaluation. 

You will save yourself time and stress and let the students focus on the sections of the rubric that are related to the primary learning objectives. You also eliminate the possibility that the category is just a placeholder for your opinion on the quality of the project and avoid the inevitable comparison of grades and discussion of your “fairness”. (This is not to say that the teacher opinion shouldn't be factored, but that it is already being factored in categories such as “organization”, “clarity”, “mechanics”, “research”, “analysis”, etc -- poor quality should be declared as more than an "opinion").

You might find that the blind-spot is a misnamed category for something else. “Use of Technology” might be a place holder for “ability to communicate a message” or “aesthetic design”. Those are more exacting terms that might be easier to expertly evaluate. after all, that awful “keyboard type” transition in a PowerPoint is a “use” of technology, but three slides of it and the only thing that is being communicated is death wish. tat-tat-tat-tat-tat.

Is a blind spot the end of the evaluative world? probably not. few students ever complained about easy points. but if our ultimate goal in assessment is to give feedback on accomplishing specific learning objectives, demonstrating skills, showing acquired knowledge, then we owe it to ourselves and our students to measure that.