On Assessing Technology in Projects: Avoid the Blind Spot

Yesterday, Brebeuf Jesuit hosted its 1st annual Jingle Bell Tech Conference. Four schools from around Indianapolis came. It was a small gathering but had great content and discussion, which is about all you can hope for this close to Christmas. Special thanks to @40ishoracle who conceived of it, did the planning and ran the show and @tcockrum who once again played guide about the #flipclass.

The topics covered included Google Apps for Education, Adapting a classroom for BYOT, Flipping your classroom, and Assessment strategies. Any of these topics are worth a few blogs, but, as is normal, the best part of the conference occurred when teachers began to discuss and share, in this case, on the subject of assessments.

Assessing Technology skills is difficult on a number of levels. This is highlighted in an age where many schools are doing away with “tech” classes in favor of an integration strategy. Even schools that have a technology focused class have raised the expectations that all teachers will integrate technology and, presumably, evaluate it. Some of the difficulties:

• Students are more comfortable with technology so they are able to “dazzle” the teacher with effects that are not really all that difficult or representative of actual skills.
• Students are able to (as one teacher put it on our ever-present chalktalk poster boards) “pull one over” on the teacher by blaming technology for procrastination or failure to implement.
• Teachers feel unqualified to grade “tech” but obligated to because of the ever-present specter of expectation
• Students replace learning material through repetition or practice with technological shortcuts so that the "shiny pretty" obscures the assessment of learning objectives

This mix of vague expectation, separation of skill from learning objective, and general dis-ease with the topic leads to the creation of Evaluative Blind-Spots. for the purpose of this rant, Blind-Spots are evaluation categories that are vague, generalized, and often skew the overall grade positively without necessarily assessing any actual learning objective.

In our experience, they often use terms like “creativity” or “use of technology” or “overall”.

Not sure if something is a blind-spot? ask yourself these questions:

• Does the category come at the end of the rubric?
• What is the link between this category and the content/skills being evaluated overall?
• Do i feel qualified to evaluate this category?
• Am i willing to drop someone a letter grade because of this category? Two letter grades?
• On a five-point scale, can i visualize what a “1” would look like? a “5”?
• Have i ever assessed someone “below average” in this category? is it just free points?
• If i ignored this category, could i still assess the primary learning objectives?

If the questions above indicate that you a) don’t really care about this category or b) are not sure how to assess this category objectively (yes, i said it), then eliminate it as a part of the evaluation. 

You will save yourself time and stress and let the students focus on the sections of the rubric that are related to the primary learning objectives. You also eliminate the possibility that the category is just a placeholder for your opinion on the quality of the project and avoid the inevitable comparison of grades and discussion of your “fairness”. (This is not to say that the teacher opinion shouldn't be factored, but that it is already being factored in categories such as “organization”, “clarity”, “mechanics”, “research”, “analysis”, etc -- poor quality should be declared as more than an "opinion").

You might find that the blind-spot is a misnamed category for something else. “Use of Technology” might be a place holder for “ability to communicate a message” or “aesthetic design”. Those are more exacting terms that might be easier to expertly evaluate. after all, that awful “keyboard type” transition in a PowerPoint is a “use” of technology, but three slides of it and the only thing that is being communicated is death wish. tat-tat-tat-tat-tat.

Is a blind spot the end of the evaluative world? probably not. few students ever complained about easy points. but if our ultimate goal in assessment is to give feedback on accomplishing specific learning objectives, demonstrating skills, showing acquired knowledge, then we owe it to ourselves and our students to measure that.

Dreaming Big: The Software I need for my "flipping" classroom

This one has been brewing in my head for awhile which means that it will probably go long, but i think the time has come for educators to start laying out what is necessary to take classrooms from analog to digital...the legislators had their chance and they give us contradictions like creativity-through-standardized-testing. In the spirit of the season, Bah Humbug.

Context: One of the best presentations I saw this year at ICE (Indiana Computer Educators) was a session on FLIPPED CLASSROOMS put on byTroy Cockrum (@tcockrum), Brian Bennett, and Brett Clark (@Mr_Brett_Clark). In this session, they outlined a simple method of teaching that could be considered innovative in some ways and a refinement of good teaching practices in others. Rather than going into details on “what is a flipped classroom” I will refer you to the livebinder @40ishoracle and I have been sending people to this year, my pearltree on the topic or suggest you follow @jonbergmann.

In summary, the #flipclass idea centers around displacing traditional classroom activities, including lecture, to the “homework” timeslot, freeing up that time period for one-to-one check-in, discussion, clarifying, activities, etc. In its simplest form, the teacher becomes available to “help” with homework problems that classically deterred students from doing their work.

This brings me to software: while it is a misconception that numerous flipclass teachers write about to say that it is all about “video taping the lecture”, when you think about the types of activities that can be displaced (reading chapters, listening to lecture, walking through a power point, taking notes), lecture pops to the top of the list in terms of what can be done.

There are a number of software solutions out there for recording screen casts, teacher video, etc. Many of our teachers have experimented with paid (like Camtasia) and free solutions and this is not an analysis of those.

Instead, i became intrigued by thinking about what software would look like if it were specifically designed to help the flipping teacher (i love the adjective wordplay). As is the case with most things, if there is an education need, there are companies who are working to fill it. I got to meet with Senthil Premraj of www.flippedlessons.com. While this company is not yet creating the software that i am about to describe, it was a phenomenal discussion and i encourage people with experience in flipping classrooms to talk to them about what you would like to see.

So here it is...My dream software for supporting the flipped classroom:

1. Web based (and only web-based): If I have a browser, it should be able to run this software. If I have a webcam or a microphone, it should be able to interface. Test it on a chromebook.
2. The ability to import, upload, embed from a variety of resources. There are too many effective videos and demonstrations out there for teachers to not have access to them. We are experts in curation, but the mechanism to bring a video into my flipped lesson should be seamless and simple
3. One Stop Video Creation with simple editing. I do NOT want to export to edit. I want one tool and one screen. So what would this entail?
1. A simple drag and drop timeline. that can include videos that we have uploaded and embedded, videos that we have created through a webcam.
2. A separate audio-only timeline for doing voiceovers.
3. A third timeline for captions, callouts, etc.
4. Want to win the special prize? give me a fifth timeline to do sideXside or picture-in-picture.
4. Teacher control: Teachers pace their lectures, display a variety of media, choose when to joke and when to be serious. Canned lessons are bad out of the textbook and bad off the web. Let the educator do the work. (yes, i heard the gasps. I am not a total liberal)

[Part of me understands that i am shooting for the moon here, but this would be the service that, as a school i would pay money for.. Besides, aviary.com already figured this out for web-based podcasting.]

5. The ability to export the final “movie” product. Teachers should be able to keep the fruit of their efforts even if the school stops paying for the product.
6. a simple mechanism for giving fair-use copyright credit. we need to be better models of this for our students. (this would be a simple piece to add to the upload/embed dialog. Where did this come from? what date? and then create a credit slide at the tail-end of the “movie” or a credit page that can be clicked on).

So on the teacher interface side, we are looking at a simple layout with timelines, audio/video bins, a callout/caption interface, a question option interface (see below) and some well-designed dialog boxes.

On the student side,

7. A system for inserting questions as a “listening check” for students. Bonus points if there is a self correcting option and if the teacher can get feedback so he/she knows going into class where students struggled with the lesson the night before.
8. A “Question” box where students can leave notes and questions for the teacher as they are going through the lecture so that they do not “forget” the questions the next day.

Finally:

9. Help me demonstrate the value add. Give the administrator statistics on numbers of teachers, numbers of lessons, number of students accessing. I want lots of numbers and data in charts and graphs.
10. Make the login interface logical and simple. Do not require a class list import or at least give me a variety of options such as a classroom passphrase that is entered after the student logs in.
11. Keep the cost reasonable. Google Docs is free. Microsoft is becoming affordable. Camtasia costs under $300. Dont price a brilliant product out of the market.

And that is it. 

My dream software. 

If you have already invented, please feel free to call me with a quote. if it hasn't been invented yet and the ideas are useful, toss a little credit to Brebeuf Jesuit, a school for all your flipping needs :). If you make millions, I want to talk to you about a library renovation.

(In all seriousness, i wouldn't be blogging about it if i didn’t want to see this happen. Prove me right, Internet, make the magic happen)