Friday, January 11, 2013

Bias in grading

From this document on approaches to learning, consider the following:
The parent indicated how frequently the child exhibited the following behaviors or characteristics. The response scale included four points ranging from "1 = never" to "4 = very often."

This subscale is composed of the following items:

10. Keep working at something until {he/she} is finished?

13. Show interest in a variety of things?

15. Concentrate on a task and ignore distractions?

18. Help with chores?

22. Eager to learn new things?

24. Creative in work or in play?

The above factors, averaged together, form (if I have not misread the documentation) the Approaches To Learning scale.

I'm not sure that item 18 really belongs here. It makes life much nicer for teachers, but says nothing whatever about how well someone will learn.

And items 10 and 15 don't address multitasking. There are times when I bear down, get in the zone, and do one task until it is done. There are others in which I pump until the well runs dry, switch to another task to refresh for a while, and come back to the original one later. But they sure make a teacher's life easier.

So one of the items doesn't deal directly with learning, and two others only partly do--but all three of them make a good impression on the teacher.

OK, look at a different set; these the teacher is supposed to evaluate:

11. Keeps belongings organized.

14. Shows eagerness to learn new things.

15. Works independently.

21. Easily adapts to changes in routine.

23. Persists in completing tasks.

24. Pays attention well.

In third and fifth grade, the following item was added to the SRS and added to the Approaches to Learning subscale:
26. Following classroom rules.

Here too: 21 is a problem. If I'm in the zone, I don't want distractions thank you very much, school bell or no. And if I'm multitasking--I mentioned that before. And I have somewhat different ideas about what constitutes "organized." Granted, disorganization can rise to a level that seriously interferes with getting things done, but below that it isn't obvious that it matters much. And 15: I've taught students who almost refused to work independently, but who mastered the subject just fine. And do I need to mention #26?

Once again the list includes things that make the teacher's life easier but don't necessarily effect learning.

This may seem a bit banal, but it turns out to make a difference.

A longitudinal study (I didn't see anything obviously wrong with it, BTW) of K-5 grade students found that teachers, unaware of their students' standardized test scores, assigned grades to boys significantly lower than those they gave girls with the same test scores. This was true in every case where there were adequate statistics (black 5'th graders were lower statistics--more had moved out of the study areas).

Girls were better than boys at reading skills on tests, but the spread in grades was bigger. Boys were slightly better at math tests, but their grades were worse; similarly with science.

Grades did not reflect mastery.

Since grades are a large factor in admission to higher education, this can have fairly far-reaching consequences.

So, why did the (mostly female) teachers grade girls higher than boys?

The researchers found that if they included the ATL score in their analysis, the differences almost vanished; in fact sometimes boys were graded slightly above the girls. Girls had significantly higher ATL scores than boys. (To try to take into account any bias due to the teacher producing both grades and ATL scores, they used ATL scores from 2 years earlier, presumably from a different teacher.)

The obvious conclusion is that teachers were including social factors in their grading.

That seems inappropriate. You can argue that these social factors will contribute to better performance in academic settings later, but AFAIK that claim is unproven, and in any event doesn't address the problem that the grade is not a clear measure of mastery. Splitting out attitudes from academic skills gives more information than blending them.

And as noticed earlier, those attitudes combine factors relevant to learning and factors relevant to class maintenance.

So, what can we do about this? Let's assume this is verified by other work.

  • The default approach is to circle the wagons, defend the institutions, pretend nothing is wrong and that mastery really entails social skills and we should leave the experts alone. This is probably what will happen.
  • We could try to ask teachers to split out ATL sorts of things in the hope that they'd create academic grades that better reflected mastery. I don't believe this will work at all.
  • We could try to find some new (or old) approaches to teaching that take squirrely boys into account better. This isn't easy to do in an industrial setting like a classroom. Home schoolers have an overwhelming advantage here.
  • We could have same-sex classes. I keep hearing rumors that these have academic advantages and I suppose I should poke around and find out the facts sometime. But I can hear the screams now. This one is political ebola.
  • We can use more and more standardized tests. I'm not thrilled about this option. It invites cheating (as seen already), teaching to the test, and generally locking everybody into the same one-size fits all mold.

I wonder if this happens with male teachers? Not that there are very many of them at that level. And I wonder if it carries over into high school and college?


bs king said...

Good post.

My brother got in to it with a male teacher for about a year in high school. Basically my brother would ask questions (hard ones) that the teacher couldn't answer, and would point it out when the teacher didn't actually answer them. He got in trouble continuously, marked down and asked to be home schooled the next year.

The next year they gave out an award in that class to another boy who "questioned everything". My mom almost flipped and asked me to get some intel on who this boy was and why it was so different from my brother. Turns out he asked lots of questions the teacher knew the answer to.

My brother is now pretty successful in a field related to the class he almost got thrown out of.

Anyway, also related:

james said...


One of my earliest memories of a teacher is from 2'nd grade. We'd had a rare hail storm in LA that morning, and the teacher tried to tell the class it was snow. I'd seen a patch of snow in the mountains during our move there, and knew better. Fortunately, so did another student.

Assistant Village Idiot said...

A lot of teachers are aware there is a problem, and in fact, are tired of having administrators "discover" every year that the boys are not thriving.

I think no one really knows what to do about it all. To teach a class, you need cooperation, eagerness, desire to please the teacher, etc. Girls are more likely to do that (though I wouldn't always bank on that, either). Without it, you don't cover the lesson. But people can tell that something is missing.