Cognitive Reflection Test

Yesterday I gave my students the Cognitive Reflection Test:


Seriously, answer the questions. It’ll only take a minute. I’ll wait.

Great. The answers are all the way at the bottom of the post where you won’t have the fun spoiled for you. You can go check. See how you did. The questions come from some psychological research — there’s a full paper here, and I stumbled across it in the book Thinking Fast and Slow, by Daniel Kahneman.

Anyway, most of my students, including some of my best students, got all three wrong. The results of the study are here:
CRT Data

Short version is — MIT students average a little over 2 questions right, and the full sample in the study, consisting mostly of college students, averaged a little over 1 question right. These are questions that are written to fool the reader into picking an easy, intuitive answer, rather than slowing down to think through whether that answer makes sense.

We talked through the answers, and then went on to discuss what Kahneman writes about as he summarizes the study. It highlights the differences between the two “systems”, as he calls them, in our brain. The first is the intuitive, quick-thinking system that does well-learned, instinctual tasks quickly and easily — recognizing that a face is happy or sad, writing your name, finding 2+2, climbing stairs, and other simple tasks. The second is the reasoning and processing system that works much more slowly, but is capable of reasoning and applying prior knowledge to a problem — finding 17 x 24, or designing a bridge made of toothpicks and marshmellows, or finding evidence that Elizabeth is a better match for John Proctor than Abigail.

The point was pretty simple. One thing I hope my students take from my class is to look at a problem and, instead of rushing to the first, easiest solution that their mind jumps to, slowing down. Our brains want to live in the first system, doing things as quickly and mindlessly as possible. But I’m looking for students to think carefully about what a question is asking, constructing an answer, and checking to see if it makes sense. That’s the type of reasoning that will make them successful in every area of their life. And problem solving in math is a great opportunity to practice this — looking at a new problem and considering it carefully rather than rushing to a conclusion, the same way I would love my students to reason about events in the news or their relationships with their peers.

So the Cognitive Reflection Test is one measure of a type of intelligence, associated with applying reasoning to simple problems rather than rushing to easy answers. I didn’t want to send the message that that is the only way that humans can be smart. Next we did a quick test that I first saw in a psychology class —

Take two minutes to think of as many uses for a paperclip as you can. Go. 

Some students wrote one or two and gave up. But some wrote that they could use it as a clothes hanger for doll-sized clothes, or an earring, or an axle for a toy car. It’s not perfect, obviously — neither is the CRT or the SAT or anything else — but it’s one way to measure creativity. And creativity — seeing as many solutions for a problem as possible, and looking for more after finding one — is one of the values I want to bring into my class more often, and this was my way of impressing that on my students.

We discussed for a few minutes whether students thought the CRT or the paperclip test measured a more important aspect of intelligence. Kids had some great ideas to share, pretty evenly split between the two, and I especially enjoyed hearing kids say that they were equally important, just different.

Anyway, after our little sojourn in psychology I gave them a problem set for partner work. Some fun questions — like this one:


Didn’t see any huge, amazing strides in reasoning, but kids were engaged and there was some really productive talk about the mathematics. Definitely wasn’t a life-changing class for any of my students, but I have some ideas to come back to as I try to push my students’ reasoning and problem solving skills.

5 cents (5 cents + 105 cents = 110 cents)
2. 5 minutes (1 machine takes 5 minutes to make 1 widget)
3. 47 days (it doubles every day, so one day later, it’s totally covered)

9 thoughts on “Cognitive Reflection Test

  1. howardat58

    3 correct !
    Reminds me of the old favorite:
    If a man and a half digs a hole and a half in a day and a half, how long does it take for one man to dig one hole ?

    1. dkane47 Post author

      That’s a fun one! One interesting piece of follow-up research was that people tend to do better on this test when they are sitting in an uncomfortable chair, or in a very hot room — when they are less relaxed, they are less likely to jump to a quick answer. I wonder if the way that question is framed would help people slow down and think more carefully about it?

  2. Five Triangles

    The bat and ball problem is a classic example of a common predilection for “rushing to the first, easiest solution”, even if the “solution” is wrong.

    A corollary of this tendency is that students often want gratification in the form of a quickly findable answer; in other words, if an answer cannot be found quickly, or the path to the answer is not obvious, they won’t have the “tolerance” to work through a task.

    However, there is workable two-prong approach to break this resistance: the first prong being to start with multi-step problems that are designed to be not particularly challenging, which makes it possible to increase problem length independently without adding to the cognitive load. We had that idea in mind when posing this problem:

    While this task uses very basic skills, it’s obvious from the beginning that there will be no rush to the finish–a psychological obstacle–but it’s partitioned in such a way that each step stands on its own, with the last step justifying the previous steps and tying them together.

    Once the patience and concentration for long problems begins to take root, then problem difficulty can be ramped up.

    1. dkane47 Post author

      I like that question very much — it’s a great test of understanding of linear relationships.

      A second approach I have found effective, borrowing from Dan Meyer’s (and many others’) work, is posing a perplexing question and having students begin by estimating an answer. Anything that creates an intellectual need in the student will encourage them to persevere rather than give up, and all of these pieces — first steps that are manageable, careful scaffolding, and problems designed to spur student investment — add up to students practicing and experiencing problem solving.

  3. mrdardy

    A non-math colleague at work asked me about the bat and ball problem recently. I tried to answer with Algebra language/symbols at first but my colleague was not following. When I tried to frame it as a ‘smart guessing and checking’ kind of approach she was a little more willing to follow along. Not sure what that says about our endeavor as math teachers.

    1. dkane47 Post author

      Reminds me of the Guess-Check-Generalize method for building algebraic reasoning — students often aren’t ready for the algebra until they’ve plugged and played with the problem for a bit — and after that, the algebra becomes necessary and logical.

  4. Pingback: Thinking About Thinking. | trigotometry

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s