“If I can do it, anyone can do it,” and other favorite SAT fallacies

It’s relatively common knowledge that “extreme” answers (ones that contain words like “always” or “never”) tend to be incorrect, but what doesn’t often get discussed is just why those sorts of answers are often wrong and how that reason relates to the overall goal of what the SAT is trying to test.

As I wrote about recently, one of the great themes of Critical Reading is that there’s “always a however” — that is, arguments are not black-and-white, that they contain nuances. Simply saying “well, x is obviously always right, and everyone who disagrees is an idiot” is not a particularly effective mode of argumentation. That’s part of the reason Passage 1/Passage 2 exists.

Ever notice that when one author presents one side of an argument in black-and-white terms, the other author will come back arguing exactly the opposite side? That’s not a coincidence. The point that the test is trying to make is that stating that something is true very strongly does not actually make it true, nor does it negate that fact that there are plenty of arguments against it. If you want to make a solid argument, you have to seriously consider the opposite side and think about how it fits into your argument — that goes a degree beyond what the SAT tests, but the skills that P1/P2 tests are the basis for that ability.

Note, by the way, that considering degrees of nuance is exactly the OPPOSITE of what most people are taught when it comes to the SAT essay (and, might I add, in school). That’s not to say you can’t earn a very high score arguing in extreme, black-and-white terms (indeed, it’s usually much easier to do so in the space of 25 minutes), simply that training yourself to look at only one side of an argument is a stellar way to blunt your understanding of how to approach Critical Reading.

So that’s point one.

The second, closely related point is that is individual experiences cannot necessarily be generalized, and that different people will hold different positions shaped by their background, social circumstances, etc. The assumptions you hold about the world might not be true for the kid who sits next to you in Bio, or who lives down the street or on another continent — even though you can all be classified as members of the category “teenager” or perhaps “teenagers studying for the SAT. In other words, what is true about a particular person or type of person (teenager, artist, scientist, anthropologist) cannot automatically be extended to every single other person or type of person — and the failure to fully grasp that idea is one of the fallacies on which many wrong Critical Reading answers are based. This is why answers that are “too broad” or “too extreme” are frequently (but not always) incorrect. It’s also why so many SAT rhetorical strategy questions ask you to notice personal anecdotes; relying on your own experience as your sole support for a claim does not a particularly solid argument make.

One of the things that I’ve noticed while tutoring is that my students have tendency to conflate the specific and the general — so, for example, if a passage discusses a particular painter and i ask them what the passage is about, I’ll often get a response like “painters.” At some level, they don’t seem to completely grasp the importance of the distinction between the singular and the plural; I know that because they often roll their eyes when I ask them how many painters the passage actually talked about. “Ok, fine,” I can hear them thinking, “it was only one painter, but who cares? Isn’t that, like, basically the same thing?”

Actually no, it’s completely different.

What’s really interesting to me, however, is the way in which this type of faulty reasoning replicates itself in discussions about SAT prep — the irony is really quite impressive. On one hand, this isn’t at all surprising: if people have trouble with the SAT because their logical reasoning skills are lacking, that weakness is going to manifest itself equally outside the test. But just for grins, let’s look at some of the most common. Notice the extreme language common to all of them:

-If I can do it, anyone can do it

Umm… Let’s examine that claim. Maybe you studied for a couple of years and raised your score 500 points to a 2350 — it certainly happens — but presumably you didn’t have a serious decoding problem (that is, you could actually read the words on the page and weren’t constantly guessing what unfamiliar words meant), were at some point capable of perceiving the underlying patterns in the test, had a good enough memory to retain all the vocabulary you were learning, and already knew many of the elementary and middle-school level words that the SAT tests (e.g. permanent, compromise, tendency. And that’s just a handful of skills for Critical Reading.

Having worked with kids who had serious trouble with all of the above, I can state pretty confidently that those are not things that can be taken for granted for every student. If a student has trouble with all of them, it’s a pretty safe bet that a 2350 — or even a 2000 — is not a realistic goal. An 1850 is actually a pretty solid score to start with; it’s usually an indicator that someone primarily needs to learn to take the test and that the underlying skills are more or less ok. In that context, 500 is points is a lot, but given the steepness of the curves on Writing and Math, it’s a difference of not all that many questions.

But there is a whole huge category of kids whose basic skills cannot be taken for granted and for whom raising their score even 100 points is a massive struggle. Kids with high-achieving peers and tutors with high-achieving students tend not be aware of their existence and therefore fall prey to another common logical misstep: if I haven’t seen it, it doesn’t exist (the underlying assumption being that they’ve already been exposed to the full range of achievement levels, or that their particular experience is somehow representative of the high school population as a whole).

Having worked with kids at pretty much the full spectrum (mid-300s to mid-700s), I can say pretty securely — and perhaps coldly — that not every kid has what it takes to improve by hundreds of points. Sometimes the deficits are just too extreme.

-SAT prep classes/tutors are completely worthless because I did really well just studying on my own

I’ve worked with lots of very smart kids who did fabulously once they were actually taught just what the test was asking them to do but who might not have figured it out by themselves. It’s nice that you did really well on your own, but you can’t conclude anything other than that tutors and/or prep classes would have been completely worthless for you. (Besides, even if you did amazingly, you might have also learned something from a tutor — for example, plenty of people manage to do very well on CR without really understanding what it’s testing.)

All SAT prep classes/tutors are completely worthless because they just teach you all the same strategies you could read in the books

That has an element of truth when it comes to strategy-based prep (at least in terms of classes), but if you need work on the actual skills, you’re probably better off sitting with someone who can actually teach them to you.

-Since I wouldn’t have done well on the SAT without a tutor, everyone who doesn’t have one must be at a huge disadvantage. Therefore, the only thing the SAT tests is how much money you have to prep.

Again, there’s a kernel of truth — as everyone knows, there is a direct correlation between SAT scores and income. Are kids who grow up in seriously educationally deficient environments at a huge disadvantage? Of course. Are wealthy kids whose parents can afford thousands of dollars for private tutoring at an advantage? Of course. Is that fair? Of course not.

But those are generalizations, and correlation does not equal causation: there are kids who can’t afford tutoring but who can sit down independently with a prep books and figure out everything they need to know; likewise, there are kids who get tutored for a year or more and don’t increase their scores at all (or worse, see them go down). Both are outliers, but they do exist.

There are also kids whose parents can afford tutoring but who are perfectly happy to sit down on their own with a prep book. If some of them score very well, it’s usually in large part because they’ve grown up in an enriched environment and got all the skills they needed just from their parents and school.

The fact that a tutor helped one particular person do well can by no means be generalized into the assumption that every person needs a tutor to do well, or that it’s impossible to do well unless you come from a well-off background.

And my personal fave:

-The only thing the SAT tests is how well you take the SAT

Does anyone truly believe that a kid who doesn’t know what “permanent” means can be expected to perform at the same level as one who’s studied three languages, reads Dickens in her spare time, and holds a position in the National Classical League? Or that a kid who struggles to get B’s in Algebra II (like I did) is seriously competitive with a national Math Olympian? Somehow I don’t think so. Yes, everybody has their strengths and weaknesses, but at the extreme ends of ability, the differences are so gaping that they’re pretty much impossible ignore.

It’s also possible to be very smart and still be missing important skills, and it’s a lot easier to blame the test for your shortcomings rather than own up to the possibility that you might not know as much as you think.

Guessing and skipping problems

Guessing and skipping problems

Of all the discussions floating around about SAT prep, the one I find most irritating — and pointless — is the guessing vs. skipping debate. I’ve heard all the debates by now, and I’m just not interested. I’m not a statistician, but after five years of teaching and writing what are essentially logic questions, I’ve gotten pretty good at spotting fallacies. I’ve also seen how things play out in the real world, where poorly thought-out guesses rarely pan out (at least when I’m watching).

The “guess if you can eliminate x number of answers” approach is based on the assumption that test-takers can reliably identify incorrect answers and that they will not eliminate the correct answer. From what I’ve seen, that is not even remotely a valid assumption. Certainly not for low-scoring students, but often not for higher-scoring ones either. (more…)

On being realistic (sometimes there is no shortcut)

As you may have heard, June SAT scores are back. And if you’re reading this, chances are you’re looking to improve your score this summer and thinking about what might be possible. Maybe you want to crack 1800…or 2000…or even 2300. You’ve got about three months, which is plenty of time to accomplish…something. If you’re unhappy with what you’ve managed to accomplish on your own, you might be thinking about thinking about taking a class or working with a tutor, or maybe you’re just planning to keep plugging away on your own. Regardless of your (or your child’s) situation, however, there are some things you should keep in mind.

So a little reality check. Some of this is going to sound awfully blunt, and probably more than a little harsh, but here are some things to keep in mind:

The SAT is hard.

Good grades are no guarantee of a high score.

Your score is the result of what you know, whether you can apply that knowledge instantaneously and under pressure, and how well you manage yourself on the particular test you happen to get; it is not something you are entitled to because you attend a particular school or have spent x amount of time or money being tutored, or even because you work hard.

Most people will, by definition, score somewhere around average.

If there really were tricks you could use to ace the exam, lots of people would get perfect scores instead of about 300 out of 1.5 million.

However hard you think you are working, there are other students out there who are putting in much, much more. If you want to equal — or surpass them — you need to be willing to work just as hard.

You are being compared to hundreds of thousands of your peers, including the very top students in the United States and some in other countries; the curve is designed to reflect that.

A tutor is not a miracle worker.

There is a skill level below which short-term strategy-based prep is usually not effective. A score below 600 after a significant amount of prep is usually a good indicator that there are a number of fundamentals missing, although higher scorers are often missing particular key skills (e.g. identifying the topic of a passage) to various degrees.

If you are missing skills, getting to the next level will require a huge amount of work, whether your score goal is 1600, 2000, or 2400. It’s as much about where you’re starting from as it is about where you want to go.

The SAT does not work like tests in school. It’s designed to gauge how well you can apply your knowledge, not whether you can simply cram in a bunch of words and formulas, to be forgotten as soon as you walk out of the exam room. If you haven’t mastered skills to the point where they’re automatic, you will not be able to apply them — or even be able to figure out when to apply them — to the test.

Today’s eleventh and twelfth grade textbooks are written at the same level that ninth grade textbooks were written at fifty years ago. If you don’t read anything other than textbooks and Sparknotes summaries, with the occasional Wikipedia article thrown in, you will most likely not be prepared for Critical Reading.

No tutor can compensate for two or five or ten years of accumulated deficits in a couple of months, never mind four or five sessions, and it is not fair or realistic to expect one to do so. A student who doesn’t know words like “surrender” and “compromise” and “permanent,” or who has reached the age of 17 without being able to consistently recognize the difference between a sentence and a fragment, is going to hit a wall unless they are willing to spend huge  amounts of time filling in some of those gaps on their own.

Now that I’m starting see lots of students who are missing important middle-school vocabulary and some who are missing basic elementary school vocabulary, I realize that Stanley Kaplan knew what he was talking about when he said that SAT prep should begin in kindergarten.

While the majority of my students improve, sometimes very dramatically, some of them do not; occasionally, their scores even go down. And students who come to me for a handful of sessions with middling scores, genuine knowledge gaps, and an unrealistic sense of just how much work they’ll need to put in to get the next level, rarely see any significant progress. (Note: taking three or four practice tests doesn’t count for much when there are people taking twenty or thirty…or more.) On the other hand, someone who has all the basics in place and just needs a little push to get to the next level might get where they want to be in a session or two. I’ve seen it happen more than once, but those people really did have things pretty much in order to start with.

I do my best to be really clear about just what I can and cannot likely accomplish in a given timeframe, but it’s a very fine line between being honest and being discouraging. I don’t want to turn away someone I could genuinely end up helping. I’ve seen enough kids pull off huge and unexpected jumps to know that it’s not my place to judge what someone is or is not ultimately capable of doing, but I don’t want to encourage people to harbor unrealistic expectations either.

I realize, by the way, that I probably shouldn’t admit all of this publicly — doing so can’t possibly be good for business — but given how convinced everyone seems to be about the existence of quick fixes, I feel responsible for saying something.

At some level, I think that the test-prep industry’s claim that there really are little “tricks” has become so ingrained in people’s psyches that they don’t fully grasp just how hard it is to raise a score, especially a Critical Reading score, until they see that 490 or 550 or 570 staring at them — again — from the computer screen. It seems impossible that they should have done what seemed (to them) like a huge amount of work and paid a lot of money, only to end up right back where they started. They don’t understand just how precisely the test has been calibrated to keep producing the same results. They hire a tutor because they think there really is some sort of magic shortcut (more than one parent has said I must know “all the tricks,” wink-wink, nudge-nudge) and are consequently very rudely shocked by just how hard they or their child will have to work to break through to the next threshold.

No matter how upfront I am about the limits of my abilities, though, I still feel responsible (and vaguely disingenuous, even though I’ve made it clear that I can promise nothing) when a student doesn’t improve. Then I start to wonder whether other tutors really do have secrets that I don’t know about.

Believe or not, I’m not trying to discourage anyone who’s less than over-the-moon about their SAT or ACT score. If you’re planning to study for this summer (and yes, I will post some actual test tips, not just whine about the decrepit state of the American school system, although I might have to get a few more posts about that in before I move on), by all means, you might actually succeed in raising your score hundreds of points.

Occasionally, like yesterday, I’ll get an email from a kid who did nothing other than work through my books and practice diligently, but who nevertheless managed to raise his CR and Writing scores by 400 points. Emails like that make my day, actually my week. They reassure me that people who put in the work actually can improve by that much, regardless of what the College Board claims.

Basically, you get out what you put in, tutor or no tutor. Most of my best students, the ones who make the 100, 150+ point improvements per section, have been incredibly self-driven. They experimented with strategies, hunted down old exams on the Internet, and read Oliver Sacks for pleasure; and when they came to me with questions, it was because they had worked through things as far as they possible could and were genuinely stuck. The ones who were dragged by their parents, who would clearly rather have been somewhere else… Well, some of them actually improved rather impressively, too (didn’t think I was going to say that, did you?), but they did stop short of their potential. The ones who did the work in only the most perfunctory manner, however, the ones who showed no interest in really understanding the test, and who expected me to give them a secret that would allow them to reach their goal without really having to think…Would you really be surprised if I told you that they almost always ended up disappointed?

So don’t think that improving is impossible. Lots of people do it, sometimes by quite a bit. But don’t expect a 200-point improvement to fall in your lap either. Or, for that matter, be hand delivered to you on a silver platter.

The answer isn’t always *in* the passage

A while back, a student who was trying to raise his Reading score came to me with complaint: “Everyone always says that the answer is right there in the passage,” he told me. “But I feel like that’s not always the case.”

He was right, of course. He’d also hit on one of the many half-truths of SAT prep, one that frequently gets repeated with the best of intentions but that ends up confusing the heck out of a lot of people.

As I’ve written about before, most SAT prep programs spend a fair amount of time drilling it into their students’ heads that the only information necessary to answer Critical Reading questions can be found in the passages themselves, and that test-takers should never, under any circumstances, use their own knowledge of a subject to try to answer a question. They’re right (well, most of the time, but the exceptions are sufficiently rare and apply to so few people that they’re not worth getting into here).

In trying to avoid one problem, however, they inadvertently create a different one. The danger in that piece of advice is that it overlooks a rather important distinction: yes, the answer can be determined solely from the information in the passage, but the answer itself is not necessarily stated word-for-word in the passage.

A lot of the time, this confusion stems from the fact that people misunderstand the fact that the SAT tests, among other things, the ability to move from concrete to abstract. That is, to draw a connection between specific wordings in the passage and their role within the argument (emphasize, criticize, assert, etc.). The entire POINT of the test is that the answers to some questions can’t be found directly in the passage.

Correct tend to either describe what is occurring rhetorically in the passage or paraphrase its content using synonyms (“same idea, different words”). You need to use the information in the passage and then make a cognitive leap. It’s the ability to make that leap, and to understand why one kind of leap is reasonable and another one isn’t, that’s being tested. If you only look at the answer choices in terms of the passage’s content, they won’t make any sense, or else they’ll seem terribly ambiguous. Only when you understand how they relate to the actual goal of the question do they begin to make sense. In other words, comprehension is necessary but not sufficient.

While knowing all this won’t necessarily help you figure out any answers, it can, at the very least, help to clarify just what the SAT is trying to do and just why the answers are phrased the way they are. If you can shift from reading just for content to reading for structure — that is, understanding that authors use particular examples or pieces of evidence to support one argument or undermine another — the test starts to make a little more sense. And if you know upfront that the answer is unlikely to be found directly in the passage and that you need to be prepared to work it out yourself, you won’t waste precious time or energy getting confused when you do look at the choices. And sooner or later, you might even get to the point of being able to predict some of the answers on your own.

Finally, some common sense…

From The Faulty Logic of The Math Wars:

A mathematical algorithm is a procedure for performing a computation. At the heart of the discipline of mathematics is a set of the most efficient — and most elegant and powerful — algorithms for specific operations. The most efficient algorithm for addition, for instance, involves stacking numbers to be added with their place values aligned, successively adding single digits beginning with the ones place column, and “carrying” any extra place values leftward.

What is striking about reform math is that the standard algorithms are either de-emphasized to students or withheld from them entirely. In one widely used and very representative math program — TERC Investigations — second grade students are repeatedly given specific addition problems and asked to explore a variety of procedures for arriving at a solution. The standard algorithm is absent from the procedures they are offered. Students in this program don’t encounter the standard algorithm until fourth grade, and even then they are not asked to regard it as a privileged method

It is easy to see why the mantle of progressivism is often taken to belong to advocates of reform math. But it doesn’t follow that this take on the math wars is correct. We could make a powerful case for putting the progressivist shoe on the other foot if we could show that reformists are wrong to deny that algorithm-based calculation involves an important kind of thinking.

What seems to speak for denying this? To begin with, it is true that algorithm-based math is not creative reasoning. Yet the same is true of many disciplines that have good claims to be taught in our schools. Children need to master bodies of fact, and not merely reason independently, in, for instance, biology and history. Does it follow that in offering these subjects schools are stunting their students’ growth and preventing them from thinking for themselves? There are admittedly reform movements in education that call for de-emphasizing the factual content of subjects like biology and history and instead stressing special kinds of reasoning. But it’s not clear that these trends are defensible. They only seem laudable if we assume that facts don’t contribute to a person’s grasp of the logical space in which reason operates.

In other words, reform movements are largely based on the rejection of a “reality-based” concept of education. We couldn’t possibly have anything as piddling as facts interfering with the joy and beauty of learning. If a child wants to believe that 2+2 =5, shouldn’t they be praised for thinking independently?

In all seriousness, though, there’s something borderline sadistic about schools refusing to teach actual, well-established knowledge, knowledge that makes learning easier. Not every student is genius capable of re-deriving the Pythagorean theorem on their own. Yes, by all means, teach students to understand why things are true – I’ve heard from math tutors who constantly encounter kids who do just fine in calculus because they’ve learned when to plug in about four formulas but who fall down on comparatively basic SAT math because they don’t really understand why things work the way they do, or how to apply simple formulas when they’re presented in unfamiliar ways. The point is, teach them something, don’t just let them flail around trying to figure it out on their own.

What’s the point in all those centuries of accumulated knowledge if schools are just going to toss it out the window?