The New York Times has an interesting article today on the simmering controvery over “flagging” the SAT results of students with disabilities who are allowed extra time. (See here for an earlier post of mine on this matter.) It seems the number of requests for extra time has been increasing dramatically — “teachers at the New York’s top private schools estimate that in recent years, at least 10 percent of their students have taken the SAT with extended time or other modifications” — and the College Board has begun investigating these requests more stringently.
But that’s not what I want to talk about. Instead, I’d like to see a little controversy over another College Board policy on time: seeding some Graduate Record Examinations with an identified, un-graded section used for research on possible future questions.
From the GRE website:
The computer-based General Test contains four sections, one of which is an unidentified pretest section that can appear in any position in the test after the analytical writing section. Questions in the pretest section are being tested for possible use in future tests and answers will not count toward your scores. An identified research section that is not scored may also be included and this section would always appear in the final section of the test. Questions in the research section are included for the purpose of ETS research and answers will not count toward your scores.
Total testing time is up to 3 1/4 hours, not including the research section. (Emphasis added)
My daughter Jessie just took the GRE. She had taken practice exams a number of times before, but she reported that her actual test did have an identified research section, that as a result her test was much longer than she expected, and that she was quite tired by the time she got to the last part of her test, which was part of the verbal test. Her score on the test, although quite high, was lower than she had gotten on her practice tests.
Is this fair? If the test results of students who are allowed more time are flagged, shouldn’t the results of those who are required to spend more time (because of an added section stuck in the middle) also be flagged? Isn’t there something amiss about involuntarily drafting unwilling research subjects? Or am I missing something?
Question: The instructions you cite read “An identified research section that is not scored may also be included and this section would always appear in the final section of the test.” At least reading this language, it leads one to believe that the identified research section comes last, after the graded sections. Was that the case with the test in question? If so, I think it is more or less fair, because although the extra section might be extra tiring, so long as it comes after the graded sections, this extra fatigue shoudl not impact the scored sections. Or am I missing something?
But in any event, glad to see you attacking standardized tests, so I guess I’ll take what I can get. ;-)
I guess maybe I was missing something. Looking at the language again, I see that it is susceptible to another reading. “[I]n the final section of the test” might mean that the “identified research section” was in the “final section“, but yet was not the last “section” of the “final section,” that is, a scored “section” came later within that “final section.”
This reading is, well, a bit strange, because I generally expect “section” and “section” in the same discussion to refer to equal levels of division (when referring to something smaller than a “section,” you could easily use “subsection” or “part,” for example). If it is the case that the “identified research section” was within the last “section” but yet was before some other, scored “section,” I’d say this is not only unfair, but the given explanation is an example of absolutely atrocious drafting.
What happens if you skip it?
Garrett – I’ll have to ask Jessie the particulars again. (She had just finished the Physics subject test when I wrote this, and I didn’t want to make her think about the GRE any more….) Anyway, my impression from what she said (and, obviously, I may have misunderstood, or I may have correctly understood but she may have been wrong) is that the offending “identified” research section was included in her test, somewhere in the last “section,” but that the only identification was the notification that it was there. But there was no indication of which questions were part of this “identified research section.” Thus it was not possible to skip it, if it was there.
Now, it is possible that she confused what the GRE web site calls an “unidentified pretest section” with what the web site calls “[a]n identified research section.” All I know at the moment is what she said when she came out, and the fact that her test did in fact last longer than 3 1/4 hours.
Whatever the fact of the matter, whoever wrote the description of the test on the GRE web site sounds like someone who failed (or should have) the new writing test that is part of the exam now.
They do not identify which of the sections is the research section. (I worked for Kaplan, teaching a general GRE test prep course).
I imagine the time taking the test is so much longer due to the writing portion. When I took the general GRE in 1995, the GRE took less time than the SAT had for me in 1992.
John – For starters, there’s no way to get around giving test takers items that are not scored, because that’s the only way to ensure that such items are pretested under high stakes conditions. Exam sections on tests such as the GRE are, even if adaptive, set to meet content requirements, so that instead of seeding experimental (also called unscored or variable) items throughout all the sections, they lump them all into one section that, because it is unscored, does not have to meet content restraints.
Normally, though, students are not told which items or sections are unscored, because the whole point is for students to treat those items the same as they treat the scored items. What’s confusing here is that the GRE uses both an “identified” and “unidentified” research section, and these sections may indeed be timed differently than the other sections.
I don’t know the point of using an identified research section. I do have a guess as to why the unscored sections might not have the same time limit. They might expand the time limit of those sections so that they have some idea of how test takers perform on certain items when the time limit is relaxed.
However, schools know this. I assume that Jessie’s score report will reflect only her scored sections, but I also believe that there might be a full report noting that she took two pretest sections. Certainly, she could say that if asked about it.
Is it unfair? Well, the identified research sections are most likely administered at random, and to be frank, if a section appears that is IDENTIFIED as being unscored, I see no reason that the test taker would feel the need to answer the items correctly. That part does seem odd to me. I agree with Garrett that this explanation should be clearer.
Oh, and if anyone is interested in reading my previous posts on the flagging flaps (yeah, I’m tootin’ my own horn on John’s dime here), here are a few links:
http://www.kimberlyswygert.com/archives/001335.html
http://www.kimberlyswygert.com/archives/001003.html
http://www.kimberlyswygert.com/archives/001147.html
Kimberly has forgotten more about testing than the rest of us will ever know, and so I should have linked to her post on flagging, which after all inspired my criticism here. Those posts make the point that it is at least arguably unfair NOT to flag the test results of those test takers who are allowed more time. As I understand what happened with Jessie’s GRE test, she was REQUIRED to spend more time to deal with a section, or with a certain number of questions, IN ADDITION to the “pre-test” section that all GRE exams have. Moreover, as I understand it, she coulnd’t skip this second section because, although she knew it was there, she didn’t know which questions it comprised. As a result, she was quite tired when she got to the last section of the test (which was part of the verbal test) and didn’t do as well as she might have otherwise. Of course, it’s possible that this very last section was the part that didn’t count.
The bottom line here, for me, is that if she was required to take a test that was longer than the tests other GRE takers took, that fact should be flagged. The problem is not unidentifed research being scattered throughout all tests; the problem is (if it’s true) some tests being longer than others because of an extra section of ungraded material thrown in the middle of it.
Special accommodations and time limits on the SAT
Hey, John of Discriminations has been blogging up a storm this week, and he ran across the recent NYTimes article on the accommodated testing flagging controversy, which he mentioned in passing in another discussion of a GRE peculiarity (I left…
I just finished taking the revised GRE for the second time. Both times I had UNIDENTIFIED EXPERIMENTAL sections. My understanding is everyone gets one of these and it can come at any (after the essays because they are always first). There is a possibility of receiving an IDENTIFIED RESEARCH section but it will always be at the end so that should have an impact on your score. It will just make the experience more painful. But since it is identified you really don’t need to stress about it. I would myself would just breeze through it.
This is not to say all tests are created equal as the unidentified experimental section could be quantitative or verbal. This can have significant impact on your results. As the experimental questions have not been tested their is a greater possibility that they could be bad questions, causing the test taker undue frustration. If you happen to be an unlucky test taker and you get the experimental section at the beginning (immediately after the essays) this could have devastating effect.
For me my verbal section is my strength and the first time I took the GRE I got an extra quantitative section. This time I got an extra verbal. It is not surprising to me that I improved my scores in both substantially.