Slashdot

If the scores are all the same, then it really doesn't matter who gets in. An essay is a shitty way to select engineering students and doesn't gauge anything other than their ability to make up 500 words of bullshit.

If there's any reason why these kinds of things tend to be bullshit, it has nothing to do with the fact that these are engineering students, or that engineers can't or shouldn't learn to use language as a tool (or, for that matter, that they shouldn't learn to bullshit).

The problem comes in the intersection of the purpose of the essay and the formation of the questions. It's an admissions essay, which means that whatever you're asked to say or whatever you're ostensibly saying, the purpose is to say whatever impresses admissions officers and get admitted to the college. Everybody knows this, and it reduces the ability of most people to speak authentically (and increases their tendency to bullshit). Particularly with essays that ask people to talk about themselves, because no matter how many distinct things there are about individual people, even smart people, there's an awful lot of sameness running through the human condition. Meanwhile, admissions officers are looking for distinction. Talk about cross-purposes.

Clare Bayley's suggestion "change the prompts, not the length [mit.edu]" is some clear thinking. Prompt the applicant away from a self-focus and you untangle the better part of the tension I describe above, while still allowing applicants to reveal expressiveness and distinctive thinking.

Absolutely not. MIT's interest is not to find the most capable geeks, it is to find balanced individuals with a broad range of interests, who are are likely to become the leaders of tomorrow. Communication skills are essential for that.

Being technically strong is only one ingredient of success, even in technical disciplines.

I am a tester.

It is not that easy to do. Something like the SAT needs to be able to ascertain the level of a very broad population of users. That is done by targeting items at examinees of a certain ability, using item response theory difficulty estimates acquired in pilot. Yes, you could add more difficult questions, but those are going to render very little information about most people who take them, because they won't get them right. Because the edges of the bell curve are so thin, throwing a bunch of hard questions at the top end will water down the results for the majority of users. Ideally, you want most of your items that are very close to the level of the examinees, so you get nice "high-resolution" (although no one uses that term but me, since I came in from IT and explain everything as though it were a computer) discriminations of ability for the most people possible.

The workaround for this problem is to tailor the test to the examinee, real-time, with computer-adaptive testing. So let's say you get an item with a difficulty estimate of 1 correct; now the computer will hit you with one at 1.2, for example, and keep ramping up until you kind of level off at getting 50/50 right, which is where it decides you belong. Once it has you figured out, it either just throws easy ones at you so you feel good about yourself, or starts serving up items still undergoing pilot testing. Either way, what you do after that point will not affect your score.

This sounds great, and it would be great, if it worked reliably. The problem is that the thing has to kick in somewhere at the beginning of the test, and define a broad range that you belong in, and then a narrower range, and then a narrower range, etc. What this basically does is unfairly "weight" the first few items of the test, because they are the ones that will determine what large band of scores you will be eligible for. Once the machine has pegged you at the lower half, say, there is no way for you to break out of that, because it's never going to give you those harder questions. If that's not where you belong, you won't be able to demonstrate that, and you'll just get the top score of that band. So if you start the thing out and you're nervous and you just make a dumb mistake, that mistake can really cost you--much more than it would later in the test. All these models are probabilistic, so guessing and just making dumb mistakes are accounted for. But the moment you go adaptive, the beauty of the model is trashed at the beginning and doesn't come into effect until later.

Many of the tests which moved to computer-adaptive methods have gone back to just serving a range of items, but one, the GRE, is still adaptive, even though ETS (the company that makes it and the SAT and the TOEFL) knows it doesn't work reliably (people taking the test over and over can get very different scores). Evidently there are financial/political reasons they can't get rid of it (rumor). And I have to take it again here in a few months to start applying for PhD programs. One of the drawbacks of researching psychometrics is that at some point you'll have to take one of these tests, knowing what the problems are.

So there you go. Yes, adding harder questions would indeed get you better discrimination among the top examinees, but at the cost of discrimination for the bulk of them. Ideally, you could just have the examinee come back and take the next-hardest test, but no one would go for that. Or perhaps the tests could be tiered, with linking items/anchoring, and the examinee could choose what level they wanted to take. I don't know of any major tests that do that, though, and having disjoint populations might cause a problem...

Anyway, there's more testing minutiae than you require.

Actually, they want people who are likely to be successful, and become leaders of tomorrow. These are the people that will go out and advertise their alma mater to the next generation. They are also the kind of people who end up making the big alumni donations.