Indiana First With Computerized Grading

Mz6 writes "Computerized grading has been talked about previously, however, the New York Times reports that Indiana has become the first state to grade high school English essays by computer. The computerized grading process, called 'e-rater', uses a 6-point rating scale and uses artificial intelligence to 'mimic the grading process of human readers'. The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers. The big question is, will other states begin to emulate Indiana by tossing human grading?"

I smell lawsuits, how about you?

[Allen Zadr]

... and produced results virtually identical to those of trained readers.

Funny, because the way I read that is, "Produced lawsuits where the cost is virtually identical to about 20 times the short-term savings."

I see this coming from both sides. The obvious, the grading was wrong, and I lost a scholarship. To other people suing after dropping out of collage level english classes (the test said I was better than I was).

Re:I smell lawsuits, how about you?

[m0rph3us0]

I think you've taken one to many collage level english classes my friend.

Re:I smell lawsuits, how about you?

[m0rph3us0]

s/to/too/. I feel stupid now.

At least the parent proves something...

[Ieshan]

At least the parent post proves one simple truth: human english teachers can be replaced by simple shell scripts.

Re:

[account_deleted]

Comment removed based on user account deletion

Re:I smell lawsuits, how about you?

[magefile]

extra points to a particularly good paper from a stupid student or take away points for a particularly bad paper from a brilliant student.

A "stupid" student doesn't deserve more points just for improving than a non-"stupid" student, and vice versa.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:I smell lawsuits, how about you?

[king-manic]

Why is that? If you had two students, one that you knew was brilliant and one that you knew wasn't the sharpest knife in the drawer, would you give them the same grade for the same paper? If the smarter person put 20 minutes of effort into the paper while the dumber person worked their ass off for a week on it, why shouldn't the grades be different?

Why punish those with ability, why reward those who are not talented. Surely they won't be punished or rewarded in the same manner in real life. If the talented one was cruising and not expending any effort, thats his perogative. If he can be a productive member of society without effort thats fine, even if he has the potential to eb the next great mind, it's still his choice. The rewarding of those who work hard is important but if they work hard to only measure up to the minimium standard then they belong to that minimium standard regaurdless of how much effort they put in.

Re:I smell lawsuits, how about you?

[cloudmaster]

Darn straight. I'm all for making dumb people feel not so bad about things they can't control - like being dumb - but not at the expense of making the non-dumb feel dumb. There's a lot to be said for working smarter, not harder, and a lot to be said for those who have figured that out on their own. English provides several correct ways to do something and some incorrect ways of communicating. An "A" paper is an "A paper, regardless of the source and effort put in by said source.

Depends on the purpose of gtrading...

[Dr. Zowie]

If grading is intended as a motivator to encourage each student to perform his/her best, then more effort should yield a higher grade. Likewise, if grading is intended to reflect the student's ability to perform in a real-world situation, effort should probably yield a higher grade: folks who work hard tend to do better than folks who are marginally smarter but don't work hard, in real-life situations. But if grading is intended to reflect only the quality of the work that was submitted, then sure -- effort shouldn't count at all.

This issue cuts deep into the heart of what grading is for -- it's possible for smart people to reasonably disagree, depending on what they think the intent of the grade is. Since grades are put to many uses, there are many answers to the question.

As a college instructor, I tend to use a strict grading protocol -- and then "bump up" a few of the students. If someone comes in to my office every week and really struggles to understand the concepts, but the computer tells me that they earned a "C+" -- they're likely to find a "B-" on their transcript. But if someone who's smart enough to get an "A" blows an exam from being hung over, that person gets little or no sympathy.

Re:Depends on the purpose of gtrading...

[hesiod]

> If grading is intended as a motivator

A grade is used to show how well you know a subject. If I knew a subject completely before starting the class & wrote the same level of paper as someone who studied his ass off, we deserve the same grade. He should not be given a better grade than me just because he waited for this particular class to learn a subject. Why am I judged differently because I took the initiative to learn the information earlier than I absolutely had to?

No, schools are not there to make you feel good about yourself (that's obvious), they are there to make sure you know the minimum information to pass a class. That'a a D. Then, if you know more, you get a better grade. The amount of work you put into it is irrelevant. In fact, if you put more work into it than you should have, it means you are not doing well, and once you get into the "real world," where you have strict deadlines, you don't have the option of getting paid more just because you worked harder for the same result. The exact OPPOSITE is true, in fact.

School should help people prepare for life. If someone is given a grade they did not deserve, they are being improperly trained how to work.

A problem, however, is with the PARENTS. Many students are C students, that's all there is to it. But they get all high & mighty towards the school if they see their child work very hard for a low grade. They figure their child isn't good enough if they don't have all As, but that they deserve them just for hard work.

If life had a payrate based on how hard you worked, vs. your productivity, I would start working as an astrophysicist. I wouldn't get anything done, since I know nothing about the work, but I would sure as hell work my ass off. Do you think anyone wants an employee like that?
(I mean the lack of knowledge -- everyone wants a hard worker, if they know enough).

Re:I smell lawsuits, how about you?

[Anonymous Custard]

The real answer is to adjust your teaching methods per student based on subjective analysis. A low objective mark on a paper or test would indicate that, ideally, the teacher needs to pay closer attention to the needs of that student, and teach him in a way he can learn. (VERY few teachers have the time and skill required for this, unfortunately).

You can't grade subjectively because those grades will be compared objectively down the line. You can't say "this is pretty good for kevin, I'll give him an A", but then say "josh's paper is way better than kevin's paper, but josh is a bright kid, so I'm giving him a C". Kevin will think he's mastered the english language while Josh will go insane trying to achieve perfection.

Grading, when used for anything other than helping the teacher learn about each students, just plains sucks, and is only used for competition.

Re:I smell lawsuits, how about you?

[Shalda]

I disagree. I see lawsuits as no more likely. Furthermore, any process where you're subjectively evaluting something there has to be quality controls and an appeals process. My wife once held a part time job grading essay questions on a high school exit exam. Every few hours of grading exams, she would have to take and pass a "calibration battery" of 10 exams. Quality control is fundamental to the process.

What I see as being problematic is kids learning to beat the system. Typically these systems are predicated on gramatical analysis (use of punctuation and sentence compeleteness) and evidence of citing the text the question is based off. I'd bet its a real easy system to beat.

Re:I smell lawsuits, how about you?

[Bombcar]

I wonder if it has bounds checking on input..... perhaps you could submit /proc/kcore and cause a buffer overflow and get r00t on the gr4d3b00k!

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:I smell lawsuits, how about you?

[neurojab]

>Using a computer to measure ____ seems like an improvement, as a computer won't mark you down for race, religion, opinion, etc - things that tend to drive humans to have lower opinions of each other (and consequently can bias grading).

Yes, a computer has no bias. It also cannot comprehend what is written AT ALL. All it can do is process a few heuristics. This means that a child that's incapable of forming an intelligent thought will get rewarded for simply putting random words in a sentence structur

Re:I smell lawsuits, how about you?

[ari_j]

I don't think that bad grading is what's hurting your ability to get a writing-based scholarship, bud. Your comment is barely comprehensible, has almost no examples of correct English grammar, and has some spelling issues and capitalization deficiencies, as well.

A third source: TurnItIn.com-style relationships

[Flexagon]

If it doesn't already, I would expect a service like this will eventually include plagiarism detection, due to marketing pressure if nothing else. This is something that human graders do, at least over the space of papers they grade and works they remember.

But if plagiarism detection is added, then the grading service would have to make and retain some encoding of each graded paper, a derivative work, in its database.

Once that happens, the grading service also becomes subject to all of the issues already

Re:I smell lawsuits, how about you?

[slam smith]

It does seem unfair. After all at least the roofers are doing something useful.

OSS?

[Karamchand]

Is this program available under an open source licence? It sounds really interesting!

Re:OSS?

[twistedcubic]

Cool idea. Imagine high school students re-writing their essays until the grader software gives them an A+.

Re:OSS?

[borkus]

Imagine high school students re-writing their essays ...
Actually, anything that would encourage students to re-write their papers and improve their writing would be pretty amazing. Most students jot something down, run a spell checker and turn in their work. If they could pre-grade their work, they might be better motivated to put out more effort and improve their writing.

Fortunately, when people graduate from high school and enter the workforce they become motivated to always make their best effort.

Re:OSS?

[arose]

The thing is that a human can actually understand what you are writing. Clever arguments? A nice flow? Humor? Why would I concern myself with things that make reading interesting, if nobody is ever going to read it. Who cares if its dull, just get the punctuation right! Nevermind that it doesn't make much sense, just get a dictionary and show the computer your rich vocabulary...

Re:OSS?

[Mad Marlin]

My guess is that they are talking only about things like this [gnu.org]. I used to use a similar program back when I was taking English classes, in order to bring my papers down to an 8th-grade reading level.

These are encredibly easy to mess around with. For example, the fog index is:

Fog Index = 0.4*(words/sentences+100*((words >= 3 syllables)/words))

Which is roughly equal to the school grade reading level required for the essay. If I remember correctly, Associated Press articles are written to a 4th-gra

Re:OSS?

[Mad Marlin]

For example, my post above (with the formula removed):

readability grades:
Kincaid: 6.4
ARI: 6.6
Coleman-Liau: 9.2
Flesch Index: 77.8
Fog Index: 8.5
Lix: 35.8 = school year 5
SMOG-Grading: 8.0
sentence info:
408 characters
96 words, average length 4.25 characters = 1.33 syllables
6 sentences, average length 16.0 words
50% (3) short sentences (at most 11 words)
33% (2) long sentences (at least 26 words)
3 paragraphs, average length 2.0 sentences
0% (0) questions
10

Re:OSS?

[Glonoinha]

Slashdot First With Computerized Moderation.

Now with Computerized Moderation the famous Slashdot message site can pre-emptively down-mod 'Redundant' posts long before they are actually 'Redundant.' The computerized modding process, called 'e-modder', uses a 6-point rating scale and uses artificial intelligence to 'mimic the modding process of human readers - including doing stupid shit like modding the first instance of a concept as Redundant'.

I would have loved this is a kid

[cheezus]

it would have been my goal to make the most wrong essay I could that would still generate a good grade from the system.

Re:I would have loved this is a kid

[jeffy124]

an opinion paper on why dyhydrogen monoxide must be banned would perhaps do the trick?

Re:I would have loved this is a kid

[Kiriwas]

In the florida school system this is a writting test called the Florida Writes. Its a standardized test, doesn't count for much, but we all had to take it. This is basically exactly what we had to do. Write the most horrible peice of trash you could think of, but make it adhere to a few preset guidelines and BAM instant grade. My first time taking it I wrote a rather good peice of work if I do say so myself. Problem was, it didn't ahhere to their simply 5 paragraph, introduction, 3 body, and conclusion. I did horrible. My second time, I wrote something that I barely called English but followed what they wanted perfectly and got top marks. I see this new computerized grading as being just exactly the same.

Re:I would have loved this is a kid

[cubicledrone]

Problem was, it didn't ahhere to their simply 5 paragraph, introduction, 3 body, and conclusion. I did horrible. My second time, I wrote something that I barely called English but followed what they wanted perfectly and got top marks. I see this new computerized grading as being just exactly the same.

It is also the exact environment of the modern workplace, as designed. Results are irrelevant. Only the process matters.

Re:I would have loved this is a kid

[LnxAddct]

Well I've read almost every post under this story and noone has mentioned how it actually works. From the e-rater site "E-rater learns to score essays on a particular topic by processing a significant number of essays on the topic, each of which has been scored by two or more faculty readers. While e-rater is a powerful scoring engine, it is not meant to replace a teacher whose judgment is essential to helping students improve their writing ability." This means that its essentialy a bayesian filter that ins

Re:I would have loved this is a kid

[Krach42]

I'll let you in on how it works, as I ported some software that does exactly what this essay grader does, as work for a professor who worked on this stuff.

It's not a baysian filter, it's Latent Semantic Analysis. LSA works by taking large amounts of text, and comparing the usage and application of the words within paragraphs. It learns very quickly what words mean, and the interesting thing is, that once it's trained far enough, it starts gaining more meaning to its words by where they're not, than by wh

I already want a copy of this.

[ShitPissFuckCuntTits]

I bet I could write the other side of the equation: a program to create nonsensical gibberish that always gets A's. What would a teacher do if you handed in something like that? Apply a double standard to the student?

In fact,

[ShitPissFuckCuntTits]

As a student the first thing I would hand in is twenty paragraphs from refreshing this [rubberducky.org]. In political science class, of course.

Re:I already want a copy of this.

[glenrm]

If I was the teacher I would give the a student the student what the deserve an A.

Re:I already want a copy of this.

[Kozar_The_Malignant]

> I bet I could write the other side of the equation: a program to create nonsensical gibberish that always gets A's.

I'll bet half the people here thought this as soon as they read the headline. The normal /. response is to spend 10 hours at the computer coding up something to do a task you can do in 1 hour with a pencil. :-) Of course the high school geek who does this will make piles of money selling the output to the football players, buy a nice ride, and get a prom date with a cheerleader. Isn't that the way it works?

Re:I already want a copy of this.

[Glonoinha]

Bah, right now the program takes an essay as input and outputs a grade. Simply reverse the pipe streams, push in an A and have the programs spit out a essay worthy of that A. Magic!

Stupid

[phreak0003]

I live in Indiana, and I have taken these. They are not graded fairly, and they determine 10% of the final grade. A computer can obviously not grade essays fairly, so it shouldn't be done. I got a 5/6, which, according to the computer, was extremely well. However, this was an 83%, which brought down my grade significantly. This computerized grading isn't fair at all.

Re:Stupid

[RoderickMcDougall]

I agree. Did you receive any comments from the computer or was it just an unjustified percentage? I would be most interested to know how they have managed to justify this questionable practice. Almost as someone is rushing to boast about being at the forefront of technology without the goods to back it up. Oh the children! Won't someone please think about the children!

Re:Stupid

[garcia]

I have taken plenty of essay exams where I felt I wasn't graded fairly by a human teacher/professor.

Some essays were graded out of a couple points. A paper out of 6 points carries less weight overall. If this is the only exam (ie AP tests) a 5/6 is looked at as a high score.

I don't see your point.

Re:Stupid

[Anonymous Coward]

Maybe you got docked for overuse of commas.

Re:Stupid

[Roland Piquepaille]

Not only computers may not give fair grades, but there's a deeper problem with grading using computers: to me, students working to get good grades from a computer conjures up images of sheeps going in the wool-extraction machine. While this may be fine for sheeps, how do you think the students feel about it?

When I was in school, I was glad to know whatever essay I was writing was being read by my teacher, whom I had real human student/teacher relationships with, and whom I could discuss whatever was or was

Re:Stupid

[garcia]

When I was in school, I was glad to know whatever essay I was writing was being read by my teacher, whom I had real human student/teacher relationships with, and whom I could discuss whatever was or wasn't right in the essay after class. The schooling system already lacks humanity, why de-humanize it even more?

You had a different school experience than I did apparently. I felt that the human's reading my papers were distant, uninteresting, and less than worthy of grading someone else's work.

Generally co

Re:Stupid

[servognome]

Generally comments were kept to a bare minimum on a good paper. "Good job!" or "Excellent research!" is about as lame as getting a 5/6 on a standardized essay exam from a computer grader.
Your feedback was actual words!? In my day all we got was a scratch-n-sniff sticker and had to guess the meaning of getting a "watermelon" on our essay.

Re:Stupid

[micromoog]

They're just converting "n/6" directly into a percentage grade?! That's ridiculous. So there's no such thing as a C (going straight from 83 to 67)?

Re:Stupid

[ibpooks]

A computer can obviously not grade essays fairly, so it shouldn't be done.

The article states that comparisons between the computer grading and human grading revealed nearly identical scoring. By this data, I don't think it's obvious that a computer can't be as fair as a human.

I got a 5/6, which, according to the computer, was extremely well. However, this was an 83%, which brought down my grade significantly. This computerized grading isn't fair at all.

What does the computer have to do with it? You

Re:Stupid

[I_Love_Pocky!]

The whole idea of an automated grading system for essays is insane to begin with. The single most important aspect of an essay is its content, not its form. Form and grammar are important in conveying a message, but the message is what is actually important. The things an automated grading system can grade should not make up much of an essay's total grade.

Besides, anyone who has read much literature knows that many great authors play with grammar, spelling, and form in non-standard ways in order convey a message. An automated system would grade them poorly, because only those who conform exactally to the rules get a good grade. Is our goal to turn all of our students into mindless automatons whose only goal is to churn out exactally the same drivell as the next guy?

They are not graded fairly, and they determine 10% of the final grade.

10% of the grade on the essay? Or in a particular class? 10% on the essay may actually be tolerable, because that means that at least a human actually read it to give the other 90% of the grade.

Re:Stupid

[rsadelle]

I certainly hope your essay was better written than this comment. If it wasn't, and you still got 5/6, that's pretty good evidence that the computer isn't doing accurate grading. IANAET (I am not an English Teacher), but I would grade this comment 3/6. Some comments:

Do you have any evidence or thought behind your statement that "A computer can obviously not grade essays fairly, so it shouldn't be done"? Why is that obvious? Is it obvious only because the grading of your essay was, in your opinion, not fair

I guess this rules out...

[Loco3KGT]

schmoozing with the teacher to get higher grades.

In unrelated news, Delicious Red Apples have suffered a terrible sales slump.

That's ok

[paranode]

schmoozing with the teacher to get higher grades

This works better for the Slashdot crowd. They are much better at romancing computers than people to get what they want.

comment moderation

[Jotaigna]

for the time being, i would trust more that program to moderate my comments.



c'mon people i was only joking dont mod me down, not noooo!!

What about tricking the software?

[gtaluvit]

SPAM filters are tricked all the time depending on the text of an email. Google was f'd up not too long ago because of trackback linking in blogs screwing up their algorithms. Isn't this a similar situation? If a student can figure out a way to beat the grader, we'll have students learning to write to beat software, not form a well written essay.

Re:What about tricking the software?

[drooling-dog]

we'll have students learning to write to beat software, not form a well written essay.

Right. The software can grade things like spelling, vocabulary, grammar, and syntax, but it won't grade the quality of the ideas expressed or the depth of understanding displayed. Therefore those latter dimensions will cease to be considered important, since grading them is expensive compared to the more mechanical stuff.

So much for those essays

[Zancarius]

Perish the thought should students start writing about the dangers of artificial intelligence. They may very well fail!

Gaming the system

[ePhil_One]

While it gives identical results for now, I can easily see the coming books: How to Write an A essay! Form essays to get you into Harvard.

The GMAT books are already giving formula essays to get you past any writers block that might happpen on the exam day...

Not the First

[dcocos]

My alma matter [wisc.edu] graded most of my computer programs with shell scripts and I graduated in 1997. So I don't think India is the first to do that.

Re:Not the First

[gregarican]

Grading freely-written essays and structured computer programming code is two separate things *to a degree.* And India isn't the first to do the latter. Indiana is, however, according to the article.

Re:Not the First

[gregarican]

Actually I meant the former, not the latter. Everyone makes mistakes I guess :-)

Yeah...

[Paulrothrock]

Sure, this won't be hacked... Some script kiddie is gonna get the algorithm and get straight A's for his whole life on gibberish and devote more time to a new virus.

Those Indianans are ruining us!

[L. VeGas]

Lets just outsource all our test grading to Indiana too.

Even more efficient?

[yndrd]

Let's just use a random generator to give out grades. We'll get the same bell curve distribution, won't we?

Trained readers...

[ari_j]

The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers

I think this says more about the training that the "trained readers" are receiving than it does about the software.

Perfect scores every time

[Rorschach1]

Ok, since you know the grading software is going to make it into the hands of the students, here's my scheme for perfect essays:

Step 1: Feed some encyclopedia articles, Wiki pages, and other random material on your subject into a Markoff chain generator.

Step 2: Use a genetic algorithm to generate variations of the text. Fitness is determined by the grade calculated.

Step 3: Repeat step 2 until desired grade is achieved. (And, of course, Profit!)

The result is totally worthless, but at first glance would probably appear legitimate even to a human reader.

Sort of like Slashdot posts.

We have computers to help grade assignments...

[Anubis333]

Now if we could only get computers to help teach our children!

Content?

[sokk]

Well, what about the content of the essays? Anyways, any way to get a hold on the software :)? Would've been nice with a "clippy" that said what grade the current document is.

AI

[Bugmaster]

That's actually a pretty novel way to approach the problem of creating Strong AI. Making smarter machines is hard, so what you do is dumb down the humans until even a coffee maker (or a grammar parser or whatever) would beat them in the Turing test. Damn, this is so sad.

e-Rater result?

[ackthpt]

Input: All your base are belong to us!

Output: A+

identical results to those of trained readers...

[GGardner]

Maybe these says more about the readers than the computer program?

And the era of virtual teacher begins!

[gnuman99]

We dont ned a techer to marc mi worc anyway. The komputer said I writee good!

Seriously, how can a program replace a human when the program cannot comprehend structure like language? Computers cannot and should not replace English teachers or math teachers (well, beyond grade shool at least!). How can a computer program mark an English paper? How can a computer program check that a mathematical proof is correct? How can a computer program say that a particular train of thought is interesting, or pointless.

This is both good and bad

[n1ywb]

Good: The computer probably won't grade you down for writing an anti-Bush essay, and it probably won't get fired for it. Good: Computers won't play favorites, and you can't kiss up to a computer. Bad: The computer really can't grade you up for expressing original ideas. Bad: It's probably possible to fool the computer somehow.

I can't wait

[proverbialcow]

...until some wiseass figures out a way spoof the grader, probably by sliding under the radar of whatever probabilistic models they've got that pass for spell- and grammar-checkers.

For example:

Flimblarm nif goondatakun, jut sekfar bel shon duc. Seempkin dar goolnac flar tefnek voz toulian; elmpar gef sogquel.

Grade: B+ Your use of double-negatives continues to haunt you, but I'm glad you've gotten over hanging participles.

Copying

[rf0]

What about if one person copies another will it pick that up and flag it?

In Other News

[ThisIsFred]

Indiana parents are the first to buy (en masse) licenses for Essay Constructor Pro v2.0. The software produces essays that are indistinguishable from those written by real students, using the latest screen-scrape-from-Internet 'n' plagiarism-from-non-credible-sources techniques.

Indiana Director of State Board of Ed comments: "Isn't it wonderful how technology is improving education?"

I wanna try, I wanna try!

[lpangelrob2]

So if I did this, what would happen?

The word emission generally means sending something out. Because of this argument, Hauser is a city located in Kootenai County, Idaho. After Idaho, the Liberals formed the government in Alberta for the first 15 years of the province's existence.

Yeah, those were random snippets from the Wikipedia. Who knows? Maybe this technology [macdevcenter.com] got around?

Grade: A+++

This says more about "trained readers"...

[Phurd Phlegm]

than it does about the software, methinks. I imagine it gives helpful hints like the ones I always turn off in Microsoft Turd. Any construction that deviates from the norm in a boring business document apparently triggers the "grammotron" or whatever they call it. A human reader has some appreciation for style and may actually accept something a little different for the sake of variety and sparkle.

Not that there's anything in this post that serves as an example. I guess that's because I was graded by humans. Seriously, I don't recall getting any encouragement in writing back in the '70s in high school, and not much in college. I guess it wouldn't have been any worse if the Grade-O-Vac was inspecting my papers instead of my mostly-marginally-literate teachers. There were several exceptions, but they focused much more on reading than on writing. I suspect they had a lot greater effect that way--I know they had a great effect on me.

No way this is sound

[Hays]

I cannot believe they are doing this. It is undoubtedly a deep AI problem to properly grade an essay. You have to be able to understand if statements connect logically to form an argument, you have to understand analogy, you have to understand sarcasm, etc... this is so far beyond our current capability.

As implemented, I'm sure there are easy ways to scam this system by writing gibberish. Who sold them this idea? I want to know the person responsible for this abuse of computing.

Re:No way this is sound

[IWannaBeAnAC]

Edsger Dijkstra would no doubt have something profoundly and humerously offensive to say to the writers of this software ;-) There doesn't seem to be anyone to take over his mantle, which is presumably why the software industry is going pot at an ever increasing rate. sigh.

Students

[Gettinglucky]

Now all the students need is e-writer so that they can just type in the subject and the score they want to achieve and then e-reader will grade it accordingly!!!

Can students get this program?

[Theovon]

If I were a student, I'd want to get a copy of this software and use it to pre-grade my papers so that I could find out what's wrong and fix it before I turned it in.

India vs. Indiana

[the eric conspiracy]

Wouldn't it just be cheaper to grade the tests at call centers in India? What are those Indians doing when there are no incomming calls? Just slacking off??? They could be grading tests.

Standard Grading

[Tony]

The system was tested over a 2-year pilot program and produced results virtually identical to those of trained readers.

So it gives its favorite students 'A's without reading, least favorite students 'F's, and the rest arbitrary grades somewhere in between to mimic a bell curve?

Excellent!

"Artificial Intelligence is easy. It's artificial stupidity that impresses me." -- Arthur Oscar

Missing feedback

[Hamster Of Death]

How are these system supposed to scribble in the margins and tell you your ideas don't fit together?

How do they judge the content? What if you submit an excellent paper on middle ages history but the assignment was on socialism?

Human feedback is required in order to learn how to write well, you can't just expect a machine to tell you how to improve your writing. Grammar perhaps, but not ideas and how to let them flow coherently.

In order for these students to get that feedback someone has to read it, and since they're reading it anyway, why not just grade it then?

Seems like they are trying to solve the wrong problem with this system, or a problem that dosen't exist. (Are there really so many papers to mark you need a machine to do it?)

Re:Missing feedback

[RzUpAnmsCwrds]

"How are these system supposed to scribble in the margins and tell you your ideas don't fit together?"

They can't. But on a standardized test, you don't get any feedback anyway.

"In order for these students to get that feedback someone has to read it, and since they're reading it anyway, why not just grade it then?"

Because it takes too long.

"Are there really so many papers to mark you need a machine to do it?"

Yes. Human graders for standardized tests get about 1-3 minutes per paper. Human graders don't h

Antidisestablishmentarianism!

[Bluesman]

By creating a vernacular consisting of elongated words and sophisticated verbiage, obviously indifferent to definition but simultaneously observing grammar regulations while eschewing colloquialisms, perhaps students may increase individual chances of achieving substantial academic acclaim.

If this works anything like the writing level indexes you find on word processors, it should be easy to fool.

Too Uniform

[Roger_Wilco]

My writing style is somewhat peculiar, though I can't exactly say how (or even approximately how). Partially as a result of this, my marks in English class over the years of high school ranged from C to A, depending not on me, but on who the teacher was. If the teacher happened to like my style, I got a good mark.

This is annoying, but at least each year there was a different teacher, who may like my style. If the marking is computerised, it will not change; if your writing doesn't fit what the computer likes, you're screwed; likewise, if it does like it, you might never learn to express yourself more creatively (ie you'll be punished for trying to write in a manner different from what you usually do).

There are possibilities in this technology, but I suspect that it will be a long while before the eccentric aren't labeled as poor writers.

Babelfish

[Waffle Iron]

I will not trust computer grading until I see a computer-translated document that isn't laughable.

To illustrate my point, I'll restate it. [English -> German -> English]:

I do not trust the computer, which arranges, until I see a computer-translated document of this laughable isn't.

That's about how well a computer "comprehends" language today.

Whoever approved this should be fired

[Retired Replicant]

A computer can check spelling and even grammar to a certain extent. However, it cannot evaluate factual accuracy, strength of argument. Even with spelling, the computer is not likely to catch improper use of homonyms. I can guarantee you that it will be possible to create a piece of writing that is utter crap that would get an A+ using this or any other possible computerized grading system. Unfortunately, there are probably many teachers out there who make poorer graders than this system does. The answer to the problem of poor-quality teaching is not replacing teachers with computers; the answer is a combination of better teacher pay and putting higher standards in place for our teachers via competency testing.

Essay grading is harder than science grading

[sustik]

I believe that (English) essay grading is harder than grading science exams based on problem solving (no bubbles please), at least if essays are about content and not just grammatical correct sentences.

I say this because there is an objective criteria for grading the solution to a physics or math problem: correctness. For essays I do not beleive that we (and the current state of AI) can come up with an exact criteria like that. You might determine whether an essay is too different from essays which were written by experts, but cannot a very different essay to be just as good?

To my knowledge the AI programs can solve physics problems which are limited to some well defined domain (for example: http://www.cs.utexas.edu/users/novak/cgi/isaacdemo .cgi though that is from 1977...) I am not aware of programs grading physics problem solutions.

I will accept an essay grading program after they grade solutions to math and physics problems.

I conjecture that some writers would feel offended if their essay did well according to the program: they might think it means they are too conformist and conservative and not novel in their approach...

Matyas

From the web site.

[Bluesman]

>If you would like to try out e-rater, you can obtain an ID and password and submit and original essay for scoring on the CriterionSM Web site.

Submit "and" essay? I guess they haven't run the software on themselves.

F.

the triumph of mediocrity

[KMonk]

Writing is not mathematics. Good writing should not go along some artificial standard. Just because my paper is grammatically correct, has a topic sentence, 3 supporting paragraphs, and a conclusion doesn't mean it is good. Good writing needs a flow of ideas from one paragraph to another. It needs finesse, style, grace. This is like an IQ test for english writing. It would do very well in identifying poor writers - but could never identify a great one. I'm sorry ee cummings, your use of punctuation is poor 1/6. There are examples like this in books on taking the various standardized tests - any truly excellent writer is likely going to do badly. Why? The rules of the english language are guidlines, which may be broken when appropriate. This is just the mechanization of another facet of society, and should be tossed out with the rest of the garbage.

As someone procrastinating grading right now...

[Dr. Spork]

I don't think that automatized, high-school-level grading is an all-bad thing. We can call it unfair if we like, but as someone who grades a lot of stuff I can tell you that I'm nothing like fair. I don't always even know how to distinguish a B- from a C+, and I just go with my gut, which, as far as I can tell, is much like flipping an internal coin. If we looked at human grade assigners as an algorithm, we would find a whole lot of stuff wrong, even among those of us who try hard to be fair (cover author names, compare close grades for consistency, keep a constant mood, that sort of stuff).

But I think that if a computer grading program which is no worse than humans could be devised, it would be a great learning tool. A lot of people make it to college as borderline illiterates. I'm not kidding. I read a lot of their crap. That's because their HS teachers were too overworked to grade their writing, so they didn't assign much. If a computer program could auto-grade and give detailed comments on how to improve the writing, high school students could be assigned an essay per week, and really get the hang of writing well. Teachers could focus on teaching instead of tedium.

Sure, the first grading applications are going to make a few serious errors. This is the first stage of every application when a computer is asked to interpret rich data. Early voice recognition sucked. Now it sucks much less, and it will just keep getting better. Same with OCR, chess software, machine translation, etc. So the right debate to have is about when this will be good enough for school use, and not whether. I'm prepared to admit that the answer to the right question is "not yet" (I'm sure how deep the current problems go), but I fully support working on this system until it works right.

Re:As someone procrastinating grading right now...

[Idarubicin]

If a computer program could auto-grade and give detailed comments on how to improve the writing, high school students could be assigned an essay per week, and really get the hang of writing well. Teachers could focus on teaching instead of tedium.

Unfortunately, the system described here doesn't return any such useful feedback. The Indiana system returns a grade from a six point scale. No comments, no criticism, no hint that the evaluation is meaningful.

Incidentally, what's this about "teaching instead of tedium"? Grading essays by evaluating construction, insight, and creativity should be part of the teaching process. Perhaps this is something that should be addressed earlier in the education of these students - if they're reaching college as "borderline illiterates" there is a problem - but grading in general is a part of teaching. If I were a student, I'd want to know that a human being - at some point - had bothered to look at the work that I did.

I took this test

[tundog]

I live in Indiana (no, NOT India) and took this test. Being a techie, I figured I'd try to fake out the system. This test works out to be 10% of the final grade and since I had a 98 going into the test, I figured I could afford to gamble a little, figuring if it back-fired I could blame it on a computer error since every one would figure the kid with a 98 MUST be telling the truth.

I almost wimped out. I wrote about 80 percent of the essay (about influence of pop-culture on society - and silly me I always thought society influences pop-culture but anyway). I had 5 paragraphs - 1 intro, 3 body - 1 half-assed conclusion. I reoreded the paragraphs, copied the one I felt was the best written and pasted it into the body 3 times.

Guess what I got.....6/6 (six point grading scale which is pretty messed up because a 5/6 is an 83%). Hopefully they won't audit mine....

Re:I took this test

[miyako]

The 5 paragraph essay is the cookie-cutter bottom line least common denominator for education these days.
I would estimate 90% of the essays given in the US educational system today are 5-paragraph essays.>br? Essentially, the 5 paragraph essay is a mold consisting of an introduction, 3 body paragraphs each focusing on one supporting fact, and a conclusion, and the teachers give students the ingredients in the form of a topic (definitely something that won't require too much thinking),and a style, The s

Computers can't grade "interesting"!

[Luscious868]

A computer can not replicate certain aspects of the grading process. Sure they can grade spelling and grammar and probably certain aspects of your writing style but there is plenty of important aspects of writing that they can not grade you on.

For instance, does your essay really grab the reader? Anyone here who reads technical documents knows what I'm talking about. There are some writers that, no matter how dull the subject, can make their work interesting and fun to read. A computer can not possibly grade one on that. I have a good friend who's a high school English teacher and occasionally I'll read some of the things written by his students. I've come across plenty of papers that are grammatically correct, have perfect spelling and are fairly well written from a syntactic and stylistic point of view, but are just plain boring to read. Then I'll move on to another paper, about the same subject, which is interesting and actually fun to read.

That's just one example of something a computer can not possible take into account when grading an essay. The bottom line is that a computer will never be able to grade you on certain subjective things, which although they are subjective and therefore open to a certain amount of interpretation depending on the person doing the grading, are nevertheless still very important aspects of good writing.

With spelling and grammar check, almost any average student can churn out a paper that is going to be mostly correct; however it still takes a good writer to produce something interesting. In my opinion, an interesting paper with a few minor spelling, grammar or syntactic errors is just as good as a boring paper with no spelling, grammar or syntactic errors.

How To Write An Essay

[dpbsmith]

A good essay always consists of an introductory paragraph, three body paragraphs, and a closing paragraph.

It is essential that every paragraph begin with a topic sentence. The first paragraph should state the thesis, or point of the essay. Since computers cannot actually understand the entire essay, you can assume that it will only be judging the local coherence of writing which is free to run like a river, past Eve and Adam's, from swerve of shore to bend of bay, taking us by a commodius vicus of recirculation back to Howth Castle and environs.

The second paragraph should make a point that present a countervailing view, the antithesis. Once again, spelling should be correct, the essay should be capable of passing a Microsoft Word grammar check, but after that we pass through grass behind the bush where a gull calls, coming far, ending here. Finn again? Take, but softly memory till thousands are given the keys to a way a lone a last a loved a long the river runs.

The third paragraph should synthesize the material covered in the first two paragraphs. It is, however, important that any material obtained from external sources be modified so that it cannot be detected as an exact match for anything on the Web. So, she went into the garden to cut a lettuce leaf to make an mince pie; and at the same time a great wolverine, coming up the street, goes into the store. "What! No laundry detergent?" So he died, and she very imprudently married the barber, and they all fell to playing the game of catch as catch can till the gunpowder ran out at the heels of their boots.

In conclusion, the final paragraph should recapitulate and summarize what has gone before: since you can be sure that a computer is capable of counting paragraphs, a good essay always consists of five paragraphs. If it has the right number of paragraphs and every word is spelled correctly, you are almost certain to get at least a passing grade.

AP Essay Rubric

[jmichaelg]

Fine and dandy if you're writing for humans who are reading for content. On the other hand, the AP Biology essay rubric assigns points based on how many keywords appear in a response. It doesn't score the essay for sense.

A friend of mine who teaches Biology said that she saw some pretty bad essays which she would have given a poor grade to because the english was atrocious but she had to follow the grading rubric and give high scores to because the keywords were present.

I scored 5/5 on the AP English exam...

[MichaelCrawford]

...and I like to write [geometricvisions.com].

I just signed up for a userid so I can take the exam online, but after submitting my info it said I may have to wait up to two days to get an account.

Curious that they can grade essays with a computer but it looks like they have to have a human pass out the user ids.

Anyway, I'll see if I can submit one of my articles to the exam, and will post here how I did. Since I have to wait for my user ID, you'll have to look back here later to see how I did.

Re:Google Bombing

[superpulpsicle]

Computerized grading is great.
Computerized grading is superb.
Computerized grading is excellent.
Computerized grading is outstanding.
Computerized grading is god.
Computerized grading is great.
Computerized grading is superb.
Computerized grading is excellent.
Computerized grading is outstanding.
Computerized grading is god.

Essay Result = A+

Re:Neat

[James Lewis]

Well, as most people around here already know, if you past the link into google, and then click on the link google gives, NYTimes will let you in without registering.