Catch up on stories from the past week (and beyond) at the Slashdot story archive

 


Forgot your password?
Close
typodupeerror
×
Education News

How Good Are Robo-Graders? 157

Posted by samzenpus from the reading-robots dept.
stoolpigeon writes "With a large study showing software grades essays as well as humans, but much faster, it might seem that soon humans will be completely out of the loop when it comes to evaluating standardized tests. But Les Perelman, a writing teacher at MIT, has shown the limits of algorithms used for grading with an essay that got a top score from an automated system but contained no relevant information and many inaccuracies. Mr. Perelman outlined his approach for the NY Times after he was given a month to analyze E-Rater, one of the software packages that grades essays."
This discussion has been archived. No new comments can be posted.

How Good Are Robo-Graders? More

How Good Are Robo-Graders?

Comments Filter:

  • More importantly (Score:5, Insightful)

    by crazyjj ( 2598719 ) * on Monday April 23, 2012 @09:15AM (#39770563)

    How quickly will students learn to game the system to get perfect scores with perfect gibberish?

    • How quickly will teachers become completely automated? That's a bit of a scary concept. You can't just have "teachers" who do nothing but press "Play" on a video machine.

      • Re: (Score:3)

        by alen ( 225700 )

        yes you can

        most of the skill of a good teacher is know child psychology and how to handle kids with different issues and different stages of development

        memorizing a few facts is fairly easy

        • Re:More importantly (Score:5, Insightful)

          by masternerdguy ( 2468142 ) on Monday April 23, 2012 @09:59AM (#39771047)
          No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

          • Re:More importantly (Score:5, Insightful)

            by anyGould ( 1295481 ) on Monday April 23, 2012 @10:32AM (#39771529)

            No. This education degree stuff is crap. A teacher should have at least a masters degree in the topic they intend to teach.

            Problem 1: Teachers don't get to choose what classes they get - I knew an English teacher who ended up teaching Intro Computing because.. they needed a computing teacher and he was available. Especially for newer teachers - you teach what they tell you to teach.

            Problem 2: Are you intending to pay all those teachers in accordance with the extra 2+ years of education you're requiring?

            Problem 3: At lower levels, you have A Teacher, not A Math Teacher and An English Teacher. Do you expect your kid's grade 1 teacher to hold multiple degrees? (And see problem #2, expanded to pay for a teacher holding half a dozen post-grad degrees so you feel comfortable letting them teach your kid ABCs.)

          • Even in Europe they only require this from highschool teachers.

            • In the UK (for science at least) it generally works (or did, when I were a lad) that you can teach a subject at a level one below yours in that subject. One that's closely related or contains a major component of it would also count.

              For example, a metallurgy or engineering grad could teach A level physics, as of course a physics grad could. A biology grad who had done A level physics could teach it at O level, but not A. He could teach A level biology.

          • I think a BA or BS in the field should be sufficient for most teachers.
        • "memorizing a few facts is fairly easy"

          If you think that's learning, you are sadly mistaken.

      • Re:More importantly (Score:4, Insightful)

        by TheRaven64 ( 641858 ) on Monday April 23, 2012 @09:50AM (#39770915) Journal
        The tube drivers in London were recently on strike over pay. Their salaries are around £40k (about $65K), but for a decade or so most of the train control has been completely automated: they're just there to press the emergency stop button if there is something wrong with the automated system (which a human will notice but another automated system won't and, for example, cut power to that segment of track). So, judging by the past, teachers that did nothing but press play on a video machine would be better paid than ones that actually taught...
      • I had teachers in both middle school and high school who couldn't figure out how to press play on a VCR. This was 15 years ago.

      • Actually if you can get a lot of that boring grading out of the way, you can free up teachers time to focus on the Human Element.
        While some kids can probably learn better without teachers, others will need them to help guide their education, to spot when they have problems and not learning something to stop and help them get caught up.

        • So that boring reading of essays to see what the students' thought processes are is better shoveled off to a machine that has no concept of what is being written. Go read TFA, it's a pathetic essay. Somehow the last 3 generations of my family's teachers managed to teach and grade and get us all into college. They even spent after hours with the kids who were slower learners. Amazing how standards have fallen.

      • But you could have students that press play themselves while a parent guides them. And instead of DVDs, you could put the videos on the internet so that they are more easily accessible. We have the ability to spread knowledge around the world, yet we still hold on to the archaic idea that knowledge must be hoarded and given only to people that sit in certain classrooms...

        • Re: (Score:2)

          by Omestes ( 471991 )

          yet we still hold on to the archaic idea that knowledge must be hoarded and given only to people that sit in certain classrooms...

          Er... don't you have libraries where you live? Or the internet? Somehow I've managed to learn vast quanitities of things outside of classrooms, and I have two whole rooms full of this "forbidden" knowledge.

          • Good luck getting anyone to take you seriously if you say you have a self-taught university-equivalent education. There's a difference between "hoarded" and "forbidden".

      • Re: (Score:2)

        by mcgrew ( 92797 ) *

        The Fun They Had [wikipedia.org]

    • Re:More importantly (Score:4, Insightful)

      by sglewis100 ( 916818 ) on Monday April 23, 2012 @09:20AM (#39770615)

      How quickly will students learn to game the system to get perfect scores with perfect gibberish?

      Spammers with poor spelling and grammar figured out combinations of gibberish to get around Bayesian spam filtering, I can only imagine relatively smart students will figure out ways to beat the software in time. But hopefully, if people implement systems like this, there will be some checks and balances. Fear of receiving a '0' for a test coupled with having essays randomly graded (smaller numbers) and reviewed / skimmed quickly (larger numbers) ought to be a good start.

      • Here's the fun part - it would entirely depend on how the teacher sells this at the beginning of the year. I could see an argument for unfairness if they're picking out kids for "manual grading" - especially when the difference in marks will be vast (the robo's "this meets all my criteria - A+" vs. the teachers "you spewed out random crap for 500 words - F"). How many teachers (and schools) are going to want to walk into that quagmire?

        Putting on "angry parent hat", the argument would go roughly - Why does m

        • Re:More importantly (Score:4, Insightful)

          by Half-pint HAL ( 718102 ) on Monday April 23, 2012 @10:53AM (#39771875)
          I don't see why this would be different from current auditing practices. If an external examiner finds that your students have been incorrectly marked, it's either an automatic scaling of grades for everyone, or back to the red pen and regrade everything.

          • I don't see why this would be different from current auditing practices. If an external examiner finds that your students have been incorrectly marked, it's either an automatic scaling of grades for everyone, or back to the red pen and regrade everything.

            The difference would be that the robo-grader becomes effectively useless. You can either use it's marks (knowing that kids are gaming the system), or you can do everything manually (removing the benefit that the robo-grader provides.

            Personally, I don't see a problem with the robo-graders being useless.

    • Re:More importantly (Score:5, Funny)

      by Anonymous Coward on Monday April 23, 2012 @09:22AM (#39770633)

      How quickly will students learn to game the system to get perfect scores with perfect gibberish?

      Noooooooooo.

      I had to deal with a Robo grader once during an exam. Time was up and I was still writing. Several large automatic weapons appeared and in a robotic voice it said, "Drop your pen!"

      I did immediately and it said, "Thank you for your cooperation."

      Or that might have been when I was taking an art class taught by Peter Weller [ew.com] .... I don't remember now.

    • 100% A+ Perfect Reply (Score:5, Insightful)

      by MyLongNickName ( 822545 ) on Monday April 23, 2012 @09:24AM (#39770655) Journal

      After thorough consideration of this first post and its contents, I find this I must respond in the most considerate and throughtful way possible. This first post was clearly written before the second post and well in advance of this reply. Based on this, it is only logical to assume that this first post was written before any other posts. This leads me to think that crazyjj was quicker reflexes and reading skills than his compatriots.

      My research has shown that people with quick reflexes make 80% more in real dollar terms than others[1] and are more likely to lead a longer life than their slower reading friends [2]. Clearly crazyjj is at an extreme advantage compared to the rest of slashdot.

      Can America survive with this type of inequality? I think not. We must institue some type of equalizer. Perhaps crazyjj should be given a keyboard with several broken keys. Or perhaps we should simply bash his fingers a few times. In the words of Abraham Lincoln, "A man who types too fast can't be trusted."[3] Abraham Lincoln saw the danger that crazyjj represents and warned us. Will we listen?

    • Re: (Score:3, Insightful)

      by BravoZuluM ( 232200 )

      What does it mean to game the system? The game paper, while not pertaining to the subject, is a well written paper. It is not gibberish. It would take some talent to produce the gamed paper and probably more time. Given that, why wouldn't the student just write an on topic paper?

      Given the bigger picture, writing is an art form. An essay is an art form. Even a human grading the paper might miss the nuances of what is being written. Who can truly say what the author has written is incorrect, when in wr

      • Re: (Score:3, Insightful)

        by bhlowe ( 1803290 )
        A student can game the system by writing their paper, running it through one or more "grading" systems... and making changes until it comes out an "A". Obviously, you would want to do this in a way that it does this while retaining the content and expected "readability" desired.

        The fact is most "jobs" that humans do will be able to be done by a robot or computer. I can easily envision a future where kids get the best personalized teaching experience from a computer "coach"... who can tailor each kid's le

    • Re:More importantly (Score:5, Insightful)

      by NReitzel ( 77941 ) on Monday April 23, 2012 @09:55AM (#39770991) Homepage

      Well, yes.

      E-Rater (a product with which I have some familiarity) is specifically sold to improve form and grammer, and the product explicitly states that it does not grade content.

      So, what you are saying is that the students will figure out how to write with excellent grammar and form, in order to get good grades.

      Well, yeah.

      That's the whole point. That, and the fact that you can have a student write a short essay in 30 minutes, and give them immediate feedback on what they have done wrong, as far as sentence form and grammar are concerned.

      Generally, a student may know what they want to say, and have difficulty putting it into English prose in a way that might convince the reader that they have a clue about that of which they speak.

      Don't think it matters? What kind of result do you think Mr. Churchill might have received if he had stated, "Them Nazis is bad, we gots to beat em."

      Mr. Perelman spent a month of effort carefully crafting an essay that said nothing, eloquently. If our students can do that, more power to them.

      • Re:More importantly (Score:4, Insightful)

        by anyGould ( 1295481 ) on Monday April 23, 2012 @10:46AM (#39771775)

        So, what you are saying is that the students will figure out how to write with excellent grammar and form, in order to get good grades.

        I think that's naive. I think one kid will figure out how to get the computer to kick out excellent grammar and form (a lot easier when you don't actually care about the content), and in short order most of the smart/cunning kids will be using that (the cunning ones because it's a cheap A; the smart ones because they'll want to concentrate on subjects where knowledge matters, as opposed to something that can be outsourced to small shell scripts).

      • Re:More importantly (Score:4, Funny)

        by jc42 ( 318812 ) on Monday April 23, 2012 @11:42AM (#39772647) Homepage Journal

        What kind of result do you think Mr. Churchill might have received if he had stated, "Them Nazis is bad, we gots to beat em."

        Here in the US, we'd just elect him president.

      • Mr. Perelman spent a month of effort carefully crafting an essay that said nothing, eloquently. If our students can do that, more power to them.

        But if you read TFA to the end, you'd see this quote:

        "Two former students who are computer science majors told him that they could design an Android app to generate essays that would receive 6’s from e-Rater."

        ...which kind of defeats the purpose of the exercise. Why would I spend a day trying to craft independent thought if I could get a guaranteed pass for a $0.99 download?

        The marker bot doesn't reward "good writing", it rewards the employ of a few very superficial metrics. Which is like the language exams I've done.

    • How quickly will students learn to game the system to get perfect scores with perfect gibberish?

      Considering TFA already notes that people can (or have) designed Android apps that would automatically generate essays designed to pass the robo-maker?

      If they're using this system at a school, I would be astounded if there *wasn't* an automated system (or twelve) already.

    • Re: (Score:2)

      by Endo13 ( 1000782 )

      I think the better question is how quickly will someone learn to game the system, and come up with a program to generate unique "top quality" essays.

    • Wouldn't the effort of getting a perfect score of gibberish be the same as actually writing a competent well informed and reasoned work?

    • For some people like me, cannabis really is a miracle cure. Maybe not the cure-all that some claim, but for migraine management it has no peer.

      It take 5 minutes to get complete symptomatic relief by using indica-dominant cannabis: no auras, no nausea, no pain, no light-sound-odour sensitivity. And it works all the time.

      Compare that to triptans, "modern medicine" for migraines that manipulate seratonin levels. They take 30-45 agonizing minutes to be absorbed by the stomach. They only work 40-50% of

  • Sorry, human intervention required (Score:5, Insightful)

    by LostCluster ( 625375 ) * on Monday April 23, 2012 @09:16AM (#39770573)

    I don't think auto-graders are a good idea. Where is the information exchange between student and teachers? Teachers need to read student essays not just to assign the grade, but to exchange knowledge with their students Opinions and comments should be two-sided exchanges, if students are writing things that aren't going to be read, how does that work?

    • Re:Sorry, human intervention required (Score:5, Insightful)

      by Dyinobal ( 1427207 ) on Monday April 23, 2012 @09:20AM (#39770611)
      Yep any essay should come back with feed back written on it in the margins/space between lines. Plus I doubt auto graders will mean anything except for kids learning to write a specific way that the auto grader is programmed to grade well.

      • At the same time, I've seen significant flaws in the grading practices of human graders. For instance, I distinctly remember the paper I got back in my college years that said something along the lines of "Really interesting, well written, and insightful. B-". I also remember some essays that were pure unadulterated nonsense that got very high grades (including a 4-week project that I started on during school the day it was due and received an A).

        • Re:Sorry, human intervention required (Score:4, Funny)

          by Zordak ( 123132 ) on Monday April 23, 2012 @11:22AM (#39772333) Homepage Journal

          When I was in high school, we read A Portrait of the Artist as a Young Man. This is literally the worst alleged novel I have ever read. I actively despised it with my entire soul. So I skipped huge chunks of it wherever I figured I could get away with doing so and still pick up the threads of the mostly nonexistent plot.

          When we (finally) finished the thing, we had to write a series of short essays responsive to several prompts. One of the prompts told us to describe the symbolism and significance of the "rose."

          Having skipped huge portions of the book, I had never encountered this purported rose. And I certainly wasn't going to go back and pick through the dense, sophomoric prose to find it. Instead, I figured I could probably pick up some partial credit by saying some random insightful-sounding thing. So I started spewing what English teachers love. I used words like "juxtaposition" and "antithesis" and compared the rose to some other random symbolic object in the book. It was pure, unadulterated, Grade A, premium All-American BS.

          I got an A on the paper. The teacher was particularly profuse in her praise of my short essay on the "rose," commented that I had captured the symbolism of the "rose" perfectly. I couldn't have agreed more.

          • So I started spewing what English teachers love. I used words like "juxtaposition" and "antithesis" and compared the rose to some other random symbolic object in the book. It was pure, unadulterated, Grade A, premium All-American BS.

            I got an A on the paper.

            If you were in the high school class that I teach, you wouldn't have fared so well: I snuff out that "premium All-American BS" as fast as possible. At my school, our "Top 10" students usually include some of the best writers on campus who are generally used to breezing through their English classes with ease--until they reach me. By the time they finish my class and graduate, they (they intelligent ones, anyway) learn that Addressing essay prompts Accurately earns A's and that Filling papers with Fluff ear

            • Re: (Score:2)

              by Zordak ( 123132 )

              But the larger theme is that Portrait itself is such a load of BS that it warrants no better treatment. And not just because it's Literature instead of mere pop fiction. I've read and enjoyed Faulkner and Melville and Shakespeare on my own. In fact, I re-read Billy Budd just a few months ago, with fresh perspective, having recently read a lot of Horatio Hornblower and Jack Aubrey. I don't mind expending effort on those authors because they have interesting things to say. Even in that same class, the next bo

        • Whats the old saying? "If you can't dazzle them with your brilliance, baffle them with your BS." Worked for me through high school and college on those silly assignments. Make some crazy theory up, make sure the paper meets the checklist of requirements for the writing assignment and you get a good grade... even if they can't figure out what the heck you were talking about.

    • Re: (Score:2)

      by fermion ( 181285 )
      These are graders for standardized tests in the middle and high school grades. There is still plenty of time to interact with students. Automated graders, however, provide significant benefit. First the provide the first level of grading to insure that the students is doing what is necessary to pass the end of course or end of semester test. If a computer is going to grade the work, then the student needs to write for the computer. Likewise, the computer grading papers helps insure the teaching is teac

    • Re: (Score:2)

      by olau ( 314197 )

      I don't know what happened where you went to school, but where I did, most of the feedback was a couple of red underlines for spelling/grammar mistakes, and a "good essay" at the end. I wouldn't have noticed if they were produced by a robot.

      I do think essay writing is mostly for your own sake, unless you really have something interesting to say. But then you should be blogging or writing for a newspaper or similar.

  • but how well does it work in the real world (Score:3)

    by LetterRip ( 30937 ) on Monday April 23, 2012 @09:21AM (#39770617)

    While it is true that you can engineer essays to be 'bad' and still score 'good' - the question is - are there natural essays that score good but are actually bad; and good essays that score bad but are actually good.

    Every analysis I've seen suggests that these algorithms do have problems with good essays that are highly creative. Essay graders also have difficulties with this kind of essay - giving drastically varied scores.

    However there doesn't seem to be much evidence of other issues except when an extremely knowledgable issue deliberately trys to make the algorithm fail. Any student or other individual who can do this probably knows that material well enough to 'get an A' if they were to properly apply what they know so this seems like a non issue.

    • Re: (Score:2)

      by Hentes ( 2461350 )

      What's the difference between "natural" and "engineered" essays? If you use algorithmic grading the students will find out, and then the engineered essays will appear.

      While it's true that an average grad student grader can have problems with a creative essay, they can just forward those few cases to the teacher. This behavior, of course, could also be implemented in software, but currently it isn't.

    • While it is true that you can engineer essays to be 'bad' and still score 'good' - the question is - are there natural essays that score good but are actually bad; and good essays that score bad but are actually good.

      Don't misinterpret the results. Just because it wasn't a natural essay doesn't make it a bad example. The prof has shown what the system responds well/badly to, irrespective of mode of writing.

      This whole thing reminds me of a summer school I took in Spanish -- the teacher did a prep session where we were rewarded for using a set of grammar points and connectors that the examiners would be looking for. We had a "good presentation" competition and my team won. Our presentation was practically content free

      • What I don't understand about this software is how does it distinguish between content, structure, and grammar? It's one thing for software to grade structure, especially if it's a rigid 5- format. It gets tricky with grammar but is doable. But here you still run into little quirky problems: When, if ever, does the program allow a sentence to end with a preposition? When, if ever, will it allow a sentence to begin with 'but,' 'and,' or 'because?'

        Content's the one I can't understand. For example, take an arg

  • Human vs. Software (Score:5, Insightful)

    by Anti_Climax ( 447121 ) on Monday April 23, 2012 @09:25AM (#39770657)

    But Les Perelman, a writing teacher at MIT, has shown the limits of algorithms used for grading with an essay that got a top score from an automated system but contained no relevant information and many inaccuracies.

    Considering the fake generated paper [googleusercontent.com] that was published in a peer reviewed journal, I'd say that means the robo-graders are on par with human proof readers.

    • Re: (Score:2)

      by sribe ( 304414 )

      ... I'd say that means the robo-graders are on par with human proof readers.

      That's an effective humorous post you made, but in the story you referenced it appears the peer review was a lie, that no human read the submission prior to acceptance.

    • There were no human proof readers and there was no peer review and there was no publication.

      Did you not read the article you posted a link to? Or is the deception intentional?

    • Re: (Score:2)

      by gweihir ( 88907 )

      There are low-quality conferences that accept everything. The paper was published at such a conference.

      • No, the paper was not published and was not accepted at a conference. According to the article, the authors received word that the fake, computer generated paper had "been accepted for publication after peer-reviewing process in TOISCIJ [The Open Information Science Journal]".

        They didn't take the hoax any further, though:

        Davis said that he considered scraping together the $800 to see if Bentham would actually publish the fake paper, but considered that taking the hoax further would be "unethical."

        "I think that the point has been made," he said. "And, I mean, it's $800, and I'm a graduate student."

        The paper is clearly nonsense; here's a few lines from the beginning:

        "Compact symmetries and compilers have garnered tremendous interest from both futurists and biologists in the last several years. The flaw of this type of solution, however, is that DHTs can be made empathic, large-scale, and extensible. Along these same lines, the drawback of this type of approach, however, is that active networks and SMPs can agree to fix this riddle."

        • Re: (Score:2)

          by gweihir ( 88907 )

          Ah, yes. I should have said "accepted". My last information was that they were scraping together those $800 to go to the conference.

    • You got me - I generated this post algorithmically... Guess I need to work on it.

    • The relevant journal is a non-traditional "open access journal" where articles are freely available (pseudo-random sample [benthamscience.com]; others here [benthamscience.com]), but article authors pay the publisher to publish. It's similar to self-publishing. I imagine TOISCIJ is not respected at all since in a brief search the only info I could find on it was related to the fake paper incident. While it is technically a "peer reviewed journal" (or at least it calls itself that, present evidence to the contrary), it's misleading not to immediatel

  • Robo-graders? (Score:5, Insightful)

    by Anonymous Coward on Monday April 23, 2012 @09:26AM (#39770673)

    So you're telling me we've not only solved the natural language problem, we're also wasting it on grading essays?

    We're not even close. Robo-grading essays is not only cheating, it's probably the worst disservice a school could do to its students. When you grade an essay you're looking at far more than technical accuracy (spelling, word count, formatting, valid citations). You're looking for meaning, articulation and interesting points of view. Robots can't teach critical analysis, can't offer helpful critiques of writing style, and certainly can't make judgement calls on how "good" an essay is.

    • The problem is human graders at the high school level only look at the things this program looks at. I've read and graded the kinds of 5 paragraph theme essays they are talking about and we don't look at content. It's sad you can replace SUBJECT in those essays with any noun (frogs, cars, china, hellcats) and the essay makes the same amount of sense.

      • Which should reveal that the problem is with the test, not with the mechanism of grading.

        • Unfortunately, with the teacher to student ratio most schools have today, there just isn't time to fix that problem...

          • Could you clarify? What do student-teacher ratios have to do with method of testing? I can see some very loose connections there, but perhaps you have a clearer picture than I do.

            • If 1 teacher has 180 students (6 classes, 30 students per class), he can not have time to look at the content of each essay portion of the tests for each student.

              • Ok, and coming back to what I originally posted, that's a grading problem, not a nature of the test problem, but I believe there is an underlying problem in what they are trying to test.

    • I'd rather be graded by a robo grader than a real person. They are more consistent and aren't biased towards giving the students they like better grades. Yes they have their short comings but teachers do also.
      • Perhaps we could have such things submitted via computers, and the computer knows who submitted the paper and attaches the grade to that student, but the teacher does not know.

  • There may be situations in which simply getting a grade is of use, but, in most cases, I'd have thought that getting feedback was as important as getting the grade — knowing I have a good essay is one thing, but knowing where I went wrong, with guidance from someone skilled in the area, is the most important thing, since, otherwise, I have to guess as to where I need to improve.

  • Its like any auto-text parsing - it gets it wrong. (Score:3)

    by Chrisq ( 894406 ) on Monday April 23, 2012 @09:36AM (#39770761)
    Our "corporate firewall" frequently gets things wrong. A site on "Sharp calculators" was classified as a weapons site, though I would imagine that stabbing anyone with one would be difficult. A "security software slap-down" was classed as "tasteless and violence", though no security software was injured. In short robo-graders are probably only any good for politicians, where the content doesn't matter as long as its delivered right.

  • It a teacher is going to phone it in what does that tell the class? Why should a student even bother to write a paper? Maybe students should have auto-generation software.

    For all of the things we screw up in the US one thing we've done (mostly) right is college education. People travel from all over the world to go to school in the US.

    It's shit like this that will ruin it.

    • It a teacher is going to phone it in what does that tell the class?

      Since when are teachers with any relationship to the students involved in standardized tests, except as proctors?

      • Re: (Score:2)

        by Picass0 ( 147474 )

        The scope of software assisted grading goes beyond standardized testing. We've not talking about software looking at what ovals you filled in. It's using in grading essays, book reports, research papers, and I shudder to think perhaps even thesis papers.

  • free graders to jusdge content (Score:4, Insightful)

    by peter303 ( 12292 ) on Monday April 23, 2012 @09:53AM (#39770977)
    The rob grader can check spelling, grammar, structural style. The human grader can check for content accuracy and essay quality and creativity.

  • beast way to fool robot is to learn how to write (Score:3)

    by peter303 ( 12292 ) on Monday April 23, 2012 @09:55AM (#39770999)
    I dont worry too much about gaming the system. To "fool" the grader you'll have to learn spelling, grammar and structural style - exactly what the test-makers want.

  • News good. Paywall bad. A Google News search for the first couple of paragraphs should bring up either the NYT article or another copy of it.

    Note that "em-dashes" have been changed to hyphens and "curly" apostrophes and quotation marks have been changed to "straight" versions marks to accommodate /. as viewed in my browser. Please avoid blocks of text that have -, ', or " when selecting text for search engines.

    --cut here--
    Testing Absurdities, Reading Worries and Robo-Grading
    April 23, 2012, 8:19 a.m.
    By Mar

  • When I saw the title I thought it was referring to those robot graders that they use to level the road substrate whilst making roads (A new bridge is being built near my work) They are quite fascinating to watch work but I wouldn't want to get in the way of one of them.
  • In Soviet Russia:

    Grades rob you!

    We now return you to normal discussion.

  • Without strong AI, robo-graders are worthless (Score:3)

    by gweihir ( 88907 ) on Monday April 23, 2012 @10:10AM (#39771201)

    Unless what you teach the students is worthless as well. If it is just conformance to secondary things like spelling, basic grammar, sentence-length, superficial structure, etc. then robo-grading will do fine. Of course, none of the students being taught this way will learn to write anything of worth, ever. For that you need a competent and intelligent human being (or at least an equivalent intelligence) that understands what the student was trying to say and whether he/she succeeded or not, and why precisely. Grading involves as its most important component the feedback to the student, the actual grade is secondary and does not help the student improve his/her writing at all.

  • ... knows its being graded, it just gets nervous and starts giving wrong answers.

  • As someone who is working in linguistics close to AI research I can attest that the whole idea of automated grading of essays is completely ridiculous and if it is indeed used as the post suggests will likely ruin generations of students. Apart from not working, it is also wrong in various other respects such as sending the wrong signals to young students, implicitly ridiculing the hard work that writing actually is, saving money in the wrong place, and so forth.

    I mean, com'on ... all of the above is so obv

  • This isn't robo-grader specific... (Score:4, Interesting)

    by digitalsolo ( 1175321 ) on Monday April 23, 2012 @10:48AM (#39771795) Homepage
    This problem is not specific to robo-graders. I made a solid rule of finding topics that I found interesting -and- were highly unlikely to be areas of specialty for the teacher/professor/TA grading the paper. It took slightly more effort to find the "right" topics, but it more than paid off in the long run, since the likelihood of the average test grader spending days researching every 10+ page paper they are grading is pretty low.

    Obviously as your volume of large papers and required topics narrows this becomes less effective, but it's quite a good system in high school through most of undergrad studies. I guess I assumed most people did this. FWIW, I did write pretty good papers, they weren't full of B.S. (well, just average volumes of B.S.) but by getting the topic as far "out" as possible, it helped minimize criticism outside of the basic structure, citation, etc.

    • I'll reply to you.

      To me, that's at least part of the "educational game". If you were really given carte blanche on topics, then chops to you for writing about the role of malnutrition in Ancient Egypt or something. No matter how exhausted, a Teacher-person looked at it, used their gut guess to decide it wasn't total spam, and gave it a grade.

      Being graded by Robo-Graders just thunders "Belly of the Beast" and is so dehumanizing that it begs the smarter students to play Beat the System with the funniest paper

  • Are robo-graders as good as or better than human graders because the quality of the human graders is so low? When you have literally millions of SAT essays to grade, you can't afford to be choosy with your staff and as a result the quality of the work is depressed.

  • The article reveals frightening things about how colleges are structured:

    They talk about how accurate the robo-graders are:

    Computer scoring produced “virtually identical levels of accuracy, with the software in some cases proving to be more reliable,” according to a University of Akron news release.

    That's amazing! So let us see why they are so good:

    Graders working as quickly as they can — the Pearson education company expects readers to spend no more than two to three minutes per essay— might be capable of scoring 30 writing samples in an hour.

    Aha! So it isn't that the robo-graders are as good as human graders. The robo-graders are as accurate as a person who is not given enough time to read the actual essay. So if I create a robo-surgeon that is as good as a surgeon who has only 5 minutes to perform open-heart surgery, can I then say that my robo-surgeon is as good as a

  • I think computers have the ability to automate huge areas people think require 'judgment'. Will they be perfect or catch odd cases? Probably not. Yet, that must be weighed against the ability to provide the service on mass.

    For example, radiographers are currently some of the highest paid medical professionals. Today, automated detection is already quite high in terms of accuracy (80%+). About the same as human radiographers. For example.
    http://www.breastcancer.org/symptoms/testing/new_research/200810 [breastcancer.org]

  • Actually, the results of the essay evaluation - that form is valued over content, that eloquence is valued over truth - strongly mirrors my own experiences in academia. So many of the "soft" arts are either teaching how to put a shiny veneer over a turd, or simply an evaluation of how closely the student's expressed beliefs match their professor's. Form exceeds function; indoctrination exceeds learning. We're coming full circle, aren't we?

    Just try expressing libertarian or conservative views on campus the

Slashdot Top Deals

Genius is ten percent inspiration and fifty percent capital gains.

Close