Follow Slashdot stories on Twitter


Forgot your password?
Check out the new SourceForge HTML5 internet speed test! No Flash necessary and runs on all devices. ×
Education Technology

The Promise and Limits of 'Learning Analytics' ( 57

jyosim writes: College students always pay attention in class and do all the readings, right? Ok, they probably never did, but today's professors can actually find out how much each student pays attention in a lecture and how much time they spent on readings, thanks to so-called "learning analytics." Some colleges are experimenting with using the data to re-engineer courses hoping students will learn more and retention will improve. Professors get "dashboards" and sophisticated charts, changing their role in the classroom. MIT is an early adopter, assigning post-docs to help professors interpret this new data. As the article on the new Re:Learning project notes, though, "How much can big data actually reveal about something as personal and subjective as learning?"
This discussion has been archived. No new comments can be posted.

The Promise and Limits of 'Learning Analytics'

Comments Filter:
  • "Analytics." Heh.

    • by Anonymous Coward

      You must have gone to college before the rise of "social justice".

      Things are very different now than they were even just a decade ago.

      It's extremely risky for a professor to give anything less than a high grade to a student.

      Suppose a student fails to study, and rightfully deserves an F or whatever the failing grade is.

      Also suppose that the professor goes ahead and gives the student this well-earned failing grade.

      If the student's skin color differs from the professor's, then all the student has to do is clai

      • by no1nose ( 993082 )

        This is where we are at now in the USA :( Hopefully we enjoy the ride down.

      • That's a beautiful, passionate, almost poetic response. Sadly, it is completely wrong. None of that stuff ever happens at any significant rate outside of your fantasy life. In fact, I can't recall having heard of any student who ever made any of those claims, although I'm sure that there must be some, somewhere, that have (personality disorders being as prevalent as they are in the general population).

        Actually, students have a pretty finely honed sense of fair play. If you flunk them and they deserve to flunk, you rarely get more than whimpering and sometimes begging. Ditto for most other grades, and grade boundaries. If you flunk them on a technicality, of course, they will be resentful -- and with some reason. But no, students do not in general argue over grades claiming that the professor is biased against them due to race, creed, color, etc, except possibly in the very rare cases where there is some reason to make the argument.

        As for learning analytics -- I have to say that it is (also sadly) mostly bullshit. I don't know about soft subjects, but in things like math and physics:

        a) It is painfully, oppressively difficult to find a good object instrument to measure "learning" at the college level. And I say this as somebody that has used what there is for upwards of a decade. The instruments themselves are badly flawed and it is impossible to prevent an instructor from teaching to the test if they so desire (indeed, it is difficult NOT to teach to the test if you know what is on it and what weight will be assigned to outcomes in terms of "ranking" teaching/learning performance in the course.

        b) There is nothing like standardization of the courses at the level required to build a uniform instrument that might be of some use. In Europe they have such a thing, supposedly, and too bad for them! If Joe gives a wussy, "physics lite" algebraic physics course but Suzie gives a tough, full calculus course covering exactly the same chapters, how do you even compare them. Now imagine comparing them and developing performance analytics when they don't even cover the same chapters from the same book in the same order and with the same basic understanding of the material they are teaching...

        Here's a single example of the problems we really do face. I give all of my entering physics students an assessment to determine how much they remember of basic math. A page of algebra. A page of simultaneous equations. A page of differential calculus. A page of integral calculus. A bit of vectors and trig. Nothing difficult as far as calculus goes -- one can manage a typical intro physics course with five -- that's right, only five -- integral/derivative rules on board, plus the chain rule/u-substitution, plus the product rule/integration by parts.

        Every student entering the class is supposed to have passed two full semester college calculus courses. Yet the mean score on the assessment is around 50%, with plenty of students scoring as low as 15 to 25%. And these are bright students at a very good university.

        Forget "analytics". The problem is deep, not shallow. It isn't going to be solved by improved statistics on more tests.


        • by Livius ( 318358 )

          None of that stuff ever happens at any significant rate outside of your fantasy life.

          Depends on your definition of 'significant'. It happens enough to frighten people, and the same outcome is achieved.

          All bullying works that way.

          • Except outside you fantasy life it doesn't even happen enough to frighten people.

            I've certainly heard academics complaining about not being able to fail enough people because the university wants to collect its tuition fees, and that's only possible for students still on the course. I've never heard anyone complain that they couldn't flunk someone because of your hypothetical gender and race bullying.

        • by tlhIngan ( 30335 )

          About the only pushback happens not because of race, gender, sexuality, religion or other "social justice" thing. Instead, it's money.

          Sometimes a professor must take back a failing grade - not because the student didn't deserve it, but because their parents either are high ranking members of society (i.e., rich) and thus threatened to withhold their annual contribution, or the administration is worried that students may drop out, and depriving them of tuition funds.

          Forget social justice. Anti-social justice

          • Again, do you have the slightest bit of actual evidence to support this? I know for a fact that students at Duke who's families have donated millions of dollars to the University have been given F's and expelled for cheating. Also, I do not understand these words: "take back a failing grade". No, professors never have to "take back a grade" that was correctly assigned. Duke has a written policy that grades CANNOT be changed once assigned. Most faculty have no idea who the family of any given student i

      • You must have gone to college before the rise of "social justice".

        Possibly, but I was teaching in college after what you consider to be the rise of "social justice". Though quite why social justice is supposed to be a bad thing eludes me. Social justice is what Martin Luther King Jr was fighting for and, well, it worked.

        Things are very different now than they were even just a decade ago.


        It's extremely risky for a professor to give anything less than a high grade to a student.

        I failed a few students on

      • You must have gone to college before the rise of "social justice".

        Things are very different now than they were even just a decade ago.

        It's extremely risky for a professor to give anything less than a high grade to a student.

        Suppose a student fails to study, and rightfully deserves an F or whatever the failing grade is.

        You must have gone to college before the rise of "common sense". This is not about "grades", it's about the quality of teaching. No teacher is perfect, and everyone can improve -- analytics are aimed at telling us what we're doing right and what we're doing wrong, and then we (in theory) give better lessons to all our pupils. Grades should improve as a consequence, but the goal of teaching is not "good grades", it's "learning". Grades are only one way of auditing learning -- a suboptimal way, but to date th

      • Wow. I guess that's why, despite being a white male, I've consistently failed students who did not do the work. Its disappointing to fail 20% of a class, but it certainly happens.

        But my example is nothing. All you have to do is look at the press: every now and then there's a kerfluffle about student cheating and all sections in a class will be failed. And the professors involved are usually white males.

        The reality is that nearly any grading scheme is defensible as long as it is applied uniformly and consist

  • It's great that they are doing this, trying to diversify learning styles and measuring their import, but for some subjects this is next to impossible to do. Engineering subjects aren't terribly amenable to this. These courses boil down to "Can student do calculation X to accomplish Y?" An essay describing the process is not useful. True/false or multiple choice grading are poor options if an instructor really wants to diagnose why the students were unable to get the result. In these classes, the student mus
    • This is just classic data mining. In an increasing number of fields we are now able to collect vast amounts of data. The next steps are a lot harder: processing the data the right way (apply proper statistical methods), and making sense of it all. The latter involves asking the right questions before drawing conclusiions from the data, else you're just playing a variant of "regex golf", resulting in "rules" fitting the data but having zero predictive value outside the dataset at hand.

      Oh, and I have se
      • by rtb61 ( 674572 )

        Problem is the larger the data set, the easier it is to poison with false results produced. So anal tics (who could resist that) the over obsessive use of data by autistic types who have fallen in love with meaningless data manipulations, watching them cycle over and over again, seeing things that are simply not there.

        Problem with student learning anal tics, is there are simply too many external variables, genetics of the student, student family life, student social life, current news and events, student

        • Problem with student learning anal tics, is there are simply too many external variables, genetics of the student, student family life, student social life, current news and events, students diet, students sleep patterns, learning demands of other classes and current assignment load (want to improve that give students more choice, do a report or do an exam).

          That's not a problem with analytics -- it's one of the problems analytics address. With large datasets, you're averaging out all those variables. It doesn't matter why 25% of your students each individually fall asleep during a particular lesson two years running, what matters is that it consistently happens, so there's something that can be changed in your teaching.

    • What we read here is that the recent failures in our education system are not due to Government intervention, policy changes, and a forced influence of what we label progressivism. It's all because the professors don't have the right data to work with, so we need to fix it with statistical methodology. I call bullshit!

      This is an attempt to market bullshit, because people are finally starting to grasp the fact that our education system (including so called Universities) has become bullshit Historically he

      • Saying we now need statistical hullabaloo is laughable. Not even close. We need to get Universities to actually teach people instead of coddle them, and we need professors and Deans willing to do the same. College has become a joke on the public in order to grow government and make a few people rich. Even if you want to teach as a professor, the Deans will not defend you because they fear a student's opinion more than care for the Universities reputation. Yes, that's right. A history teacher can't teach his

        • by s.petry ( 762400 )

          I have to wonder if you went to a one class college, if you went to one at all. You state several times that my statements make no sense, and the only reasoning you provide is a claim that somehow a Professor becomes a god.

          You are choosing to ignore the problem based on what appears to be an extremely shallow personal anecdote.

          Look at the position I presented from a different profession which has similar institutional problems, Law enforcement. Many cops get in to the job because they are altruistic and

    • by AHuxley ( 892839 )
      Some people would do better in a more hands on setting.
      Introduce the history of books and printing. A field trip to see a working linotype machine. Then further learning about how to read a book. Offer a wide selection of books eg: Fyodor Dostoevsky, F. Scott Fitzgerald, George Orwell, William Golding, Charles Dickens, J.D. Salinger, Jane Eyre, William Shakespeare, James Joyce, Anthony Burgess, Edgar Allen Poe...or this decades best sellers.
      While given time and help to read a book introduce a histor
  • Would entry exams not ensure people who can and want to study get in and have the skills to keep learning for a few more years?
    Its not the course.
    The selection of people been allowed to attend should ensure a limited number of places go to people who put in some effort to show they had some study skills in past academic settings.
    "personal and subjective as learning?"
    Years of testing, standardized testing and good education should have sorted that issue out a few years before consideration entry into
  • MIT is an early adopter, assigning post-docs to help professors interpret this new data.

    It's the post-docz fault, really. If they want to be employable they need at LEAST some post-post-doc education / research experience.
    Surely then the old professors will retire / die and let one of the 35-year-old youngins fight over a single assistant professor position.

    If they play their cards right by the time they're 50 they'll be an associate professor, and if they publish, publish, publish they'll make tenure and become a full vampire^w professor.

  • Yes, we're collecting a bunch of data on student learning, and you can do data mining on it.

    Doesn't mean that academics have any time or incentive to do anything based on it.

    Academics are not rated on their actual teaching performance, they are rated on a) grant money brought in, c) research money brought in... y) pass rates, z) student satisfaction surveys. Note the complete absence of whether students actually learn anything as an evaluation criteria.

    But, then, the universities that employ them aren

  • What appears to be being talked about doesn't even really count as analytics. Actual educational analytics requires that the student work out problems on a computer (possibly also using scratch paper), so that it can be noted in detail which particular steps are not being understood and then that addressed.

    Mind you, while I've heard of this being done, I've never actually witnessed it. It sounds like a good idea, but one that would require an extreme amount of time from the teacher/professor unless it was

  • The whole theme of TFA seems to be TL;DR.

    The students find course materials to be TL;DR:

    One of the biggest surprises [a professor] found: Only half the students ever used the home page he had so carefully built for the course. Instead, many students just jumped to the homework, and only clicked to a reading assignment or lecture if they didn't know the answer to a question.

    The professors find reading student assignments to be TL;DR, so they plan to read summaries of stats and cool-looking graphs instead:

    Mr. Chuang was blunt with his colleagues at the conference about the typical faculty reaction to an older experiment with learning analytics, which flagged struggling students at the fifth week of classes, an approach many colleges have experimented with. "The number one thing our faculty would like is an ease in the burden of teaching so they can go back to research," he said.

    But the profs then find the new analytics to be TL;DR, so they need to hire other people to look at them for them:

    How did those professors react? "A big Huh, and Why?, and What can I use this for?" explained Mr. Chuang at a session at recent education-technology conference. "I think part of the challenge is addressing that gap of understanding the potential of analytics," he added. To do that, MIT started a Digital Learning Lab and hired postdoctoral fellows who serve as "ambassadors to a revolution" to help professors interpret the numbers.

    This all seems rather ridiculous, particularly if you look at the ONE practical example in TFA of something useful they seem to have discovered through all of this complex dat

  • Analytics like this might be capable of informing professors what doesn't work: if students aren't paying any attention to something, then either the professor should find a way to get them to pay attention to it, or should just drop it from the curriculum entirely.

    They won't, of course, provide any information about what actually does work. But at least some things that don't work could potentially be eliminated.

  • "This content is available exclusively to Chronicle subscribers"
  • good information for this article. thank you visit my web that discuss about education too education []

Brain off-line, please wait.