The Promise and Limits of 'Learning Analytics' (shar.es) 57
jyosim writes: College students always pay attention in class and do all the readings, right? Ok, they probably never did, but today's professors can actually find out how much each student pays attention in a lecture and how much time they spent on readings, thanks to so-called "learning analytics." Some colleges are experimenting with using the data to re-engineer courses hoping students will learn more and retention will improve. Professors get "dashboards" and sophisticated charts, changing their role in the classroom. MIT is an early adopter, assigning post-docs to help professors interpret this new data. As the article on the new Re:Learning project notes, though, "How much can big data actually reveal about something as personal and subjective as learning?"
We used to call them "grades" (Score:2)
"Analytics." Heh.
"Social justice" made giving bad grades dangerous (Score:2, Interesting)
You must have gone to college before the rise of "social justice".
Things are very different now than they were even just a decade ago.
It's extremely risky for a professor to give anything less than a high grade to a student.
Suppose a student fails to study, and rightfully deserves an F or whatever the failing grade is.
Also suppose that the professor goes ahead and gives the student this well-earned failing grade.
If the student's skin color differs from the professor's, then all the student has to do is clai
Re: (Score:2)
This is where we are at now in the USA :( Hopefully we enjoy the ride down.
Re:"Social justice" made giving bad grades dangero (Score:4, Interesting)
That's a beautiful, passionate, almost poetic response. Sadly, it is completely wrong. None of that stuff ever happens at any significant rate outside of your fantasy life. In fact, I can't recall having heard of any student who ever made any of those claims, although I'm sure that there must be some, somewhere, that have (personality disorders being as prevalent as they are in the general population).
Actually, students have a pretty finely honed sense of fair play. If you flunk them and they deserve to flunk, you rarely get more than whimpering and sometimes begging. Ditto for most other grades, and grade boundaries. If you flunk them on a technicality, of course, they will be resentful -- and with some reason. But no, students do not in general argue over grades claiming that the professor is biased against them due to race, creed, color, etc, except possibly in the very rare cases where there is some reason to make the argument.
As for learning analytics -- I have to say that it is (also sadly) mostly bullshit. I don't know about soft subjects, but in things like math and physics:
a) It is painfully, oppressively difficult to find a good object instrument to measure "learning" at the college level. And I say this as somebody that has used what there is for upwards of a decade. The instruments themselves are badly flawed and it is impossible to prevent an instructor from teaching to the test if they so desire (indeed, it is difficult NOT to teach to the test if you know what is on it and what weight will be assigned to outcomes in terms of "ranking" teaching/learning performance in the course.
b) There is nothing like standardization of the courses at the level required to build a uniform instrument that might be of some use. In Europe they have such a thing, supposedly, and too bad for them! If Joe gives a wussy, "physics lite" algebraic physics course but Suzie gives a tough, full calculus course covering exactly the same chapters, how do you even compare them. Now imagine comparing them and developing performance analytics when they don't even cover the same chapters from the same book in the same order and with the same basic understanding of the material they are teaching...
Here's a single example of the problems we really do face. I give all of my entering physics students an assessment to determine how much they remember of basic math. A page of algebra. A page of simultaneous equations. A page of differential calculus. A page of integral calculus. A bit of vectors and trig. Nothing difficult as far as calculus goes -- one can manage a typical intro physics course with five -- that's right, only five -- integral/derivative rules on board, plus the chain rule/u-substitution, plus the product rule/integration by parts.
Every student entering the class is supposed to have passed two full semester college calculus courses. Yet the mean score on the assessment is around 50%, with plenty of students scoring as low as 15 to 25%. And these are bright students at a very good university.
Forget "analytics". The problem is deep, not shallow. It isn't going to be solved by improved statistics on more tests.
rgb
Re: (Score:2)
None of that stuff ever happens at any significant rate outside of your fantasy life.
Depends on your definition of 'significant'. It happens enough to frighten people, and the same outcome is achieved.
All bullying works that way.
Re: (Score:2)
Except outside you fantasy life it doesn't even happen enough to frighten people.
I've certainly heard academics complaining about not being able to fail enough people because the university wants to collect its tuition fees, and that's only possible for students still on the course. I've never heard anyone complain that they couldn't flunk someone because of your hypothetical gender and race bullying.
Re: (Score:2)
... and where were grades raised in that discussion? Oh, right, they weren't.
The issue raised was a perceived lack of concern about racial issues on the part of the university administration. Every single instance I recall seeing reported was outside of the classroom (and some off of the campus).
Re: (Score:2)
About the only pushback happens not because of race, gender, sexuality, religion or other "social justice" thing. Instead, it's money.
Sometimes a professor must take back a failing grade - not because the student didn't deserve it, but because their parents either are high ranking members of society (i.e., rich) and thus threatened to withhold their annual contribution, or the administration is worried that students may drop out, and depriving them of tuition funds.
Forget social justice. Anti-social justice
Re: (Score:2)
Again, do you have the slightest bit of actual evidence to support this? I know for a fact that students at Duke who's families have donated millions of dollars to the University have been given F's and expelled for cheating. Also, I do not understand these words: "take back a failing grade". No, professors never have to "take back a grade" that was correctly assigned. Duke has a written policy that grades CANNOT be changed once assigned. Most faculty have no idea who the family of any given student i
Re: (Score:2)
You must have gone to college before the rise of "social justice".
Possibly, but I was teaching in college after what you consider to be the rise of "social justice". Though quite why social justice is supposed to be a bad thing eludes me. Social justice is what Martin Luther King Jr was fighting for and, well, it worked.
Things are very different now than they were even just a decade ago.
Nope.
It's extremely risky for a professor to give anything less than a high grade to a student.
I failed a few students on
Re: (Score:2)
You must have gone to college before the rise of "social justice".
Things are very different now than they were even just a decade ago.
It's extremely risky for a professor to give anything less than a high grade to a student.
Suppose a student fails to study, and rightfully deserves an F or whatever the failing grade is.
You must have gone to college before the rise of "common sense". This is not about "grades", it's about the quality of teaching. No teacher is perfect, and everyone can improve -- analytics are aimed at telling us what we're doing right and what we're doing wrong, and then we (in theory) give better lessons to all our pupils. Grades should improve as a consequence, but the goal of teaching is not "good grades", it's "learning". Grades are only one way of auditing learning -- a suboptimal way, but to date th
Re: (Score:2)
Wow. I guess that's why, despite being a white male, I've consistently failed students who did not do the work. Its disappointing to fail 20% of a class, but it certainly happens.
But my example is nothing. All you have to do is look at the press: every now and then there's a kerfluffle about student cheating and all sections in a class will be failed. And the professors involved are usually white males.
The reality is that nearly any grading scheme is defensible as long as it is applied uniformly and consist
Re:Maybe they'll make a shocking realization (Score:4, Insightful)
Most professors are bad at teaching.
When I went to college, the best instructors were professional lecturers, the second best were grad students, and the professors were the worst. Why do universities waste the talents of brilliant researchers by requiring them to teach introductory courses?
Also: How hard would it be to write an app to game these analytics, and make the professor think you are doing the reading, when you are actually goofing off? It should just be some JavaScript to generate randomly timed scroll events.
Re: (Score:2)
It all depends on where you are at, which department is involved and the faculty pool that is available.
When I was a student I happened to take Soc 101 from the department chair. Normally she taught graduate level classes, but something had happened (I never did know just what) so that she taught the intro class. It was a large class, typical of required introductory classes. It was clear that the vast majority of the students hated it, but I loved it. She had a hard time slowing down to high school graduat
Must be nice to teach in the humanities. (Score:2)
Re: (Score:2)
Every new requirement to punch data into some dashboard or portal reduces a professor's time to teach. Yes many are bad -- they will continue to be so. But the good ones are already good, and this will just waste their time.
...which is why good anayltics gathers the data for you and does the analysis. Trivial example: Auto Multiple Choice [gna.org] is a piece of software that generates multiple-choice question papers for you, then marks the completed papers and generates individual feedback sheets. It also generates a CSV summary of correct and incorrect answers. Whenever I've marked tests by hand, I haven't had time to capture that sort of detail, so at the end of the process I only have a very fuzzy idea of what my students had diffic
Re: (Score:1)
This is Slashdot. The URL to your site is GNA.org. We're not gonna click that! (Unless we're gay or black.)
In all seriousness, after I retired and moved to Maine I was invited to teach at UMF. I taught a basic maths and two advanced mathematics courses. I did this for two semesters. Oh the things I have to say... But, now is not that time.
Suffice to say, even with such a limited course load - I still lacked time to give the attention that I felt the kids deserved (even if they didn't know it, but I digress)
Re: (Score:1)
Oh, I know - I figure they're just trying to trick me! (I did click the link. Looks interesting.) I just couldn't help but notice it was GNA and just missing a letter.
Re: (Score:2)
Oh, and I have se
Re: (Score:3)
Problem is the larger the data set, the easier it is to poison with false results produced. So anal tics (who could resist that) the over obsessive use of data by autistic types who have fallen in love with meaningless data manipulations, watching them cycle over and over again, seeing things that are simply not there.
Problem with student learning anal tics, is there are simply too many external variables, genetics of the student, student family life, student social life, current news and events, student
Re: (Score:2)
Problem with student learning anal tics, is there are simply too many external variables, genetics of the student, student family life, student social life, current news and events, students diet, students sleep patterns, learning demands of other classes and current assignment load (want to improve that give students more choice, do a report or do an exam).
That's not a problem with analytics -- it's one of the problems analytics address. With large datasets, you're averaging out all those variables. It doesn't matter why 25% of your students each individually fall asleep during a particular lesson two years running, what matters is that it consistently happens, so there's something that can be changed in your teaching.
Bah, more Rubbish! (Score:2)
What we read here is that the recent failures in our education system are not due to Government intervention, policy changes, and a forced influence of what we label progressivism. It's all because the professors don't have the right data to work with, so we need to fix it with statistical methodology. I call bullshit!
This is an attempt to market bullshit, because people are finally starting to grasp the fact that our education system (including so called Universities) has become bullshit Historically he
Re: (Score:2)
Saying we now need statistical hullabaloo is laughable. Not even close. We need to get Universities to actually teach people instead of coddle them, and we need professors and Deans willing to do the same. College has become a joke on the public in order to grow government and make a few people rich. Even if you want to teach as a professor, the Deans will not defend you because they fear a student's opinion more than care for the Universities reputation. Yes, that's right. A history teacher can't teach his
Re: (Score:2)
Re: (Score:2)
I have to wonder if you went to a one class college, if you went to one at all. You state several times that my statements make no sense, and the only reasoning you provide is a claim that somehow a Professor becomes a god.
You are choosing to ignore the problem based on what appears to be an extremely shallow personal anecdote.
Look at the position I presented from a different profession which has similar institutional problems, Law enforcement. Many cops get in to the job because they are altruistic and
Re: (Score:2)
Introduce the history of books and printing. A field trip to see a working linotype machine. Then further learning about how to read a book. Offer a wide selection of books eg: Fyodor Dostoevsky, F. Scott Fitzgerald, George Orwell, William Golding, Charles Dickens, J.D. Salinger, Jane Eyre, William Shakespeare, James Joyce, Anthony Burgess, Edgar Allen Poe...or this decades best sellers.
While given time and help to read a book introduce a histor
No, that's not the biggest problem (Score:2)
The problem is the attempt to correlate specific activities with learning, when in fact they are not the same thing. Collecting data on how often a person hits the nail with a hammer has no bearing on whether or not they can build a shelf. How difficult is it to teach a monkey to use a mouse? Do you somehow believe that the monkey who uses a mouse better than a person learned more history in a class? Even more appropriate, how easy is it to write a program which reports events that never occurred to a r
Merit (Score:2)
Its not the course.
The selection of people been allowed to attend should ensure a limited number of places go to people who put in some effort to show they had some study skills in past academic settings.
"personal and subjective as learning?"
Years of testing, standardized testing and good education should have sorted that issue out a few years before consideration entry into
Post Docz (Score:2)
MIT is an early adopter, assigning post-docs to help professors interpret this new data.
It's the post-docz fault, really. If they want to be employable they need at LEAST some post-post-doc education / research experience.
Surely then the old professors will retire / die and let one of the 35-year-old youngins fight over a single assistant professor position.
If they play their cards right by the time they're 50 they'll be an associate professor, and if they publish, publish, publish they'll make tenure and become a full vampire^w professor.
Yes, but (Score:2)
Doesn't mean that academics have any time or incentive to do anything based on it.
Academics are not rated on their actual teaching performance, they are rated on a) grant money brought in, c) research money brought in... y) pass rates, z) student satisfaction surveys. Note the complete absence of whether students actually learn anything as an evaluation criteria.
But, then, the universities that employ them aren
That is NOT the promise of analytics. (Score:2)
What appears to be being talked about doesn't even really count as analytics. Actual educational analytics requires that the student work out problems on a computer (possibly also using scratch paper), so that it can be noted in detail which particular steps are not being understood and then that addressed.
Mind you, while I've heard of this being done, I've never actually witnessed it. It sounds like a good idea, but one that would require an extreme amount of time from the teacher/professor unless it was
TL;DR (Score:2)
The whole theme of TFA seems to be TL;DR.
The students find course materials to be TL;DR:
One of the biggest surprises [a professor] found: Only half the students ever used the home page he had so carefully built for the course. Instead, many students just jumped to the homework, and only clicked to a reading assignment or lecture if they didn't know the answer to a question.
The professors find reading student assignments to be TL;DR, so they plan to read summaries of stats and cool-looking graphs instead:
Mr. Chuang was blunt with his colleagues at the conference about the typical faculty reaction to an older experiment with learning analytics, which flagged struggling students at the fifth week of classes, an approach many colleges have experimented with. "The number one thing our faculty would like is an ease in the burden of teaching so they can go back to research," he said.
But the profs then find the new analytics to be TL;DR, so they need to hire other people to look at them for them:
How did those professors react? "A big Huh, and Why?, and What can I use this for?" explained Mr. Chuang at a session at recent education-technology conference. "I think part of the challenge is addressing that gap of understanding the potential of analytics," he added. To do that, MIT started a Digital Learning Lab and hired postdoctoral fellows who serve as "ambassadors to a revolution" to help professors interpret the numbers.
This all seems rather ridiculous, particularly if you look at the ONE practical example in TFA of something useful they seem to have discovered through all of this complex dat
Could be useful, but limited. (Score:2)
Analytics like this might be capable of informing professors what doesn't work: if students aren't paying any attention to something, then either the professor should find a way to get them to pay attention to it, or should just drop it from the curriculum entirely.
They won't, of course, provide any information about what actually does work. But at least some things that don't work could potentially be eliminated.
Non-paywalled link please? (Score:2)
great information! (Score:1)