MIT CS Professor Tests AI's Impact on Educating Programmers (acm.org) 84
Long-time Slashdot reader theodp writes: "The Impact of AI on Computer Science Education" recounts an experiment Eric Klopfer conducted in his undergrad CS class at MIT. He divided the class into three groups and gave them a programming task to solve in the Fortran language, which none of them knew. Reminiscent of how The Three Little Pigs used straw, sticks, and bricks to build their houses with very different results, Klopfer allowed one group to use ChatGPT to solve the problem, while the second group was told to use Meta's Code Llama LLM, and the third group could only use Google. The group that used ChatGPT, predictably, solved the problem quickest, while it took the second group longer to solve it. It took the group using Google even longer, because they had to break the task down into components.
Then, the students were tested on how they solved the problem from memory, and the tables turned. The ChatGPT group "remembered nothing, and they all failed," recalled Klopfer. Meanwhile, half of the Code Llama group passed the test. The group that used Google? Every student passed.
"This is an important educational lesson," said Klopfer. "Working hard and struggling is actually an important way of learning. When you're given an answer, you're not struggling and you're not learning. And when you get more of a complex problem, it's tedious to go back to the beginning of a large language model and troubleshoot it and integrate it." In contrast, breaking the problem into components allows you to use an LLM to work on small aspects, as opposed to trying to use the model for an entire project, he says. "These skills, of how to break down the problem, are critical to learn."
Then, the students were tested on how they solved the problem from memory, and the tables turned. The ChatGPT group "remembered nothing, and they all failed," recalled Klopfer. Meanwhile, half of the Code Llama group passed the test. The group that used Google? Every student passed.
"This is an important educational lesson," said Klopfer. "Working hard and struggling is actually an important way of learning. When you're given an answer, you're not struggling and you're not learning. And when you get more of a complex problem, it's tedious to go back to the beginning of a large language model and troubleshoot it and integrate it." In contrast, breaking the problem into components allows you to use an LLM to work on small aspects, as opposed to trying to use the model for an entire project, he says. "These skills, of how to break down the problem, are critical to learn."
Calc Based Physics (Score:5, Interesting)
It's often asked if physics is absolutely necessary for a CS degree. But the one thing that physics gives you a lot of is practice in breaking down a problem and modeling it. It teaches you about compromises you can make to simplify the problem, and how to account for those inaccuracies.
Similar to how in grade school you are graded more on your work than the answer. It's more important to understand and be able to explain your process as to how you arrived at your answer, this allows you to check that they understand the concepts, and becomes more critical the more complex things become.
Re: (Score:1)
This explains why game physics look so bad then, and computer CG physics look even worse.
Not only should they make CS students study MORE physics, they should probably make them study how to write fiction as well !
Re:Calc Based Physics (Score:4, Insightful)
Most CS students don't become video game developers. The industry is notorious for long hours, low pay, and burnout.
And even those who do become programmers will usually be part of a team where someone else is making the graphics, the plot, etc.
Re:Calc Based Physics (Score:4, Interesting)
Alternatively, CS education could focus on software-development tasks that involve breaking down a problem and modeling it, accounting for inaccuracies, etc.
Physics is one way of teaching those skills of course, but not the only way, and not the most applicable way for someone who is going to be writing software as a career. One exception would be someone who is planning on software development that focuses on physics modelling, in some way.
While I DO agree that there is value in a well-rounded education, much of that value does not translate directly to a trade. It makes sense for people who are looking to work a trade to have their education focus on it, and get "well rounded" after they have pulled themselves out of debt and have an interest.
Re: (Score:2)
Re: (Score:2)
Umnnnnn..... This isn't quite right. Not for any CS class I've ever encountered. Usually they focus on one particular small aspect of CS, and teach you how to handle that. But CS is extremely broad in what problems it covers.
I'm not sure physics is better than carpentry, though, or plumbing, and both of those are easier to "get adequate in" than it physics. Even literary criticism deals with some problems that aren't covered by most specialties (like physics) that an application of CS may need to handle
Re: (Score:2)
Re: (Score:2)
The classes I've encountered were very narrowly focused on one small(?) subdomain. This doesn't mean those specialized courses aren't important, just that they aren't sufficient.
OTOH, CS is too broad for any single class to cover it except at an extremely superficial level. I'm not saying those classes aren't important. Just that the ones I've encountered aren't sufficient. And that you can EXPECT to need to pull in skills from other domains. (And that they ought to teach you that expectation.)
Re: (Score:2)
Re: Calc Based Physics (Score:2)
I've modeled stuff for a pharmaceutical company. I didn't take a chemistry course, I grabbed the nearest domain expert.
Most CS people would be better off with courses in semantics, journalism (the article writing part) and philosophy of science (the part about how to find out what is true and false). Also skills like meeting techniques, interviewing and presenting.
Physics is about the last thing most CS people need.
Re: (Score:2)
That's too extreme. There are lots of domains of CS where a physics background could be useful. (And much of the rest of what you recommend would be good for anyone.)
Re: (Score:2)
Re: (Score:2)
It's often asked if physics is absolutely necessary for a CS degree. But the one thing that physics gives you a lot of is practice in breaking down a problem and modeling it. It teaches you about compromises you can make to simplify the problem, and how to account for those inaccuracies.
I took 1 year of physics in high school; I took a year of CS there too . I took 1 semester of physics in college. Physics made no difference to me in my CS classes. If you want "learning to break problems down" into parts to help solving, that started in my HS algebra 1 class with word problems, then was extended by my HS CS class. My college's CS 301 course (Intro to CS) taught me "top-down programing", which refined previous knowledge and was the ultimate in "breaking problems down". No physics required.
P
Re: (Score:2)
in 2024 you missed the point entirely. you should work on that english, 2040 is around the corner.
Re: (Score:3)
Apparently I do need to work on my English, because I have no idea what you're rambling on about.
Someone has to code it first (Score:5, Insightful)
The only reason ChatGPT can provide solutions is because someone has already done them and publicly provided those solutions for it to copy. For anything novel it'll make a mess and then it's up to your skills to write the solution for ChatGPT to again copy.
Re: (Score:3)
And that is the second problem with the statement by the OP. The first is that as soon as you need to actually do some design or fix some problem in existing code, you need to know the language.
Re: (Score:2)
Re: Someone has to code it first (Score:2)
Exactly . I gave it some fairly trivial problems that i thought it could write in much less time than me. I asked it to write Windows batch files for these simple tasks. It clearly hadn't seen this task before. Even after an hour, I couldn't lead it to water and get it to produce a working script. I would have been far, far better off writing it myself. Same with Gemini, btw. If I asked them to write Python, I would probably have gotten a better result. There are some languages these LLMs seem to suck at p
Re: (Score:2)
Here is a follow-up question. The only reason I can program something in Python is because someone has already done the work and the interpreter is written in C. And they could only write the interpreter in C because somebody else did the work making machine code possible.
While ChatGPT isn't ideal for coding, why couldn't the next programming language be English and interpreted by a LLM interpreter specializing in code?
Re: (Score:2)
Writing in C is not that different than writing in assembly language (also not a assembly is not much of a leap to machine code in logic terms just a mapping), I have done both, sure you have to understand how the stack is used to make function calls, registers etc but it is still the same basic concept of breaking things down. Same with python you loose all the memory management bullshit you have with C but still thinking logically and breaking the problem down is a requirement. If you move to an English a
Re: (Score:1)
I would argue that there are many people out there who never developed those skill sets and are already writing code. Some of it is really bad, buggy, and full of security vulnerabilities. I don't think if they use AI for programming that the quality of their code is likely to decrease.
Why not have a programming AI that allows you to write the outline of the classes and methods along with maybe some required parameters or variables. Then let AI do the coding? Will there be errors, sure, but that isn't anyth
Re: (Score:2)
Re: (Score:1)
But what if you don't have a solid CS education but have a good idea for an app or program? Is it not possible that in the future you could write quality prompts that outline the classes and functions and just let an AI programmer write the specific code? If not, why not?
I have seen (and written) some really bad code over the years. I don't think letting AI write the code is likely to bring the quality down much, especially if it is used by individuals with minimal training in programming who are already wr
Re: (Score:2)
why couldn't the next programming language be English and interpreted by a LLM interpreter specializing in code?
The point is that LLMs aren't capable of doing that.
Maybe AI in the future will be able to do that, but it will be a new technology.
Re: (Score:1)
I used LLM as an example, but I think you are correct. It would likely be a different variation of AI, but my point is really the end result of not having to write in Python because you could just outline the classes and functions and let the AI programmer do the rest.
Re: Someone has to code it first (Score:2)
Re: (Score:2)
IIUC, that's not quite right.
_ !!THIS IS A GUESS!!
What seems to happen is that ChatGPT has swallowed a bunch of text and correlated it. When you ask a question, it fantasizes what the answer could be, comes up with a bunch, and picks the one that looks "best". Frequently this matches reality. It isn't however, because someone else has solved the problem before. It's quite willing to invent fictions that "taste better".
Re: (Score:2)
Yes and if we stop people trying to solve problems in the future there will be no new innovations. The way you learn to solve problems is by solving them, not getting others to solve them for you. Making things easier at some point does more harm than good, we need a challenge in our lives to learn and give it purpose. I think we have passed the point in western society where things are just far to easy, you don't have to get out of our houses to buy food, we don't have to enter a pin on our cards. We are g
Re: (Score:2)
If you want to do more than simplistic coding in Fortran, then yes, you NEED to know the language. Otherwise you will stay stuck on the level of an amateur.
Re: (Score:2)
I'm assuming that modern Fortran has things like strings in this comment, but:
Suppose ChatGPT could answer "Code me up a hash table in Fortran with string keys and (whatever Fortran is for struct) values."? That would be a fairly complex chunk of code, and (at least for the FORTRAN that I remember) well beyond the amateur level.
But standard English is NOT a good programming language. Nobody understands it well enough, because it's got too many inherent ambiguities. And artificial languages intended to so
Re: (Score:1)
While true, that will still not cut it. As somebody who has done several full-custom production hash tables, I know that "hash table" does not represent a "known good" thing as soon as the one that comes with the standard library does not do it anymore. There is just too much design choices and they can have massive impact. And in order to deal with that, you have to know what you are doing and probably read actual books (gasp!) on the topic and run some experiments. What "AI" will do is that there are fewe
Re: (Score:2)
It's not just that there are fewer people who can do it. There are also fewer people who will pay to have it done. When I started out it was expected that programmers would research the algorithms that they would use. Libraries were extremely limited. But that takes time as well as skill and effort. Once the libraries showed up, companies started to not want to pay for that investment of time, which often resulted in sub-optimal choices anyway.
That same thing is continuing to happen, and it will happen
Re:The professor doesn't get it (Score:5, Informative)
Tell me you missed the point of the article, without outright telling me you missed the point of the article.
Hint: you missed the point of the article.
Full explanation: It's not about learning FORTRAN (or other obsolete programming language). The professor just chose FORTRAN because he was quite certain none of his students knew anything about it - ensuring that everybody started from a blank slate. The point is that when presented with a completely foreign topic to learn, the students who retained the most information about it did so because they learned it themselves. The students who used AI to do the work for them were able to recall very little, or in the case of ChatGPT, nothing at all.
Re: (Score:3)
You and everyone else replying to this thread have a big problem. You see, I've been running thousands of lines of Python on a daily basis for the past couple of months, code that generates revenue.
Code that didn't exist until I asked ChatGPT to write it, using carefully-specified prompts in plain English.
Now, here's the big problem for you: I know basically nothing about Python.
What do you say to that?
What will you say when the models get ten times better at programming, or more, over a period of time in
Re: The professor doesn't get it (Score:1)
The models don't get better at programming when you don't get better at prompting. You won't get better at prompting if you're no good at programming.
Re: (Score:1)
Is it possible to get better at prompting without getting better at programming? I would argue that it is. With current models, it is definitely easier to get better at prompting if you get better at programming, but wouldn't say it is necessary.
Re: (Score:2)
Getting better at prompting is getting better at programming. You're 'programming' when you walk up to the counter at Starbucks and order a parameterized coffee beverage. You're 'programming' when you ask your SO to do something specific in the sack. You're 'programming' when you pick up the phone and order a pizza. You're 'programming' when you set your alarm for a certain time. You're 'programming' when you tell the contractor what rooms you want in your house. And you're 'programming' when you tel
Re: (Score:2)
See, the thing is, that's just false. You're wrong, and evidently you don't mind being wrong. There's nowhere for the conversation to go after an assertion like yours is thrown out there. LLMs have obviously gotten MUCH better at turning English -- even sloppy English -- into working code over the last couple of years.
If they haven't gotten better, as you say, then they must have stopped getting better at some point, because at first they couldn't do it at all. When did that happen? When, exactly, did
Re: The professor doesn't get it (Score:3)
Yeah, about that. I recently asked chatgpt to refactor a python program. It did that nicely and it had some useful suggestions. It also left out about 25% of fairly critical code. Maybe it thought it didn't live up to the standards :)
You can use gpt and I do, but it helps a lot if you already know what you're doing. I'm not a python expert though, so that helps: I'm limiting myself to standard constructs that are common to most languages.
Except array slicing. That should become a standard part of every lang
Re: (Score:2)
As for the impact of AI on the CS profession: that's undoubtedly huge, but it's still too early to tell which way it will go. AI may not make programmers obsolete, but if you're a programmer and you are not using AI in your workf
Re: The professor doesn't get it (Score:2)
Re: (Score:2)
Re: (Score:2)
I think you're right, and I also think you're hitting on the key takeaway:
ChatGPT let someone solve a problem quickly, but the person didn't learn the skill.
So, when faced with a task, we should ask ourselves: Do we want to solve this quickly, or do we want to learn the skill?
It used to be we always had to learn the skill to solve the problem. It's interesting that now we have a choice.
Since I, as an individual, have no interest in being a FORTRAN programmer, I'm OK with not learning the skill. Sometimes yo
Flawed test. Not science. (Score:1)
When you force people to use a particular tool you've picked out, definitionally you've created an artificial environment that has results that don't generalize to anything much. For example, "here are three nails in the ground, and you have to use a screwdriver."
Worse, you then don't let the modify the tool, but you select which extra tool they get to use to hold the first tool. (LLM, Google, etc.) So now we have "here are three nails in teh ground, and you have to use a screwdriver, but Team 1, you mus
Re: (Score:1)
The professor doesn't like AI, so he designed a test to make it look bad.
Re: (Score:2, Insightful)
LOL...
No my friend. "AI" is quite capable of making itself look bad all on its own. It doesn't require a professor to rig a test against it.
"Hey ChatGPT, how many letter r's are in the word strawberry?"
Re: (Score:2)
"how many rs are in the word revenge best served cold?"
and get:
"The phrase “revenge best served cold” contains four letter "r"s. If you break it down, you can see them in the positions: r-e-v-e-n-g-e b-e-s-t s-e-r-v-e-d c-o-l-d."
Re: Flawed test. Not science. (Score:3)
Except the professor did not say LLMs were useless.
Rather that they are better used to help solve small parts of a problem rather then just having the entire thing solved by a model. The experiment was more to prove the importance of old-school problem solving skills even in the age of generative models.
Re: (Score:2)
Professor was talking about learning. Learning is different than using something on your work.
Re: (Score:2)
It wasn't the llm that was made to look bad though.
It was human learning skills when using too powerful tools, just as drivers relying on GPS take longer learning the routes they drive.
Re:Flawed test. Not science. (Score:5, Informative)
This isn't science.
we don't have any detailed information about the experiment so i wonder how you would be able to say so. then again as a simple experiment the outcome is intuitive enough and exactly what you would expect: automation gives faster results, the very act of working with information is what enacts changes in the brain, what we would call "learning".
you are welcome to repeat the experiment with "proper science" and report back so we can compare results.
btw, you also completely failed to understand the choice of fortran (even if that's clearly stated in the abstract. did you even read it?).
Re:Flawed test. Not science. (Score:4, Insightful)
The key point you completely and utterly missed is breaking down the problem into discrete pieces. In engineering these pieces are called unit operations. Deciding what to do to get from input to output is usually the hard part. Seldom do you need to invent a whole new process.
Re: (Score:3)
This is actually science and, for a first experiment, pretty good science. Obviously, it needs to be firmed up, but you do not do that at the beginning. The thing being crap here is your grasp of how Science works.
Re: (Score:3)
1) Student aces his university courses, but cheats by using ChatGPT throughout.
2) Student tries to get his first job. Student talks about all the wonderful advanced courses and how those could help make the company $$$. Interviewer is impressed.
3) Interviewer asks student to solve small toy problem right there on the spot, apologizes for the necessary step. Student fails complete
Re: (Score:2)
When you force people to use a particular tool you've picked out, definitionally you've created an artificial environment that has results that don't generalize to anything much.
This is false.
Re: (Score:3)
Re: (Score:2)
Indeed. All I hear in the OP is "I do not like the finding". The experiment itself is pretty nice and well designed. Obviously, it has to be firmed up and repeated, but you do not do that right at the start.
Mutatis mutandis (Score:4, Insightful)
Re: (Score:2)
you might be surprised how many slashdotters don't get it ...
Re: (Score:2)
Re: (Score:2)
Considering the amount of meatheads in the human population I don't find the term meatworld that offensive.
Is meatspace a better or more specific term in this case, I am not certain, I could argue not, but also, probably, yes.
Wait, did I just imply myself a member of said species?
useless test wasting his students time (Score:2)
It's like asking a group to do the same with translating a language.... the people who used a translation app didn't learn, while those that were forced to learn each word, and translate it learned.
the problem as i see it with AI and programming is that - we are currently using it to simply enable anyone to be able to fumble through and be able to have a viable piece of code.
The thing we SHOULD be using it for is to use AI to actually improve coding practices, security and efficiency and I'm not seeing too
Re: (Score:3)
"Fumble through", yes. "Viable piece of code", not so much. I recently did a coding exam where LLMs were allowed (not my decision). On my task I took a somewhat known algorithm and changed it. Guess what, most solutions completely missed the changes, and did so when they were very clearly specified and the whole description could not have been clearer. LLMs at work, missing important details to make _your_ failure possible.
Completely agree on the "wannabe coders". Not a new thing though: https://blog.coding [codinghorror.com]
Re: (Score:2)
well, that's unfortunate, but also not surprising.
fun anecdote- back in high school- students allowed to use their textbook on tests/exams performed worse than those that knew they couldn't use it.
Guess AI is the new open book :p
Good experiment (Score:2)
The results are absolutely no surprise though. "LLMs make you dumb" would be the short version. Or maybe "live by the crutch, die by the crutch". Obviously, when LLMs already mostly prevent you from learning simple skills, forget about learning advanced skills when using LLMs.
My personal experience with LLMs is that I miss all the context and additional topics I find when doping a conventional search. You know, context and things which may come in handy later.
Re: Good experiment (Score:2)
Re: (Score:2)
Only as long as the GPS functions, and not leads you down a route you took yesterday and you know is a dead end because of road work.
Re: Good experiment (Score:2)
Re: (Score:2)
It's still a net win.
Yep, keep telling yourself that. Until you actually find out. Resilience is _valuable_.
Re: (Score:2)
Re: (Score:2)
It really just depends whether they are skills you want to master. I've programmed in python for years and I'll never learn the whole matplotlib API. Not interested. ChatGPT has improved my productivity in getting plots that look how I want.
Google be like (Score:2)
Picasso saw the future (Score:2)
"Computers are useless.They can only give you answers"
- Pablo Picasso
Evolution? (Score:2)
Most people nowadays cannot calculate a square root by hand, and everyone is fine with it, we just use computers to calculate square roots and nobody is considering it a big loss that people cannot calculate square roots by hand any more. Surely some people see LLMs as a next iteration of that same effect, "just ask the computer". One obvious argument against is "LLMs" give incorrect answers" but that has been improving a lot, what are other arguments against?
Re: (Score:2)
The fact of it is it's a pita to calculate square roots. Unless there is some nice algorithm noone ever bothered trying teaching us in 12 years of school. Using a square root in a function and logarithms and exponents to get around them, that we was taught, something a llm could give us the answer to today but we wouldn't learn how to do it.
There is a lot of this conundrum in sci-fi, the more advanced our tools become the fewer people will be able to repair or construct them, and if they get too complicated
Re: (Score:2)
I believe my grandmother learnt the second method described here [wikihow.com]. Possibly my parents as well. It looks a bit complicated because it is described step by step but it is certainly manageable.
YMMV (Score:2)
I take no issue with the conclusion that you will learn more by doing things manually (duh - or at any rate, you will learn more about doing them manually), but I do take a bit of issue with this:
And when you get more of a complex problem, it's tedious to go back to the beginning of a large language model and troubleshoot it and integrate it.
It depends, I suppose. It can be tedious in some scenarios.
In other scenarios, "going back to the beginning and integrating additional complexity" can be astonishingly faster with an LLM than without.
Recent real world example:
"Hey, with smaller amounts of data, this heat map works, but isn't very visually useful
Toy problems (Score:2)
What I worry the most (as someone that used to teach programming) is the loss of toy problems.
In the "old days" a teacher might ask you to whip out solutions to some simple programming problems such as: print all items in an array skipping every Nth item. Or something similar just to get you used to using loops.
I'm fairly certain ChatGPT (etc..) can solve this quite easily. The temptation to use this shortcut might be higher than ever, especially when CoPilot just bursts the answer right at you. So I worry
Misleading: That's "Educating with AI"--its bypass (Score:2)
The results were obvious. This was an example of bypassing education with AI, not educating with AI. I spent a number of years developing techniques to teach using Chatbots. These could be powerful educational tools even before transformer models.
Here is some quick advice:
1. Use the give-take approach of giving a fact or problem-solution and asking a question/giving a problem, in sets.
2. Employee timings based on Graduated Interval Recall where if the answer isn't found within that time frame then give i
nope (Score:2)
Nope, it's not "working hard and struggling." It's simply the need to understand what exactly you are actually doing. For many it's not that hard does not involve struggling; it can be fun, too, even if it's also possible to just be lazy.
If ChatGPT existed ... (Score:2)