ChatGPT Passes MBA Exam Given By a Wharton Professor (nbcnews.com) 155
An anonymous reader quotes a report from NBC News: New research (PDF) conducted by a professor at University of Pennsylvania's Wharton School found that the artificial intelligence-driven chatbot GPT-3 was able to pass the final exam for the school's Master of Business Administration (MBA) program. Professor Christian Terwiesch, who authored the research paper "Would Chat GPT3 Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course," said that the bot scored between a B- and B on the exam.
The bot's score, Terwiesch wrote, shows its "remarkable ability to automate some of the skills of highly compensated knowledge workers in general and specifically the knowledge workers in the jobs held by MBA graduates including analysts, managers, and consultants." The bot did an "amazing job at basic operations management and process analysis questions including those that are based on case studies," Terwiesch wrote in the paper, which was published on Jan. 17. He also said the bot's explanations were "excellent." The bot is also "remarkably good at modifying its answers in response to human hints," he concluded.
While Chat GPT3's results were impressive, Terwiesch noted that Chat GPT3 "at times makes surprising mistakes in relatively simple calculations at the level of 6th grade Math." The present version of Chat GPT is "not capable of handling more advanced process analysis questions, even when they are based on fairly standard templates," Terwiesch added. "This includes process flows with multiple products and problems with stochastic effects such as demand variability." Still, Terwiesch said ChatGPT3's performance on the test has "important implications for business school education, including the need for exam policies, curriculum design focusing on collaboration between human and AI, opportunities to simulate real world decision making processes, the need to teach creative problem solving, improved teaching productivity, and more." The latest findings come as educators become increasingly concerned that AI chatbots like ChatGPT could inspire cheating. Earlier this month, New York City's education department banned access to ChatGPT. While the education department cited "safety and accuracy" as reasons for the decision, the Washington Post notes how some teachers are "in a near-panic" about the technology enabling students to cheat on assignments.
Yesterday, for example, The Stanford Daily reported that a large number of Stanford students have already used ChatGPT on their final exams. It's prompting anti-plagiarism software Turnitin to build a tool to detect text generated by AI.
The bot's score, Terwiesch wrote, shows its "remarkable ability to automate some of the skills of highly compensated knowledge workers in general and specifically the knowledge workers in the jobs held by MBA graduates including analysts, managers, and consultants." The bot did an "amazing job at basic operations management and process analysis questions including those that are based on case studies," Terwiesch wrote in the paper, which was published on Jan. 17. He also said the bot's explanations were "excellent." The bot is also "remarkably good at modifying its answers in response to human hints," he concluded.
While Chat GPT3's results were impressive, Terwiesch noted that Chat GPT3 "at times makes surprising mistakes in relatively simple calculations at the level of 6th grade Math." The present version of Chat GPT is "not capable of handling more advanced process analysis questions, even when they are based on fairly standard templates," Terwiesch added. "This includes process flows with multiple products and problems with stochastic effects such as demand variability." Still, Terwiesch said ChatGPT3's performance on the test has "important implications for business school education, including the need for exam policies, curriculum design focusing on collaboration between human and AI, opportunities to simulate real world decision making processes, the need to teach creative problem solving, improved teaching productivity, and more." The latest findings come as educators become increasingly concerned that AI chatbots like ChatGPT could inspire cheating. Earlier this month, New York City's education department banned access to ChatGPT. While the education department cited "safety and accuracy" as reasons for the decision, the Washington Post notes how some teachers are "in a near-panic" about the technology enabling students to cheat on assignments.
Yesterday, for example, The Stanford Daily reported that a large number of Stanford students have already used ChatGPT on their final exams. It's prompting anti-plagiarism software Turnitin to build a tool to detect text generated by AI.
It's an MBA exam... (Score:5, Funny)
It's just an MBA... ever spend some time talking with those graduates?
Didn't Trump pass one of those tests?
Re:It's an MBA exam... (Score:5, Insightful)
Didn't Trump pass one of those tests?
Trump is an impulsive narcissist with a poor attention span and no filter between his brain and his mouth.
But he is not stupid.
Re:It's an MBA exam... (Score:4, Funny)
But he is not stupid.
[Citation needed]
Re: (Score:2, Insightful)
[Citation needed]
He's a billionaire, and you're not. He may have made his billions by being an unethical slimeball, but that doesn't make him stupid.
He was a successful TV star, and you weren't.
He defeated an entire field of supposedly intelligent people to become President of the United States. He was ingenious at demagoguing the electorate's prejudices, phobias, and insecurities. No other candidate had a clue how to counter that.
Re:It's an MBA exam... (Score:5, Insightful)
Re: (Score:2)
You can be stupid and successful, as long as your stupidity is consonant with the viewers, or the electorate, or whatever is your target audience.
It also helps to have money so that people tend to say "yes" to you.
(or at least the appearance of it, the "tax returns" never actually appeared...)
Re:It's an MBA exam... (Score:5, Insightful)
He inherited his wealth, I could be a billionaire too if I did not have to work for it. That, and he underperformed index funds (though figures are disputed since it's difficult to quantify his actual wealth).
Your argument is a bit like "Charles III is a king and you're not". Well duh.
Re: (Score:2)
Only on Slashdot could a ChatGPT discussion evolve into a Trump debate!
But no, I don't think the "Trump was handed everything" argument holds as much water as many people would like it to. Take a look at the success rates of lottery winners in the USA, for an example. It's incredible how many of them lose everything in a really short period of time and wind up practically homeless.
The mere fact Trump invested in real estate shows he was making smarter financial decisions than many people who are just given
Re:It's an MBA exam... (Score:5, Informative)
We will never know if Trump is a better businessman than I, because I won't be able to inherit considerable wealth and business connections, and then take a "small" $1,000,000 loan from my parents to get started.
I feel like if I had all those things I could do better than Trump. I reckon I could take a mongoose in a fight too.
Re: (Score:2)
I feel like if I had all those things I could do better than Trump. I reckon I could take a mongoose in a fight too.
You could make such claims only if you can show that you have done something with your life absent such advantage. Are you already independently wealthy? It is more likely that if you had a million dollar loan you would still owe 900K on it.
Re:It's an MBA exam... (Score:4, Interesting)
That's Trump's version of the story. According to the Times, it's more like a $413,000,000 tax evading gift. https://www.washingtonpost.com... [washingtonpost.com]
Re: (Score:2)
Re: (Score:3, Insightful)
Trump steaks
Trump Casino
Trump Beverages
Trump The game
Trump University
The list goes on...
Trump makes George Santos look like an honest man
To Help Get Him Into Wharton, Trump Allegedly Paid Someone To Take The SAT
As for TV star... only for those with incredibly low standards, he was nasty back then too
https://slate.com/news-and-pol... [slate.com]
If trump did not inherit his wealth he would be running scams on tourists somewhere, and being in and out of prison.
Re: (Score:2, Insightful)
He's a billionaire, and you're not.
False. We have no evidence he is a billionaire as he has refused to provide any such evidence. He keeps saying he is, but he lies about a lot of things.
Re:It's an MBA exam... (Score:5, Insightful)
He says he's a billionaire. His relationship with the truth is in negative territory.
Re: (Score:2)
So is Elon Musk. Who seems to care more about tweeting than actually running a company competently. I think he's less brains and more "money mule" given recent antics.
Re: (Score:3)
[Citation needed]
He's a billionaire, and you're not.
So his dad was smart. And, in fact, Trump is probably insolvent rather than being a billionaire.
He was a successful TV star, and you weren't.
Interesting view of TV stars.
He defeated an entire field of supposedly intelligent people to become President of the United States. He was ingenious at demagoguing the electorate's prejudices, phobias, and insecurities. No other candidate had a clue how to counter that.
Well, he lost the vote so they seem to have had some clue. The garbage electoral collage system in the US handed him the office, but that was dumb luck; he didn't design the system.
The thing about having 350m+ people in your country is that the one that happens to be luckier than everyone else will be very bloody lucky indeed. If Trump had been born to different parents he would have
Re: (Score:2)
A reality TV star needs brains?? The production makes the show; "the talent" is the spetical in front of the camera they help create.
Investigators estimate Trump inherited and swindled from his father and uncle approximately a billion. He was a massive failure the whole time; it's hard to screw up that much inheritance into the poor house. One could buy slow growth bonds and significantly out perform Trump. He has probably only actually grown when the kids starting taking over and they got all those Russi
Re: (Score:2)
You don't need to not be stupid to be a billionaire either. The two are unrelated. Also it's likely he's not a billionaire, he doesn't have much proof beyond his own assertions. He's been bankrupt many times, each time a way to shed debt that he earned onto others. That's smart, but that's also likely advice from others.
There are stupid people who are TV stars. Big deal. He had a lame TV show. He was only there because he spent decades building up his public image by appearing all the time in media,
Re: (Score:2)
Re: (Score:2)
Because the elected politicians every where else are brilliant, honest, and have their people's best interests at heart.
Lol
Re: (Score:2)
Re: (Score:2)
Trump is an impulsive narcissist with a poor attention span and no filter between his brain and his mouth.
But he is not stupid.
Yes he is.
He also only did undergrad at Wharton.
Re: (Score:3)
But he is not stupid.
No, he is clearly above average, but that is a low bar for a president. Most past presidents and contenders have been exceptionally smart people.
Obama, Romney, Bush the first, Clinton x2, Nixon, Kerry, McCain - lots of really smart people.
GWB less so, but he was a legacy hire. And a decent guy who knew his limits. Ronald Reagan was probably in decline, he seemed smarter in his younger days. Unlike Bush or Trump, he was an awesome public speaker.
Actually, if you watch some old interviews of Trump from the 8
Re: (Score:2)
It's just an MBA... ever spend some time talking with those graduates?
Didn't Trump pass one of those tests?
We don't know. The con artist has threatened to sue anyone who releases any of his supposed college information.
Because he was such a great student and all. The best. Top of his class. No one does class better than him.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Trump also went to Wharton. He seemed to have others do much of the work for him. Possibly ChatGPT also was allowed to cheat in many ways, and the whole test likely set up as a show piece rather than being a real test in a real examination room with real restrictions on what source material you could use. Was ChatGPT marked down for plagiarism at all here?
Proving once again... (Score:4, Informative)
That MBA degrees are the most useless shit on the planet.
Re: (Score:2)
^ This.
Re:Proving once again... (Score:5, Interesting)
They get you more money so I don't see how that is the case.
Also before anyone gets too smug, ChatGPT also passed an astrophysics exam with about an average score: https://youtu.be/K0cmmKPklp4?t... [youtu.be]
Re: (Score:2)
Re: (Score:3)
It's a computer, it's obvious it can't do math and never will.
Re: (Score:2)
That's a pretty broad statement. "...it cannot do math as of now."
I think I can play guitar. I can play about 80% of Stevie's cover of "Little Wing" well. I can play 70% of Eric Johnson's "Cliffs of Dover".
Does this mean I can't play guitar?
ChatGPT could no doubt replace many MBAs (Score:3)
They say the robots are coming for our jobs, and even for our programming jobs. The MBA jobs will undoubtedly go first. Once that happens, then I might start to become concerned about software development!
Re:ChatGPT could no doubt replace many MBAs (Score:5, Insightful)
They say the robots are coming for our jobs, and even for our programming jobs. The MBA jobs will undoubtedly go first. Once that happens, then I might start to become concerned about software development!
You've got it wrong (unfortunately). The professional-managerial class will protect its turf the hardest. Automation of factories, grocery store checkouts, gas stations, farms, etc? Fuck those jobs. And yet, as an examples, lawyers and the ABA screech at the hint of anything that might give power or agency to non-lawyers in the legal system. Professional licensure is another big stumbling block across many fields. MBAs will undoubtedly find a reason, and laws to back it up, as to why they need to have a human present.
Hopefully the herd at least gets thinned a good bit.
Re: (Score:3)
You are absolutely right, they'll fight tooth and nail. They won't be automated in companies where they are already entrenched. Instead, it will be new upstart businesses, that want to focus on just making a great product, or delivering a great service, that will have no use for the kind of business managers that can be automated. Those new companies will save so much money through automation, that they'll begin to displace the entrenched bureaucracy of the big corporations. And so the business cycle goes.
Re:ChatGPT could no doubt replace many MBAs (Score:5, Interesting)
AI can read, analyse and report on huge amounts of data quickly.
The old "bury them with paperwork" to hide what was important is rapidly going away, the AI works 24/7.
Because of this junior lawyers who used to do all that donkey work are no longer needed.
Re: (Score:2)
I do so hope you are correct! Yes, automation could replace a lot of what lawyers do. But lawyers know how to make themselves important legally. These days you can already buy software to help you with routine legal filings, such as last wills. But lawyers in many cases have found ways to make themselves a required step in the process, even if you did all the work yourself already (with the help of software).
Re: (Score:2)
They say the robots are coming for our jobs, and even for our programming jobs. The MBA jobs will undoubtedly go first. Once that happens, then I might start to become concerned about software development!
Why? You think ChatGPT could possibly do worse than MBAs at managing software development?
Re: (Score:2)
The opposite.
Yes, but does it know how to... (Score:2)
...fire one million workers instead of five hundred thousand like Jean-Baptiste Emanuel Zorg?
Re: (Score:2)
It will know how to automatically lay off everyone at Brawndo (The THIRST MUTILATOR!) when the stock price dips.
This post brought to you by Brawndo (TM) "It's what plants crave!"
BS generator (Score:5, Insightful)
So ChatGPT is good enough to generate vision statements and other BS coming out of the exec suite, often after a whole week of offsite brainstorming in a faraway ski resort?
Nice, the next job class to be automated away could be the upper management.
Re: (Score:3)
I for one welcome our new AI CEOs.
At least they'll be using data.
Re: (Score:2)
Re:BS generator (Score:4, Interesting)
Certainly creatively focused jobs can be downsized a bit.
Need a slogan for a new food or car? Ask ChatGPT for 10 slogans and then brainstorm with those options as a starting point.
We'll still need creative people. But less of them and they can use their time more effectively.
Re: (Score:2)
Upper, but not top-level management.
Yes, but we've all been aware for decades that most management could be replaced by fairly simple scripts.
What ChatGPT can't do is provide the "human interface" that people need. We're social animals. Much of management is simply there to provide a human face to the company processes.
This proves bullshiting is key to passing exams (Score:5, Interesting)
I tried ChatGPT with real, actual, precise technical questions: the thing was confidently giving me very verbose, very well argumented, very well presented, totally incorrect answers. When I corrected it, it apologized, acknowledge the correct bit of information I just fed it and proceeded to give me more wrong answers, making shit up as it went.
But if you didn't know the answer was wrong, the thing was really quite convincing.
ChatGPT is a cross between a webcrawler and BonziBuddy. It's not intelligent: it feeds you any old incorrect data it finds on the internet that matches what you ask it. It doesn't understand what it's saying. But what it's very good at is packaging its response into something convincing. I wouldn't be surprised if ChatGPT won an election some day...
I'm not surprised at all it was able to bullshit a MBA exam jury. But it would never mass muster with a proper engineering exam jury asking basic engineering questions.
Re:This proves bullshiting is key to passing exams (Score:5, Insightful)
Your comment is so vague and nonspecific that it's, well, meaningless. I have been utterly fascinated watching people attack chatGPT, including people who should know better. I'm not going to say anything like it's approaching general purpose intelligence, but it's shockingly beyond anything we've seen before. It's like when Google's Go AI came out of absolutely nowhere and crushed the best human players. Exponential change is hard to deal with, and we're in the middle of some exponential change right here.
Here's something I just tried. There was additional explanatory text I cut. Tell me with a straight face that chatGPT is no better than a "cross between a webcrawler and BonziBuddy"
If you think math and engineering (and physics, etc) isn't on the improvement plan, and attainable, you're completely deluding yourself.
Write a C program that picks a random number between 0-100, then allows the user 3 guesses to find the number. If the user gets it right, the program prints out "Yay!". If the user does not find the number it says "ERROR"
(Note: you could also have said ASM, Pascal, Ada, PL/I, Cobol, etc.)
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
int main() {
srand(time(0));
int secret = rand() % 101;
int guess;
int guesses = 0;
while (guesses < 3) {
printf("Guess the number between 0 and 100: ");
scanf("%d", &guess);
if (guess == secret) {
printf("Yay!\n");
return 0;
}
else {
printf("Nope, try again\n");
}
guesses++;
}
printf("ERROR\n");
return 0;
}
Edit the program so that after each guess, the program tells the user if their guess was too high or too low
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
int main() {
srand(time(0));
int secret = rand() % 101;
int guess;
int guesses = 0;
while (guesses < 3) {
printf("Guess the number between 0 and 100: ");
scanf("%d", &guess);
if (guess == secret) {
printf("Yay!\n");
return 0;
}
else if (guess < secret) {
printf("Too low, try again\n");
}
else {
printf("Too high, try again\n");
Re: (Score:2)
Try something more complicated. Something which requires ChatGPT to construct an equation based on verbal description and solve it to return the right answer.
Re: (Score:2)
It would do that easily, provided that fragments of solutions are somewhere in its model.
It can find common programming errors in code you provide it as well, which is an interesting use case.
But we'll find its weak spots by experimenting, the way we were first astonished by Stable Diffusion, then figured out it doesn't know how many fingers people have and other silly stuff.
What worries me is that lots of the "knowledge" it has ingested is likely to be outdated or outright wrong. We already have trouble en
Re: (Score:3)
Interesting use case. I edited the "guess" program to change the comparison from "guess == secret" to "guess = secret" and asked if there was a problem with the program. It correctly found the bug. I came up with another contrived off-by-one error in array indexing, and it located that bug too.
Cool stuff.
Re:This proves bullshiting is key to passing exams (Score:4, Insightful)
This is a very common and simple problem.
That certainly depends on the definitions of "common" and "simple!"
Maybe what we're really finding out is that a very large percentage of the tasks that most humans undertake on a daily basis are "common" and "simple," even when they seem more complicated.
Re: (Score:2)
Most tasks we do are common and simple. If somebody sees them complicated then it is a wrong assessment of that person.
It will be great when AIs will do all the common and simple things for us. That will leave only the interesting tasks to be done by humans.
Re: (Score:3)
This is a very common and simple problem.
What percentage of coding problems are not simple problems? I think most of the coding tasks are routine. Plus ChatGPT could likely be trained not allow overflows and such, while humanity on the whole cannot be.
Re: (Score:2)
I have been utterly fascinated watching people defend chatGPT, including people who should know better.
fixed.
Re: (Score:2)
A pithy, yet useless, reply. ChatGPT is what it is. Fail to understand what it is at your own professional peril.
Re: (Score:2)
oh boy will there be peril.
Lang. model and context: if they are large enough (Score:2)
Here's something I just tried. There was additional explanatory text I cut.
...(and is actually critical to the whole process, as it is necessary to load the current context.)
Tell me with a straight face that chatGPT is no better than a "cross between a webcrawler and BonziBuddy"
{...}
Write a C program that picks a random number between 0-100, then allows the user 3 guesses to find the number. If the user gets it right, the program prints out "Yay!". If the user does not find the number it says "ERROR"
(Note: you could also have said ASM, Pascal, Ada, PL/I, Cobol, etc.)
Now just realize that the language model of chatGPT and the context in the dialog don't necessarily need to be in English.
It's a realistic output, according to the language model, that fits the cues present in the dialog context.
That's it.
We can be amazed at the emergent properties of the system (there's no specialised english-to-C transpiler inside chatGPT, that's just a natural output of the model trained on
Re: (Score:2)
The exam was online due to COVID so was likely dumbed down, but there are discussions being held as to how to mitigate it.
At this stage they are working on the basis that the spelling and grammar are well above the usual student level, so its relatively easy to spot.
Re: (Score:2)
Agree 1000%. The danger of ChatGPT is that the answers it gives sounds right and (to some) have the appearance of authority because it was generated by a computer.
Re: (Score:2)
I wouldn't be surprised if ChatGPT won an election some day...
Just tried using ChatGPT now and their "we're too busy now" page reads:
Write an inspirational speech about the status of ChatGPT.
My fellow citizens, the time has come. The future is here, and it is called ChatGPT. This revolutionary AI chatbot website is here to change the world, and we must embrace it with all our strength and determination.
I know that many of you are eager to try ChatGPT for yourselves. And I assure you, we are doing everything in our power to make that happen. Our team is working tirelessly to accommodate the incredible demand for this groundbreaking technology.
But I want to remind you that greatness is never achieved without sacrifice. And the road to the future of AI will not be without its challenges. So please, be patient. Trust that the right time for you to try ChatGPT will come.
And when it does, let us seize this opportunity with all our might. Let us embrace the power of AI and use it to build a brighter, more prosperous future for ourselves and for generations to come. Thank you.
Sure sounds presidential enough! Why wouldn't a chatbot be able to write inspiring speeches using some A/B-testing to find out what resonates best with the public.
Re: (Score:2)
You're stretching a young tool until it breaks. Good for you. Not unexpected. I can get it to tie itself in knots too. Just understand you're not proving anything, really. Just that it's not perfect. And it never will be. But it'll get better and better.
Language model != inference engine. (Score:5, Interesting)
You'd be more convincing if you provided the interaction. It's not hard to cut & paste.
I am not close with the parent poster, so I don't have seen these interactions either, but seems entirely plausible.
chatGPT is an (insanely) large language model. Not an inference engine. It doesn't (formely, as part of its primary design,) do logic.
It just strings together things words that "sound nice".
It's like the autocomplete function of your smartphone, but with many orders of magnitude larger model.
So much larger, that the model can manage to encode some semblance of truthness, understanding, etc. but it's not part of the primary design.
It's exactly like the autocomplete of your phone: that one "seems to have an understanding of grammar rules", because it manages (most of the time) to string together correct words. But it actually never had real grammar rules encoded into it. It's just that the autocomplete model is large and advanced enough that the "what words would best fit in there" probabilities take enough of the previous context into account that it DOES end-up making grammatical sense. There's a implicit "hidden grammar" understanding "emergent" in the model. But it's still a statistical model, not a grammar rule system.
chatGPT is the same but with the model's size blown a zillion times up. It is mainly able to generate text that sounds nice and fits the style. But it doesn't have a true understanding of the text (it doesn't have a true internal representation of the logic). The model is large and complex enough, that it can encode in a "hidden fashion" some things like the current subject, and informations/instructions/criterias it has picked up from the dialogue with the user. But it's an emergent property of the language model, not an actual inference engine powering up some logical analysis. It has no direct concept of truth.
ChatGPT is a cross between a webcrawler and BonziBuddy. It's not intelligent: it feeds you any old incorrect data it finds on the internet that matches what you ask it.
This is completely wrong. It's good at finding primary sources.
In the experience of my colleagues, it's also very good at completely hallucinating primary sources that sound nice. ...completely made up titles.
One colleagues gave it a request to list key publications in a field.
chatGPT spat out a very convincing list of references, citing names of real world scientists in the field (i.e.: the model is complex enough to know that these scientists' names are linked to the subject requested), perfectly formated as a reference list (again good at following the models it was trained on) that listed...
(i.e.: the model knew that these words are related to the field, they are arenged in a coherent manner that looks like the typical title. Which is still impressive, by the way).
It doesn't understand what it's saying.
This is also wrong. It has rudimentary understanding and will even explain to you how that works.
Nope. It doesn't
It's not an inference engine designed to do logical analysis. It's a very large language model. It's mainly good at stringing words together in a way that sounds realistic.
That model is large enough that it can manage to encode some semblance of understanding. (i.e.: unlike the punny autocomplete on your phone it won't steer its rambling in some completely random direction. It will stay on subject and in style), but understanding and reasonning ; being able to infere stuff, etc. were never part of the main design of chatGPT. They are just emergent property that kind of accidentally happen once a language model is ginormous enough (again, it's like autocomplete and grammar - but on steroids).
It's very good at finding answers if those answers are in a single source.
As long as:
- either: this knowledge is enc
nerfed (Score:4, Interesting)
ChatGPT is nerfed by being unable to do math. It can clearly recognize math, and usually knows what formulas to apply, but does calculations right almost never. If it called out to WolframAlpha when it recognized it was doing math it'd get it right closer to 95% of the time. It can't even multiply correctly. A computer is clearly nerfed if it can recognize the need to multiply but can't multiply correctly.
What's more, if it could do math correctly, it could up the importance of doing math-based sanity checks, because doing math correctly would make those sanity checks useful.
ChatGPT is the perfect MBA (Score:2)
In other words, ChatGPT was an even more convincing substitute for a typical MBA than Terwiesch expected.
Hard to judge without knowledge of training set (Score:4, Insightful)
Re: (Score:3, Informative)
a better analogy is what a person would produce when taking an exam while having access to the internet where they can search for an answer posted online and then take the time to reword it.
Did you read the paper? Those were not essay-type of questions, but more of optimization problems - "identify the bottleneck", etc... Even though it failed to do some math in the more complex problems (math was not complex, but the question setup was), it was able to "solve the riddle". I repeat, it is not just copying answers from the internet. This is best proven by giving it coding tasks. There is some "understanding" of the problem logic - and it is able to generate code from scratch for new problems.
Re: (Score:2)
I repeat, it is not just copying answers from the internet.
You really cannot be sure of that. There is a lot of information on the Internet. Well, maybe it has some Wolfram-Alpha-like capabilities as well (or actually licensed that engine as you can do), but that does not mean it has any clue what it is doing.
Re: (Score:2)
These MBA questions will be in there. This stunt here was probably planned a long time ago. A lot of the marketing for ChatGPT is carefully orchestrated misdirection.
Re: (Score:2)
ChatGPT has the interesting feature of essentially repeating your prompt back to you, then elaborating. That is exactly how exam questions are often phrased. When the question is "Explain how XY does AB." then you can't go wrong by doing what ChatGPT does: "XY does AB by ..." and then add whatever your memory digs up about XY and AB. Putting a bunch of connected facts through a natural language generator to put them into sentences isn't that hard.
Drawing (correct) conclusions from your known facts is what i
The subject direction (Score:4, Insightful)
there would be less worry about cheating if the new technology we will be using in the very near future and need to understand well how to use is made to be part of the coursework. my math teacher back when calculators first came out (mine was an HP-35) included them in his teaching under the idea that we would be using them and needed to learn how to work with this subject with calculators. he even arranged for loaners for those who could not afford one. he encouraged everyone to bring one to each class. he "required" on on each exam with questions specifically about using a calculator to solve the problems.
we will be using AI from now on (until it replaces us). we need to, for now, learn how to incorporate and integrate AI into the near future work we do. the course needs to include this. the exams need to include this. use of AI needs to be mandatory in all future subjects.
ChatGPT would outperform my manager too (Score:2)
Unfortunately he just got sacked by G..gle so they must have figured that out as well.
Let me get this straight. (Score:2)
This is like taking an open book exam.
I'm sure I could pass any exam you give to me, as long as you give me Google whatever i want.
ChatGPT is trained on the world's knowledge. It literally has access to any question it wants.
hmm... (Score:5, Interesting)
I can't tell if this is a story about the advancement of AI technology or the steady decline of MBA's
Parrot (Score:2)
As soon as the system (Score:2)
Re: (Score:2)
As that is probably not going to happen anytime soon, the middle-managers will sadly say. Remember this thing was trained on a set of carefully curated data and I do not even want to know how often they had to reset it to an earlier stage. Even so it often is really badly off. Training it on data that is uncurated will likely blow it up within a very short time.
While ChatGPT is an impressive stunt, it really is nothing more than a very early, very experimental prototype. It is human to go the animistic pat
Not great at 6th grade math (Score:2)
Quote: Terwiesch noted that Chat GPT3 "at times makes surprising mistakes in relatively simple calculations at the level of 6th grade Math."
So it's superior to most MBA's then?
Where's the information retrain loop in this? (Score:2)
Everyone keeps bawking about how ChatGPT will, or will not, replace X job because #reason.
Either way, this thing needs to keep learning to remain relevant. So where's the docs on how this thing is trained, and how it acquires and retains relevant knowledge to perform within a particular role or skill? I keep seeing disclaimers that it's trained on 2020 information, etc.
This toy is trained on past data, so it might be able to theoreticize potential futures based on that past data, but it needs to be able to
Re: (Score:2)
Indeed. This thing can serve as an insight-less and not-so-rarely badly wrong oracle, but it cannot do daily business. It will also get outdated rapidly.
Finally you can do it (Score:3)
You can go to your manager and say "STFU or I replace you with a small script".
So Warton's MBA is crap? (Score:3)
Because this is pretty much what that means. ChatGPT can only pass an exam if that exam requires no insight or understanding.
Not that I am surprised an MBA program has bad exams...
Re: (Score:3)
I suspect it would only take a perl script to pass some of my engineering and mechanics exams. Especially the early ones like statics, I remember the final was copying values from the diagram into a matrix for the simultaneous solver on my calculator.
Re: (Score:2)
For introductory and early classes that is not good, but not nearly as bad. This was apparently a final exam on their MBA programm.
Re: (Score:2)
It was a subtle jab that I equivocate the difficulty of first year engineering with the completion of an MBA. Even so, I likely had to use more integrals than an MBA. Those even showed up in the chemistry classes I took "for fun".
the problem with exams (Score:3)
The problem with exams is that they are written assuming a human college student is taking them. Those assumptions let us skip over a lot of tedious probing and focus on the incremental knowledge that the course provides. All of these machine learning systems are quite specialized and unbalanced in their knowledge set when compared to a human adult with two or more decades of experience. Sadly I suspect some professors will feel the need to adjust exams in order to filter out ML systems, probably with some trap questions that will just end up tripping up humans with false accusations of cheating.
MBA (Score:3, Funny)
Isn't that just parroting business buzzwords that have no real meaning?
Hardly surprising (Score:3)
Mechanical Turk? (Score:2)
Don't count your chicks... (Score:2)
Don't count your chicks before you've bought the eggs.
Thus proving... (Score:3)
... that the MBA is bs.
Basically a hallucination (Score:2)
It will pass not only MBA exams (Score:5, Interesting)
Just yesterday I tested ChatGPT on questions for molecular genetics exam I plan to give students this week. ChatGPT has a decent chance at a B grade. Answers are very well written. If only reproduction of well established knowledge is required the answers are excellent most of the time (except when it tries to "volunteer" information that the question is not asking for specificaly). In followup questions that aim to clarify details or logic it fails very quickly and in a way that is very similar to a liar that has been caught or a student that is trying to mask that they don't know the answer: it starts providing verbose, repetitive and circuitous answers that pile on flaws. The reasons are two fold: algorithm based on associations that has no means of distinguishing fact from fiction, and reliance on information that has been repeated multiple times on the internet.
Whether it will undermine education depends on how people in education adapt to it. I have been transitioning to take home open-book exams for my students because I feel it is impossible to know everything and is more important to be able to find, critically review and synthesize information. This was until know slow process because it requires a lot more work both in the way we teach and the way we test. It is much easier to run the multiple choice exam conveyor. I guess I will be forced to move faster now, and perhaps modify the exam to add verbal Q&A sessions with the students (more work ....). Other faculty are reacting by moving to tightly proctored exams or looking for tools that would detect AI generated answers.
Re: Clickbait (Score:5, Insightful)
I think the point is a bit inverted. It's not attempting to be Turing test. instead it's a mockery of standardized tests. A test with a highly specific question, seeking a specific answer, can be "cheated" by this AI with a good enough answer.
Consider China's Gaokao. English is standard education but it all narrows on English ability for these tests. This means at a spoken level, they cannot hold a simple conversation even if they had a high score... So highly standardized tests, seek highly standardized answers... mutability or ingenuity is not necessary or encouraged...
Re: Clickbait (Score:5, Interesting)
My grad student job was to be an assistant academic secretary at a continental European university that had many English programs to attract foreign students. In this capacity I handled the admission files. Every single one of the Chinese applicants had the exact same score on their TOEFLs, just meeting our official requirement. Yet - indeed - almost none could hold a conversation with me, and most failed miserably as the lectures were in heavily accented English spoken by non-native speakers.
I know - cool story bro. Completely anecdotal. YMMV
Re: (Score:2)
Yep. Standardized tests test against the standard, not against actual real-world usage scenarios. They are easy to create, easy to grade, easy to make money with. At the same time, they are pretty worthless, just the same as applying a check-list to a complex problem usually is.
Re: (Score:2)
I think the point is a bit inverted. It's not attempting to be Turing test. instead it's a mockery of standardized tests. A test with a highly specific question, seeking a specific answer, can be "cheated" by this AI with a good enough answer.
Yep, pretty much. That may not have been the marketing-idea behind this stunt, but it is what it shows to any smart person: Standardized tests are easy to make, easy to grade and show nothing of relevance about the person being tested. In other words, standardized tests are pretty much bullshit.
Re: (Score:2)
The details of this "test" are unknown too. What ChatGPT locked in a room with no access to the internet, did it have extensive databases or was it limited to a 3" by 5" index card of notes?
Right now, ChatGPT is just basic AI technology about stringing together textual bits. It's good at parsing natural language and searching for good textual patterns to respond with. Without that big database of "answers to past exams" it would have gotten nowhere.