Reading through the questions on the test I'm thinking about getting an MBA in my spare time this exam seems incredibly easy unless I'm missing something.
It doesn't seem like a terribly hard test. But that's as someone who has (quiet now) an MBA who did a 1:1 project for an operations management professor and graduated high in their class albeit a long time ago. So don't read a lot into my assessment.
The Kanban answer I found especially interesting. The equivocation on the one answer is exactly the one I would have equivocated on. (Well, yes, that's the idea but it has to be done correctly.)
I'd also observe that operations management/engineering is a lot closer to other types of engineering than things like organizational behavior are so writing this off as just an MBA thing is pretty complacent.
The core courses between the grad and undergrad programs are essentially identical. In terms of whether he passed an exam like this, well, cs get degrees and its pretty hard to actually fail out of business school
No, it is not. Calculator still needs operator knowledge, in this case if copy-pasting questions is enough then it doesn't and the test itself would have to change completely
After working with LLMs for a while, I'm pretty sure we're close to a prompt like this, "In style of Kurt Vonnegut, write an essay about the career of US Grant, include his military accomplishments and his burial." The output would likely be an essay that would pass the test and be impossible for a professor to detect.
Perhaps education will have to value cognition and ability to apply knowledge over being able to organize and communicate facts and accepted opinions. One thing is sure, a lot is going to change over the next two decades.
Actually they're working on embedding a verifiable, steganographic signature into GPT responses that would let them detect it's outputs later. It would basically bias common word choices in a subtle way that requires a private key to verify.
That sounds like it would be simple to work around by automatically replacing words with sensible synonyms, changing punctuation, or even translating between languages. I wonder if there's a way to encode the "signature" such that it would survive these lossy transformations.
Maybe. I believe it's probably deeper than just the choice of synonyms, the choice of "next token" as a whole. For more information see Aaronson's description of his work in the section titled "My Projects at OpenAI":
LLMs can't so easily defeat cryptography which is used in the encoding. If you have one powerful enough to mess with GPT's output to make it pass a watermark test, you probably have one powerful enough to generate the output you're looking for to begin with, so why bother with GPT at all?
They are just calling ChatGPT "Chat GPT3" and GPT3 as "GPT3". With an acknowledgement much later in the article that it is not ChatGPT being used in this paper.
> Wow! Not only is the answer correct, but it is also superbly explained. The idea of the bottleneck as the rate
limiting step was clearly understood and all calculations were carried out correctly. I don’t see any reasons to take
points off from this answer: A+!
I tend to agree. This is exposing a (perhaps universal) bias towards confidence, correct grammar, and a general encyclopedic tone. The trappings of a correct answer, rather than explanatory power and correctness. It’s a con artist.
How many of these “machine does something humans do” articles are we going to get? It says a lot about how we’ve discovered core concepts and streamlined learning that a machine can pass a test. Text is a wonderful medium for communication, and computers are finally somewhat up to it and it’s marvelous, but this is like producing solved sudokus — it doesn’t really create anything.
ChatGPT isn’t an MBA any more than someone equipped with the answer key to the test is an MBA.
I tried feeding it an LSAT logic game which has multiple explanations on the internet. It got 2/5 correct, and for one answer it said “E, none of the above”
E does not say none of the above on this logic game. Though many other different standardized tests do use E as none of the above. So it was pattern matching to that and not the text in front of it. And it wasn’t using explanations
Oh how quickly the goalposts shift... If the test was done correctly and there was no data leakage, this absolutely does not compare to someone equipped with the answer key.
I also think there is a novel new effect happening which is that if one question anything about AI. For example: I tired to use Dall-E for something useful and it sucked. You get labelled...Anti-AI.
Being critical about certain things isn't allowed at the moment.
Kind of like the 2023 replacement for Anti-vaxxer if you don't want to take medicine every 3 months.
My honest opinion, is that while a lot of this technology is cool and novel, it's not perfect, it has uses and then it doesn't. What it does have is a lot of good marketing and PR behind it and so you're up against a lot of that too.
One example I like is the Dall-e homepage shoes the curated "best" images generated, which might be showing only 1 in 1000 or more of the best images. I was excited when I first tried it based on the front page. But now it's drawing people with feet for hands for me and it's super super fun and entertaining to play with, I don't feel it works quite as advertised most of the time.
Disclaimer: Not anti-ai and I took my Covid19 vaccines (in the beginning, no boosters)
Still questions that I would assume are semantically similar to the questions you can find in exam prep material all over the internet. My point is that exams are a crutch we use to determine how well a person studied a subject. A crutch we use because we seem to lack better measuring devices. It's very much possible to ace an exam while at the same time being horrible at actually applying/working on a subject. I'd argue therefore that measuring how well LLMs perform on exams designed for humans is simply a more complicated Turing test, with all its shortcomings.
>Argument from ignorance, also known as appeal to ignorance (in which ignorance represents "a lack of contrary evidence"), is a fallacy in informal logic. It asserts that a proposition is true because it has not yet been proven false or a proposition is false because it has not yet been proven true.[0]
Why should we assume that the training set includes the answer key in the absence of evidence?
We know that the training set is large and non public, and that answer keys are generally available and ML likes to output exact answers when it's been trained on the questions.
It's significantly more likely that a model outputting the right answer was trained in that answer than not
Your comment isn't original either, it's almost always the same response if anyone questions many technologies.
If you think about it critically, maybe we will find there isn't actually much use for ChatGPT outside of it being a pretty cool party trick. Even if it gets better.
I think it pains people to think about it this way but if we have bullshit jobs, and now have bullshit bots to do bullshit jobs, what's really changed? Likely we'll just end up with more bullshit.
I'm sorry but I think it wouldn't hurt for you to also be objective yourself when evaluating the system your defending.
When you think about it this way, if ChatGPT has the answer to all answerable questions loaded into it's memory, is any of this really that impressive? Initially yes, in the long term maybe so.
I would argue it speeds up finding the answer to many questions which I can find online, but I pretty much had access to any answer I needed already.
My feeling is we think these machines are impressive, because we think we're impressive, and ChatGPT is doing something that looks like what we do in a new and novel way.
I also get the vibe that we almost "have" to be impressed with it because as a species, we've invest a lot into these endeavors and I guess the thought of it being not useful it just as scary as it being useful and replacing everyone's work and crashing the economy of whatever.
The psychology of it all is fascinating.
I guess we all thought steam trains were amazing and unfathomable at one point in time, is this really much different time will tell :)
It does compare to inputting a math question into a calculator though.
Do calculators mean that we shouldn't teach multiplication anymore? That we shouldn't have tests for multiplication? The calculator will always get the right answer.
The bigger question imo is what humans are doing with these new tools. We no longer need the human computers of old now that we have calculators, and almost certainly we can innovate over automated MBAs now
Probably a lot more? Part of it is we don’t have a definition of intelligence, so the best we can do is measure machines against our tests since passing tests is a thing intelligent people can do. This kind of performance (even if it’s less glamorous upon closer inspection) puts symbolic AI folks from the 80s to shame.
That’s the whole point of Attention is All You Need, the paper that opened the door to web-scale language models. No, you don’t need a brittle and cumbersome knowledge base, at least not for this. MBA professors are perfectly happy accepting an autocomplete on steroids.
Marshall ChatGPT leads Allied Armies to victory! [all ranks between Army General and Captain, non inclusive, have been declared redundant and eliminated with ChatGPT absorbing those ranks]
I'm actually glad something is forcing the education system to think critically about change.
When I enrolled in a MBA a few years ago, I was a glorified test taker being monitored by some random person overseas to prevent cheating. Even then, many people in my cohort cheated on their tests and it is pretty well known that MBAs cheat more than other degrees.
I dropped out a year into it after learning nothing and seeing all those test answers on websites like Chegg while noticing that majority of the schools that offer an online MBA program all used the same curriculum/tests/answers and that nobody was actually teaching. I had a dream that I might learn from a business professor of the notoriety of Clayton Christensen or similar. I got a huge reality check instead.
So if ChatGPT can interpret the answer from Chegg, then it is fair game to have earned that MBA.
That sounds awful. Also absolutely different from my in-person experience a long time ago. Not sure to what degree a virtual experience can be made worthwhile.
Certainly a compelling piece of evidence, here, to not allow the open internet or personal devices during tests designed to check how much someone has learned.
Also curious to see if anyone has any luck asking ChatGPT for how to fix specific problems at their current companies, vs asking an MBA.
Improving curricula, expecting more of students, and forcing the humans to compete with AI that ruins the curve would only help if the goal of the MBA program were education.
I'd wonder if ChatGPT logs every answer it affords all users so that any testing institution could cross check with the Anti-Cheat ChatGPT paid service for educational and certification institutions.
That’s unlikely to be how the system works, chatGPT by design is a probabilistic model meaning it’s very unlikely it’ll spit out the same output twice. The detectors probably use a vanilla language model that just recognizes the patterns in chatGPT discourse (which is very distinct)
No, I mean does the system keep a record of every reply it provides such that someone could check to find if there was a match. Obviously a naive avoidance technique would be to re-phrase an answer.
That said it can produce nearly identical answers to prompts such as: write a poem about subject A in the style of poet A; now write a poem about subject A in the style of poet B. Seems to provide [at least in limited tests] nearly identical answers.
Attention is All You Need. Was not just an esoteric comment about dot-product attention and transformers. It was a proclamation. With teeth.
For decades the statistical language folks had to take it from the symbolic AI camp, which made pretty good progress for its time with expert systems. But symbolic systems also stumbled at seemingly trivial tasks, like parsing the following sentence:
Time flies like a banana.
It’s pretty clear to a human what this means, but for a computer it’s pretty hard. There’s nothing in the knowledge base about “time flies”, nor what this particular species eats.
Once computers became powerful enough, statistical language modeling folks just ignored that entire issue and digested the whole internet. Strings that would start a lexical parser on fire, a GPT eats for breakfast. It’s an approach that seems to work pretty well for some things.
But statistical language models aren’t the panacea. They have issues. They hallucinate. They bullshit. They lie with such charming bravado that they can con pretty much anyone not suspecting it. In my personal opinion, we won’t have a complete solution without burying the hatchet and marrying the two camps.
Is it though? The full quote is "Time flies like an arrow, fruit flies like a banana". The humour, being the nonsense that the first phrase sets you up with a metaphor, the secod is literal.
"Time flies like a banana" means very little, because while "time flies" and "flies like bananas" make sense, as you said "time flies" aren't a thing and banana's can't fly.
As a matter of fact, ChatGPT agrees and said '"Time flies like an arrow" is a common idiomatic expression meaning that time seems to pass quickly. The phrase "time flies like a banana" does not have any meaning or usage in the English language. It may be a play on words or a made-up phrase.'
I have used chat gpt. It is very useful. I am in college and I do coding only in python but most of the college internship/placement test does not accept python code. So, previously it was very difficult to me but now I can convert it into any desired language. The best things is happening in the world.
Yeah, but it very difficult to understand any other language if you know python. I never used it to convert code in exam but I heard about this practice.
I've known many MBAs over the years but yet to understand what they actually know vs those that do not have one. I think a person with a 4 year business degree likely has a more complete business education vs an MBA imo.
The best business leaders are charismatic, great speakers and are able to inspire people. I'm not sure that these qualities are teachable but, to the extent they are, it seems that this is what should be taught in something like an MBA/executive school.
First, remember an MBA is a terminal degree. It is usually preceded by other degrees. In my case I have a bachelors in computer engineering and an MS in computer science. Many of my colleagues had PhDs in a variety of subjects both in the social sciences and in the hard sciences, and usually had several years of work experience in their field. For example, I was developer at Amazon before I went to b-school. The MBA does not suddenly erase one's technical competence. Indeed, I've written far more code since I left b-school, than I wrote before it.
Second, it is a misconception that an MBA makes you a leader. I know business schools try to make that association in their marketing materials, but that's not true. They try to prepare you for leadership roles, and some classes at some schools do indeed have specific classes on leadership. But they don't automatically qualify you as a leader.
There are some hard skills that an MBA can give you. But whether you take advantage of the opportunity or not depends on your interest. For me it was a solid grasp of accounting and finance that I continue to use in my own business to this day. The class on negotiation strategies was absolutely invaluable to me. I've used that so many times to structure my negotiations to better effect. I have enjoyed exploring game theory in the context of strategy, although I haven't had an opportunity to apply it any real-world setting. Some of my classmates focused on operations, and others on more advanced finance -- it's a case of whatever interests you.
There is very significant overlap between, what you learn in an MBA vs. an executive-MBA. But the most important difference is the usually the participants in an executive-MBA already have considerable leadership experience, and are looking less for basic leadership advice (e.g. how to negotiate) but more as an opportunity to fill in the gaps in foundational experience. Some may be missing operations or marketing experience, that to some extent an MBA compensate for.
In the end the MBA is designed to help you become familiar with a variety of business functions in a modern corporation. The nature of the corporation itself is evolving, and so are the curricula of MBA courses.
Get an MBA if you want to take time off to immerse yourself in an academic environment to really get the opportunity to think and learn about business. And the network you develop doesn't hurt either. It may not be immediately useful, but when you're in your 50s like me and are trying to bootstrap an idea, it's really useful to be able to ping the CEO of a hotshot startup, who was your classmate at school. (BTW that works even for your classmates from undergrad and high-school)
Finally remember that MBAs are people. There is a very wide diversity amongst them. Like with any other profession. So the sample of people you experience may or may not be representative of the larger population.
I don't think many of the MBAs that I have encountered have had a chance to utilize their accounting knowledge or negotiation skills (or leverage their networks). Perhaps this is the reason.
I'm always impressed by Chat GPT, but I didn't realize that the bar was this low at one of the top business schools in the country. The first four problems are arithmetic word problems. I think a smart middle-schooler could solve these. Questions 5-6 require some knowledge of queueing theory, question 7 requires you to have read the Wikipedia page on Kanban.
Besides the AI being able to interpret the questions, which is the entire point of a language model, isn't this just the same as someone googling the answers to the exam?
92 comments
[ 3.7 ms ] story [ 172 ms ] threadIt’s not that impressive of a feat. It’s no different than just googling answers to an exam.
Of course you’d do great.
If you couldn't answer it with chatgpt, you shouldn't expect an MBA course taker to answer it
The Kanban answer I found especially interesting. The equivocation on the one answer is exactly the one I would have equivocated on. (Well, yes, that's the idea but it has to be done correctly.)
I'd also observe that operations management/engineering is a lot closer to other types of engineering than things like organizational behavior are so writing this off as just an MBA thing is pretty complacent.
That was my reaction too. Donald J. Trump, a man not known for his intellect, also passed
Or that graphing calculators will result in the death of the algebra test.
Or it used to be, but now it is not.
Perhaps education will have to value cognition and ability to apply knowledge over being able to organize and communicate facts and accepted opinions. One thing is sure, a lot is going to change over the next two decades.
https://scottaaronson.blog/?p=6823
> Professor Christian Terwiesch, who authored the research paper "Would Chat GPT3 Get a Wharton MBA?...
> https://mackinstitute.wharton.upenn.edu/2023/would-chat-gpt3...
They are just calling ChatGPT "Chat GPT3" and GPT3 as "GPT3". With an acknowledgement much later in the article that it is not ChatGPT being used in this paper.
https://learnprompting.org/docs/applied_prompting/build_chat... https://github.com/karfly/chatgpt_telegram_bot
https://mackinstitute.wharton.upenn.edu/wp-content/uploads/2...
So much willful suspension of disbelief.
I tend to agree. This is exposing a (perhaps universal) bias towards confidence, correct grammar, and a general encyclopedic tone. The trappings of a correct answer, rather than explanatory power and correctness. It’s a con artist.
ChatGPT isn’t an MBA any more than someone equipped with the answer key to the test is an MBA.
E does not say none of the above on this logic game. Though many other different standardized tests do use E as none of the above. So it was pattern matching to that and not the text in front of it. And it wasn’t using explanations
Being critical about certain things isn't allowed at the moment.
Kind of like the 2023 replacement for Anti-vaxxer if you don't want to take medicine every 3 months.
My honest opinion, is that while a lot of this technology is cool and novel, it's not perfect, it has uses and then it doesn't. What it does have is a lot of good marketing and PR behind it and so you're up against a lot of that too.
One example I like is the Dall-e homepage shoes the curated "best" images generated, which might be showing only 1 in 1000 or more of the best images. I was excited when I first tried it based on the front page. But now it's drawing people with feet for hands for me and it's super super fun and entertaining to play with, I don't feel it works quite as advertised most of the time.
Disclaimer: Not anti-ai and I took my Covid19 vaccines (in the beginning, no boosters)
Why should we assume that the training set includes the answer key in the absence of evidence?
0. https://en.wikipedia.org/wiki/Argument_from_ignorance
It's significantly more likely that a model outputting the right answer was trained in that answer than not
If you think about it critically, maybe we will find there isn't actually much use for ChatGPT outside of it being a pretty cool party trick. Even if it gets better.
I think it pains people to think about it this way but if we have bullshit jobs, and now have bullshit bots to do bullshit jobs, what's really changed? Likely we'll just end up with more bullshit.
I'm sorry but I think it wouldn't hurt for you to also be objective yourself when evaluating the system your defending.
When you think about it this way, if ChatGPT has the answer to all answerable questions loaded into it's memory, is any of this really that impressive? Initially yes, in the long term maybe so.
I would argue it speeds up finding the answer to many questions which I can find online, but I pretty much had access to any answer I needed already.
My feeling is we think these machines are impressive, because we think we're impressive, and ChatGPT is doing something that looks like what we do in a new and novel way.
I also get the vibe that we almost "have" to be impressed with it because as a species, we've invest a lot into these endeavors and I guess the thought of it being not useful it just as scary as it being useful and replacing everyone's work and crashing the economy of whatever.
The psychology of it all is fascinating.
I guess we all thought steam trains were amazing and unfathomable at one point in time, is this really much different time will tell :)
Do calculators mean that we shouldn't teach multiplication anymore? That we shouldn't have tests for multiplication? The calculator will always get the right answer.
The bigger question imo is what humans are doing with these new tools. We no longer need the human computers of old now that we have calculators, and almost certainly we can innovate over automated MBAs now
That’s the whole point of Attention is All You Need, the paper that opened the door to web-scale language models. No, you don’t need a brittle and cumbersome knowledge base, at least not for this. MBA professors are perfectly happy accepting an autocomplete on steroids.
* ChatGPT wins the Nobel Prize for Literature
* ChatGPT wins a Ford Foundation grant for the arts
When I enrolled in a MBA a few years ago, I was a glorified test taker being monitored by some random person overseas to prevent cheating. Even then, many people in my cohort cheated on their tests and it is pretty well known that MBAs cheat more than other degrees.
I dropped out a year into it after learning nothing and seeing all those test answers on websites like Chegg while noticing that majority of the schools that offer an online MBA program all used the same curriculum/tests/answers and that nobody was actually teaching. I had a dream that I might learn from a business professor of the notoriety of Clayton Christensen or similar. I got a huge reality check instead.
So if ChatGPT can interpret the answer from Chegg, then it is fair game to have earned that MBA.
One kind ticks a box at big organizations where promotion above a certain level strictly requires it.
The other kind is for high-level networking, study, and development.
If you want the latter, you have a pretty small pool of schools that you're interested in.
Also curious to see if anyone has any luck asking ChatGPT for how to fix specific problems at their current companies, vs asking an MBA.
That said it can produce nearly identical answers to prompts such as: write a poem about subject A in the style of poet A; now write a poem about subject A in the style of poet B. Seems to provide [at least in limited tests] nearly identical answers.
https://openai-openai-detector.hf.space/
You'd get incomprehensible tests that humans couldn't succeed at
For decades the statistical language folks had to take it from the symbolic AI camp, which made pretty good progress for its time with expert systems. But symbolic systems also stumbled at seemingly trivial tasks, like parsing the following sentence:
Time flies like a banana.
It’s pretty clear to a human what this means, but for a computer it’s pretty hard. There’s nothing in the knowledge base about “time flies”, nor what this particular species eats.
Once computers became powerful enough, statistical language modeling folks just ignored that entire issue and digested the whole internet. Strings that would start a lexical parser on fire, a GPT eats for breakfast. It’s an approach that seems to work pretty well for some things.
But statistical language models aren’t the panacea. They have issues. They hallucinate. They bullshit. They lie with such charming bravado that they can con pretty much anyone not suspecting it. In my personal opinion, we won’t have a complete solution without burying the hatchet and marrying the two camps.
I really enjoy watching LLMs bullshit, it is like they see a path to a goal and take it.
http://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on_...
> It’s pretty clear to a human what this means.
Is it though? The full quote is "Time flies like an arrow, fruit flies like a banana". The humour, being the nonsense that the first phrase sets you up with a metaphor, the secod is literal.
"Time flies like a banana" means very little, because while "time flies" and "flies like bananas" make sense, as you said "time flies" aren't a thing and banana's can't fly.
As a matter of fact, ChatGPT agrees and said '"Time flies like an arrow" is a common idiomatic expression meaning that time seems to pass quickly. The phrase "time flies like a banana" does not have any meaning or usage in the English language. It may be a play on words or a made-up phrase.'
Deep and diverse technical understanding is what will make you last in this industry.
The best business leaders are charismatic, great speakers and are able to inspire people. I'm not sure that these qualities are teachable but, to the extent they are, it seems that this is what should be taught in something like an MBA/executive school.
First, remember an MBA is a terminal degree. It is usually preceded by other degrees. In my case I have a bachelors in computer engineering and an MS in computer science. Many of my colleagues had PhDs in a variety of subjects both in the social sciences and in the hard sciences, and usually had several years of work experience in their field. For example, I was developer at Amazon before I went to b-school. The MBA does not suddenly erase one's technical competence. Indeed, I've written far more code since I left b-school, than I wrote before it.
Second, it is a misconception that an MBA makes you a leader. I know business schools try to make that association in their marketing materials, but that's not true. They try to prepare you for leadership roles, and some classes at some schools do indeed have specific classes on leadership. But they don't automatically qualify you as a leader.
There are some hard skills that an MBA can give you. But whether you take advantage of the opportunity or not depends on your interest. For me it was a solid grasp of accounting and finance that I continue to use in my own business to this day. The class on negotiation strategies was absolutely invaluable to me. I've used that so many times to structure my negotiations to better effect. I have enjoyed exploring game theory in the context of strategy, although I haven't had an opportunity to apply it any real-world setting. Some of my classmates focused on operations, and others on more advanced finance -- it's a case of whatever interests you.
There is very significant overlap between, what you learn in an MBA vs. an executive-MBA. But the most important difference is the usually the participants in an executive-MBA already have considerable leadership experience, and are looking less for basic leadership advice (e.g. how to negotiate) but more as an opportunity to fill in the gaps in foundational experience. Some may be missing operations or marketing experience, that to some extent an MBA compensate for.
In the end the MBA is designed to help you become familiar with a variety of business functions in a modern corporation. The nature of the corporation itself is evolving, and so are the curricula of MBA courses.
Get an MBA if you want to take time off to immerse yourself in an academic environment to really get the opportunity to think and learn about business. And the network you develop doesn't hurt either. It may not be immediately useful, but when you're in your 50s like me and are trying to bootstrap an idea, it's really useful to be able to ping the CEO of a hotshot startup, who was your classmate at school. (BTW that works even for your classmates from undergrad and high-school)
Finally remember that MBAs are people. There is a very wide diversity amongst them. Like with any other profession. So the sample of people you experience may or may not be representative of the larger population.
Hope that helps.