HN Explorer

Ask HN: Is ChatGPT becoming useless for many tasks?

18 points by prmph ↗ HN
I asked ChatGPT 4 to convert some code using a rather simple pattern, only to get get this message:

> Sorry, but I can't convert this as per your requirement.

Not explanation, no attempt to even do something. Not sure if this is a recent development; I thought v4 was supposed to be even more powerful that v3 (which at least gives me something).

I also constantly have to stop myself from asking it to perform even relatively simple programming tasks. For example, no matter what I did or the hints I provided (including sample code), it could not write anything resembling a correct implementation of the the following function signature:

  const tokenizeWords = (
    str: string, 
    args?: {
      separatorCaseChanges?: | "upper" | "lower" | "all" | "none", // default is "upper"
      separatorChars?: Array<string>, // default is white-space
    }
  ): string[] ...
It could not even write unit tests for the function without making errors in test cases all over.

At this point it quite clear to me that it is more of a time waster than useful. I think the hype surrounding LLMs and AI coding assistants is far beyond the reality. Developers, you can relax knowing that your jobs are safe for the foreseeable future.

11 comments

[ 1.4 ms ] story [ 24.5 ms ] thread
latexr
Submitted a few minutes after your question: https://news.ycombinator.com/item?id=39156643
razodactyl
Interesting read. Makes me wonder if it's fresh statistics appearing since we're now a few years in on such tools.

Imagine a developer who ignores all principles of development and ships a product vs someone who bike-sheds all day. Copilot definitely allows code to be written faster and in greater volume but it's never been one to provide perfect code most of the time.

I love it for menial work though, writing out structs and boilerplate.

kadir1234
this begs the question - what was it useful for to begin with? I am not a coder, so CoPilot has never been for me (so I cannot speak for that use case)... but for creative/writing/content-generation... feel like good content needed to be heavily edited and selected in any case... I wouldn't say that has gotten worse, but it was never quite good to begin with unless your bar was quite low.
subjectsigma
Not really a developer but I write code in Python and Java a few times a week. Recently I had to edit some JavaScript.

I also found ChatGPT to be mostly useless. In all three languages, the only hard problems were interacting with libraries that ChatGPT knows nothing about. I really wanted to say something like “Create a function that pulls data from this internal API, calls a library function, and then displays it as a React component”, but it took more time trying to get ChatGPT to understand how the API and libraries worked that just actually writing the damn code

speedgoose
Have you tried to re-roll when you get a bad answer? If you use the user interface it has a button for that purpose.
razodactyl
No, the model has probably diverged over time to what you're used to. Imagine if Google search suddenly had new quirks to get the searches you want (actually come to think of it, maybe I need to delete my account and start fresh because it never gives me results I want anymore).

It's the way you're prompting your model. Additionally, I've been training my own toy models to understand the workings so I can also state that they're sensitive to WHERE in the conversation you're asking the question. Some have even pointed out that they're more belligerent depending on the day of the week they're provided.

It's a fuzzy computer program that changes its abilities based on instructions and prompts.... just because it's communicating doesn't mean it truly understands, it's more like: surface as much of the similar features by mentioning the most appropriate keywords and suddenly it can perform the tasks...

jasonjmcghee
For one, try saying “rewrite” instead of “convert”. Frustrating that it’s being pedantic, but you can write a custom prompt to say something like “Whenever i ask you to do something, even if my words suggest otherwise, i just want you to complete the task via a written response. For example, if i ask you to build a program, you just need to write the code” (etc)

For what it’s worth i spent a bunch of time prompt engineering this gpt for writing code https://chat.openai.com/g/g-7k9sZvoD7-the-full-imp

If you ask it for its prompt it should output it.

gardenhedge
I've cancelled my sub for now. I'll use Phind and Bing in the meantime while I wait to hear if ChatGPT improves.
jryb
It's hit and miss but the responses i get are all super detailed and usually quite helpful. I've never had interactions like yours. I'm mostly coding in Python with some Rust fwiw