HN Explorer

8 comments

[ 3.0 ms ] story [ 100 ms ] thread
krmbzds
Side note: Add this to your robots.txt if you don't want ChatGPT to scrape your website:

  User-Agent: ChatGPT-User
  User-Agent: GPTBot
  Disallow: /
Sources:

- https://platform.openai.com/docs/gptbot

- https://platform.openai.com/docs/plugins/bot

GaggiX
Well this has little to do with the article here, ChatGPT is not accessing the website itself, I even think that this feature is disable at the moment.
[deleted]
(comment deleted)
krmbzds
They opened browsing feature for Plus and Enterprise users 2 days ago.
jondwillis
Side note to your side note: the official beta ChatGPT browser plugin’s user agent is Python/3.9 aiohttp/3.8.4 (I just tested it on a domain that I control)

And like a sibling comment says, any plugins/agents that are trying to scrape will have various fingerprints.

xbpx
If you're feeding the specific nodes to scrape, iterating and bug fixing what is Chatgpt doing other than giving you someone to talk with while you code?

Call me when I can ask a LLM to pull structured data in CSV form from website X and deliver it to me each morning. And it does it.

all2
This doesn't seem too difficult of an ask. Let me see if I can scrape something together.