Testing AI/LLMs to see how well they can build a website from scratch

2 points by swax 2y ago ↗ HN

Wrote an open source command shell wrapper for LLMs to use. Just testing it with with writing a website from scratch. Comparing GPT-4 and Claude 3 Opus. The results: Pretty similar, they can write a PHP page, with components, styles, and active elements like a fun quiz on one of the pages.

Video: https://www.youtube.com/watch?v=4TnaJLq0E-o

Article: https://naisys.org/articles/3-dollar-website-gpt4-vs-claude-opus

GitHub repo: https://github.com/swax/NAISYS

2 comments

[ 2.6 ms ] story [ 15.1 ms ] thread

matt3210 2y ago ↗

A real test would be to invent a framework and see if it can do anything on the new framework. Otherwise it’s just a search engine that amalgamates pieces

swax 2y ago ↗

Amalgamating the pieces is the test. LLMs really can't do it that well.

Is it context size, training, number of iterations, number of agents? That's what's being tried to improve the results.