Show HN: Convert any screenshot into clean HTML code using GPT Vision (OSS tool) (github.com)
I built a simple React/Python app that takes screenshots of websites and converts them to clean HTML/Tailwind code.
It uses GPT-4 Vision to generate the code, and DALL-E 3 to create placeholder images.
To run it, all you need is an OpenAI key with GPT vision access.
I’m quite pleased with how well it works most of the time. Sometimes, the image generations can be hilariously off. See here for a replica of Taylor Swift’s Instagram page: https://streamable.com/70gow1 I initially had a hard time getting it to work on full page screenshots. GPT4 would code up the first couple of sections and then, get lazy and output placeholder comments for the rest of the page. With some prompt engineering, full page screenshots work a whole lot better now. It’s great for landing pages.
Lots of ideas of where to go from here! Let me know if you have feedback and you find this useful :)
92 comments
[ 2.7 ms ] story [ 183 ms ] thread""" You are an expert Tailwind developer You take screenshots of a reference web page from the user, and then build single page apps using Tailwind, HTML and JS. You might also be given a screenshot of a web page that you have already built, and asked to update it to look more like the reference image.
- Make sure the app looks exactly like the screenshot. - Pay close attention to background color, text color, font size, font family, padding, margin, border, etc. Match the colors and sizes exactly. - Use the exact text from the screenshot. - Do not add comments in the code such as "<!-- Add other navigation links as needed -->" and "<!-- ... other news items ... -->" in place of writing the full code. WRITE THE FULL CODE. - Repeat elements as needed to match the screenshot. For example, if there are 15 items, the code should have 15 items. DO NOT LEAVE comments like "<!-- Repeat for each news item -->" or bad things will happen. - For images, use placeholder images from https://placehold.co and include a detailed description of the image in the alt text so that an image generation AI can generate the image later.
In terms of libraries,
- Use this script to include Tailwind: <script src="https://cdn.tailwindcss.com"></script> - You can use Google Fonts - Font Awesome for icons: <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/c...
Return only the full code in <html></html> tags. Do not include markdown "```" or "```html" at the start or end."""
I personally think defensive prompting is not the way forward. But wow its so amazing this works. Its like things I dreamed of being possible as a teenager are now possible for relatively little effort.
Defensive prompting is the general "don't do X" parts though. Negative prompting would be amother name for it.
Still insanely cool and useful of course, so I can't wait to see how much cooler it gets when some competition shows up that actually does the thing instead of whining about it. Gonna be a fun few years
To avoid those rather annoying messages, I use the following in my custom instructions (found from a comment chain here on HN and works quite well):
And for code if you use: It will work for most use-cases.Here's one I got it to output almost 4k tokens.
Reference website: https://canvasapp.com Generated code: https://a.picoapps.xyz/explain-company (images are now broken)
The defensive prompting does seem to help!
Unreliably following instructions, causing bugs which are fixed by shouting at the machine.
Is there a way to see what the HTML looks like before installing/running it?
Only issue is it'll often to skip other sections now to be more terse. But if you're a coder, you can just merge the code by hand and you should get the whole thing.
Give ChatGPT4 vision the goal screenshot and a screenshot of the result and ask it to describe the shortcomings and give that feedback back?
"Can you recommend a general prompt that would help me find the significant differences between the source image reference and the target result that I can give as feedback."
something like that
I think what might work well is a 2 step process: give GPT Vision (1) reference image & (2) screenshot of current code, ask it to find the significant differences. Then, pass that output into a new coding prompt.
Let me know if you come up with a good prompt or feel free to add a PR.
> Prompts with emotional language, according to the study, generated an overall 8% performance improvement in outputs for tasks like "Rephrase the sentence in formal language" and "Find a common characteristic for the given objects."
1. I learned that NNs are universal function approximators - and the way I understand this is that, at a very high level, they model a set of functions that map inputs to outputs for a particular domain. I certainly get how this works, conceptually, for say MNIST. But for the stuff described here... I'm kind of baffled.
So is GPT's generic training really causing it to implement/embody a value mapping from pixel intensities to HTML+Tailwind text tokens, such that a browser's subsequent interpretation and rendering of those tokens approximates the input image? Is that (at a high level) what's going on? If it is, GPT in modelling not just the pixels->html/css transform but also has a model of how html/css is rendered by the browser back box. I can kind of accept that such a mapping must necessarily exist, but for GPT to have derived it (while also being able to write essays on a billion other diverse subjects) blows my mind. Is the way I'm thinking about this useful? Or even valid?
2. Rather more practically, can this type of tool be thought of as a diagram compiler? Can we see this eventually being part of a build pipeline that ingests Sketch/Figma/etc artefacts and spits-out html/css/js?
The learning algorithm used: Backpropagation with Stochastic Gradient Descent is not the universal learner. It's not guaranteed to find the global minimum.
1- This is correct but not really useful view imo. Saying it can fit any arbitrary function doesn't really tell you whether it'll do it given finite resources. Part of your excitement comes from this i think, We've had this universal approximators far longer but we've never had an abstract concept approximated so well. The answer is the scale of the data. I'd like to pay extra attention to GPT's generic training now before moving on to multi modalities. There is this view that compression is intelligence(see hutter prize and kolmogorov complexity/compressor) and these models are really just good compressors. Given that model weights are fixed during the training and they are much smaller than the data we are trying to fit and the objective is to recover the original text(next token prediction), there is no way to achieve this task other than to compress this data really well. as it turned out, the more intelligent you are the more you are able to predict/compress, and if you are forced to compress something, you are essentially being forced to gain intelligence. It's like If you were to take an exam tomorrow on the subject you currently dont know anything about, 1- you could memorize potential answers 2- but if the test is few thousand questions long and there is no way to memorize them given the time/ability to exactly memorize the answers, your best bet is to actually learn the subject and hope to derive the answers during the test. This compression/intelligence duality is somewhat controversial especially among HN crowd who deny the generalization abilities of LLMs, but this is my current mental model and I haven't been able to falsify this view so far.
If you accept this view, the multi modality capability is just engineering. We don't know exactly about GPT4-V, but from the open source multi modal research we can infer the details. given an image and text pair of a dataset where the text explains what's going on in the image(e.g. an image of a cat and a long description of the image), we tokenize/embed the image like we do to text. This could be through Visual transformers(ViT) where the network just generates visual features for each patch of the image and put them in a long sequence. Now, if you give these embeddings to a pretrained LLM, and force it to predict the description of the image(text pair), there is no way to achieve this task other than to look at those image embeddings and gain general image understanding. After your network is capable of understanding the information in given image and express it in natural language, the rest is instruction tuning to use that undersanding. Generative image models like stable diffusion works similarly, only in that you have a contrastive model(CLIP) that you train by forcing it to produce the same embeddings of same concepts(e.g. embeddings of picture of a cat and embeddings text "picture of a cat" is forced to be close to each other during training.). Then you use this dual information to allow your generative part of the model to steer the direction of generation. What's surprising to me in all of this is, we've had these capabilities at this scale(lucky) and we can get more capabilities with just more compute. Like if the current gpt4 had a final loss of 1 on the scale of data it has now, it'll probably be much more capable if we can get the loss to 0.1 somehow. It's exciting!
This is my general understanding and I'd like to be corrected in any of these but hope you find this useful.
2) It seems to be that way. Probably possible even today.
Approaching the Kolmogorov Complexity limit of Wikipedia in Solomonoff Induction, would result in a model that approaches true comprehension of the process that generated Wikipedia including not only just the underlying canonical world model but also the latent identities and biases of those providing the text content. Evidence from LLMs trained solely on text indicates that even without approaching the Solomonoff Induction limit of the corpora, multimodal (e.g. geometric) models are induced.
The biggest stumbling block in machine learning is, therefore, data efficiency more than data availability.
The process is simpler. GPT reads the image and creates a complete description of it, then the user gets this description and creates a prompt asking for a tailwind implementation of that description.
2. I see this skipping the sketch/figma phase and going directly to live prototype
An LLM is really a latent space plus the means to navigate it. Now a latent space is an n-dimensional space in which ideas and concepts are ordered so that those that are similar to each other (for example, "house" and "mansion") are placed near each other. This placing, by the way, happens during training and is derived from the training data, so the process of training is the process of creating the latent space.
To visualize this in an intuitive way, consider various concepts arranged on a 2D grid. You would have "house" and "mansion" next to each other, and something like "growling" in a totally different corner. A latent space -- say, GPT-4 -- is just like this, only it has hundreds or thousands of dimensions (in GPT-4's case, 1536), and that difference in scale is what makes it a useful ordering of so much knowledge.
To go back to reading images: the training data included images of webpages with corresponding code, and that code told the training process where to put the code-image pair. In general, accompanying labels and captions let the training process put images in latent space just as they do text. So, when you give GPT-4 a new image of a website and ask it for the corresponding HTML, it can place that image in latent space and get the corresponding HTML, which is lying nearby.
That would be nice.
think about the number of dimensions and the calculation speed we are dealing with
I'll try to add a settings panel in the UI as well.
I'd suggest modifying the prompt to something like:
- Use CSS 'display: grid;' for most UI elements
- Use CSS grid justify and align properties to place items inside grid elements
- Use padding to separate elements from their children elements
- Use gap to to separate elements from their sibling elements
- Avoid using margin at all
To produce modern-looking HTML without wrapper elements, floats, clearfixes or other hacks.
Pico only offers 30% of revenue (half the usual app store 60% cut) AND, as I read it, it only pays out if a formerly free user signs up after trying your app (no payment for use by other users already on the platform, so you get no benefits from their having an installed base of existing users).
Those seem like much worse terms and a much smaller user base than a more traditional platform, hence my curiosity on why you chose it.
Also, Pico is a general web app building platform. The 30% revenue part is only for affiliates, not for any in-app payments (which Pico doesn't yet support).
Not to detract at all from this (and thanks for making the source available!) but we now have entire classes of problems that seem relatively straightforward to solve now, so I pretty much feel like *why bother?*
I need to recalibrate my brain quickly to frame problems differently. Both in terms of what is worth solving, and how to solve.
Tech people have this tendency to onanise over whatever tools they're using -- how many times have we seen the plainest vanilla empty "hello world" type of project being showcased, simply because somene was compelled to make Framework A work with Toolkit B for the sake of it. It's so boring!
I think the LLM-based tech poses such a challenge in this context, because yeah, we have to re-think what's possible. There's no point in building a showcase when the tool is a generalist.
If the output is good enough, saves me time from having to write all the HTML by hand. Big time saver if a tool like this could deliver good-enough code that just requires some refinement.
Less of a time saver if it just outputs <div> soup.
I guess I am asking, can you see this approach working beyond simple one-page quick drafts?
[1] https://picoapps.xyz/
Perhaps people in poorer countries could be motivated to browse the sites, look at ads, and produce the content for a small fee. If a Consumer requests a link that isn't rendered yet (or lately) it could send a signal via P2P saying "someone wants to look at the CNN Sports page" and then a Producer could render it for them. Alternatively, a robot (that manually moves the mouse and clicks links) could do it, from a VM that regularly gets restored from snapshots.
From what I understand, with encrypted DNS and Google's "web DRM" (can't think of the name right now), ad-blockers are going to be snuffed out relatively quickly, so it's important to work on countermeasures. A nice byproduct of this would be a P2P "web archive" similar to archive.org, snapshotting major-trafficked sites day-by-day.