Using Multimodal LLMs to Understand UI Elements on Websites (qa.tech) 11 points by daniel_mp 2y ago ↗ HN
[–] patricklef 2y ago ↗ MLLMs are surprisingly bad at this out of the box and to some extent even with fine tuning. https://jina.ai/news/the-what-and-why-of-text-image-modality...
[–] while1 2y ago ↗ Loving this! Very surprising that the LLMs of today are so bad at understanding interfaces but it also makes it a very interesting case for finetuning!
[–] antonoo 2y ago ↗ Love that you made an interactive app to visualize how the model performs similar to how Meta AI usually releases their models(direct link: https://qa-tech-minicpm-demo.gptengineer.run) [–] antonoo 2y ago ↗ Did you use AI to generate this app? [–] daniel_mp 2y ago ↗ yeah, we used https://gptengineer.app/ to create the playground! highly recommended
[–] antonoo 2y ago ↗ Did you use AI to generate this app? [–] daniel_mp 2y ago ↗ yeah, we used https://gptengineer.app/ to create the playground! highly recommended
[–] daniel_mp 2y ago ↗ yeah, we used https://gptengineer.app/ to create the playground! highly recommended
6 comments
[ 70.7 ms ] story [ 511 ms ] thread(direct link: https://qa-tech-minicpm-demo.gptengineer.run)