HN Explorer

Show HN: How to use ChatGPT+ARKit to script experiences with natural language (github.com)

54 points by trzy ↗ HN
Video demonstration: https://youtu.be/bJFa5HOmhl4

I created a demo involving ChatGPT, OpenAI's Whisper running on-device for speech-to-text (because why not?), and ARKit. By defining a custom JavaScript environment with JavaScriptCore and describing it to ChatGPT, I was able to get it to produce functioning scripts that instantiate and manipulate 3D objects.

Sketchfab is used to import 3D assets on the fly. Google Poly had a much better catalog of assets for this sort of thing but sadly, it's no longer available.

The README.md describes how it all works but the gist of it is that user prompts are wrapped in a larger prompt that describes the environment (i.e., which functions are available to use and how they work) and constraints. The code is then executed directly. Assets for objects are fetched as needed by searching Sketchfab.

18 comments

[ 3.2 ms ] story [ 58.0 ms ] thread
coyotespike
I love this, a real future shock moment. Jarvis wen??

Have you seen LangChain? It has a similar idea of getting GPT to output runnable code, and then using that to search the web or run Python, and giving the results back to GPT.

trzy
Thanks for sharing that! I hadn't heard of LangChain. That looks super cool. I really hope ChatGPT gets turned into a proper API because fighting OpenAI's bot countermeasures is a real time sink.
ladberg
JavaScriptCore is definitely underrated! I've used it for quickly prototyping business logic of apps because you can iterate without having to recompile, reload, or even reopen the app on device. You can also use Safari to open up a web inspector for the context in your app and do some remote debugging.
trzy
Question: I couldn’t figure out how to export properties that were writable in JS. I describe the problem in Entity.js. Any pointers would be much appreciated! :)
ladberg
Unfortunately I also can't figure that out...

The autogenerated bindings from JS <-> ObjC/Swift are pretty opaque and if it doesn't handle your use case then there isn't much you can do without manually binding everything you need.

I might recommend using a custom class with getters/setters that don't use the subscript operator. I think on the JS side you can run some code to override [] and point it to the correct functions but I've never tried it out so ymmv.

trzy
Well, I feel less bad about myself for getting stuck there :D The semantics of how properties work in JS are so obnoxious. A read-only property can still be replaced entirely, which means e.g.:

entity.position = [0,0,0];

Is ok but:

entity.position[0] = 1; entity.position[1] = 1; entity.position[3] = 1;

Fails (and ChatGPT occasionally tries to do the latter).

photoGrant
This just gave me chills knowing at some point in the future, Children will be able to dream up their perfect fantasy land and program their surroundings by just asking for it. Then these children will become adults. The future is (undefined)
trzy
I would love for this to become robust enough to give to kids and see what they dream up. Unlike adults, kids are far less inhibited at coming up with fun prompts, but that also means they would break this admittedly brittle prototype :D
RHSman2
A world that has clean air and no inequality?
cloudmike
It's great to see you applying the latest tech to this idea.

We got pretty far with this a few years ago using more basic ML/NLP. The app was called Moatboat: https://twitter.com/moatboat/status/1082425681210859520

The area I think is most exciting (and in need of more innovation) is using natural language to create (and modify!) the actual simulation / rules / behaviors. Our approach was to map language outputs to actions that could be chained together using Goal Oriented Action Planning plus an Entity Component System. The user's verbs / prepositions / etc. would add layers of goals, each of which would enable or disable certain behavior components when triggered.

More details here for anyone interested: https://medium.com/@mikejohnstn/whatever-you-say-happens-2fa...

Directly generating source code from natural language would be a fun alternative approach to try today.

trzy
This is very cool! And I think it's a perfectly good approach. My understanding is that the usual way of doing things with LLMs is to train them on a series of specialized tokens that represent actions in your environment. E.g., this sequence of words results in this completion of action tokens.

The code approach is intriguing and I'd like to explore further but controllability is a real problem and bullet-proofing it would require a lot of effort, if it's even possible at all. I do think that a hybrid environment where one speaks what they want and then sees the code and can interact with it in a friendly way would be very intriguing for a sandbox experience.

paul7986
This so cool and I need to learn ARKit...there's so many cool things to be created using Apple's future Smart Glasses.

Anyone know if there's an iPhone developer boot camp with a focus on ARKit around anywhere?

To me it's like the start of the iPhone ..a land grab of innovation to be created.

trzy
I’m not really sure what the state of ARKit learning materials is but there are great tutorials online by e.g., Ray Wenderlich. Getting started with Xcode and iOS can be a bit of a struggle. Follow some tutorials and take it step by step. Apple has some pretty good samples, too.

Unfortunately one big problem is the quality of Apple’s documentation. It’s nowhere near as good as e.g., the Unity engine documentation or Microsoft’s API docs. At best, it’s useful for discovering what exists and then you either figure it out or look for specific examples online.

paul7986
Thanks, I just need to kick my own butt and get motivated. If I found a iPhone development bootcamp that also offered ARKit that would be great motivation for me. Heck I'd even pay an iPhone dev a lot of money to be my tutor and motivation. I have so many ideas for Apple's Smart Glasses.

I do have ten years of Front End Design/Development experience. So, am skilled in HTML/CSS/JS, thus not a complete noob.

redgetan
Cool project. Does the code generated from ChatGPT for ARKit always work? Or are there instances where it doesn't do what you want?
trzy
There are many cases where it does not work or does something insane. However, the number of hooks exposed are very low and I’m impressed with how often it works. When it goes off the rails, it is usually because it 1) tries to use functions that don’t exist (the distance() function is one that I had to relent and implement; but I’ve also seen euclideanDistance() occasionally), 2) defines functions of its own after invoking them in code, or 3) misunderstands the relationship between the user (whose position is exposed through “cameraPosition”) and planes. I’ve put very little effort into mitigating these but have found that simply adding functions for it to use in common scenarios helps a lot.

I think the window size of ChatGPT is quite limited so I can’t use prompts that are arbitrarily long but it’s not hard to see how increasing this window size and fine tuning on custom APIs could produce a model that would work more often than not.

matthieubulte
This idea is genius, I love it! It inspired me to do exactly the same thing for my app. I built a small UI/server to interactively build routines via natural language by teaching ChatGPT about our APIs and automatically running the code coming out of ChatGPT. This is a small video of the result: https://twitter.com/matthieubulte/status/1606057139645992960