Show HN: AI climbing coach – visualize how to climb any route based on your body (climbing.ai)
I made SABR - an AI model that helps you visualize the beta/technique on any route, based on your body parameters. You can input a video of you climbing any route, in any orientation or lighting condition (it's truly versatile!). SABR then creates a virtual avatar of your body shape and uses it to climb the route you're climbing. Then, you can compare/contrast.
You can see the demo here: https://www.youtube.com/watch?v=cnvNPWoYZz4
Will be open sourcing the model, backend, and frontend codebase soon!
148 comments
[ 2.9 ms ] story [ 202 ms ] threadCan I choose the color of the avatar? (It's not an important feature, but people will love it.)
Does it work with natural walls? (I guess the artificial walls have "handles" with an standard size that makes guesing the scale easier for the computer.)
- don't have that functionality but its really easy to implement.
- ideally, it works for anything. I will attach some outputs of it climbing outdoor routes. The model doesn't know what a wall means. It just has seen enough data of people climbing that it can somehow correlate certain features in videos to certain human movements.
(Disclaimer: I never climbed a wall, but from time to time I fall in the https://www.youtube.com/@AlbertOkay rabit hole. In particular this video looks relevant for my question https://www.youtube.com/watch?v=onXrJ5sX9BU (but I din't watch it yet.))
Will attach more example outputs and make a detailed document about how the model was built and the research behind it. If this is interesting to you, feel free to sign up to the waitlist on www.climbing.ai (and make sure to sign up for our Discord!)
I originally planned on open sourcing the model, data, weights, and code. Only a few people (me and a couple friends) have access to the hosted model on the web app.
If enough people are on the waitlist, I will consider releasing access. This is very expensive to run so was only considering open sourcing it.
Note: the model works well sometimes, but most of the times it does not. This is an early research preview. Please tamper any expectations. A really good general model requires millions of videos and much more training time so it is really prohibitively expensive. As mentioned before, if someone has the compute, all they have to do is scale the existing dataset and training pipeline (which I will publish open-source in the coming weeks).
I assume, at minimum, there's:
Worst case able to build this, scenario, author builds something else entirely.
I think OSS is best for this category right now.
- if you're using SMPL body parameters this will have to stay research / open-source - is this leveraging some sort of monocular depth estimation to estimate the wall in 3D space? Also, do you have assumed camera parameters, or is that also estimated? If there isn't any depth information, this will be highly inaccurate on any cliff routes, but still useful on flat wall climbing.
Overall, a good idea (that I've also thought about building as a climber) - the tricky part that I'm impressed you have a solution to is path planning up the wall. Even assuming a flat wall with no depth estimation, it's still looks effective.
I'm assuming this is evolutionary / brute force of some nature, given OP's comments about it being expensive to run.
This is an end-end system that just takes in video frames. Camera parameters are one of the things that is predicted. It gives promising results for a wide variety of environments (cliffs, diff types of bouldering walls, diff outdoor walls, etc.), though not always accurate. Path planning is also part of the end-end system. Will share more details in the paper.
Also, this claims that the wall type or video quality doesn't matter, but I have a hard time understanding how the model would be able to understand that a small crimp could possibly be dual textured and therefore has only a few specific ways of approaching it.
So it seems that this is more for visualizing a climb (which is a skill most climbers should develop) and not really for dialing in some sort of microbeta for a problem.
(Climbing rock without ropes may be more "pure" but is so dangerous that it should never be idolized.)
And then there's indoors vs. outdoors, with some dedicated outdoor climbers regarding anything done in a climbing gym as "not real climbing".
Most people don't take this literally, and it's generally considered to be a standing joke in climbing. Sadly, though, some people take it very seriously.
As for the usefullness of the software, I'm sceptical too as it don't really solve a problem. But maybe I'm not seeing it and it could be good for beginners :) A good improvement would be adding a comparison between you and the model in term of body position and fluidity of movements.
The "Boss" from Pusher is arguably the most famous climbing hold ever made. For a decade or more, every gym had one, but they were all unique. Lots of them had micro chips that became critical to usage of the hold. Some had decent texture and some were glassy smooth from years and years and years of use. A lot of the accidental variation in new holds has gone away as the industry has standardized around a handful of industrial fabricators like Aragon, but even over the course of a single indoor boulder problem's life, the accumulation of chalk, sweat, and shoe rubber can have a significant impact on how a hold climbs.
I guess the real question is, do these changes just make routes harder or do they make them fundamentally different? Do they actually change the set of moves that constitutes the easiest way to the top? To be honest, I'm not entirely sure. But it's something interesting to think about.
It's an interesting project and it could be fun to watch, but it's completely useless.
To me, the customer here would be climbing gyms, offering a service to climbers.
3 being accomplished by reasoning "if a movement should be possible using the identified hold, but no one successfully does it, the hold must be misidentified or have different properties."To which I pointed out that, with enough data, you could reason backwards to figure out their properties.
Assuming that's solved, if the question is "What is the point?" then I'd answer the same point as golf swing analysis -- structured comparison feedback for continual improvement.
"Have you thought about trying X move at Y point?" or "You're trying X move at Y point, but here's how you differ from someone successfully doing it" both seem useful feedback.
And essentially what's manually generated now, from someone watching and then providing feedback.
With regards to strength, hell, if you wanted to get fancy you could also deduce a specific user's strength, comparing their moves against others' moves on the same features.
Even if you know the exact hold model and it’s in pristine condition, it’s basically impossible to tell how it’s gonna work from a single angle on video at a distance. Even tiny variations in angle of the wall and rotation of the hold on the wall can completely change how you use it.
So, in terms of solving complicated beta faster, I see real utility to this.
It is very interesting that since the AI climber is trained on actual climbers, it could, in principle, provide beta to climb consistent with your own style. If you train the bot exclusively on footage of yourself, it would return movement based on your style. If your style is finessy-all-backstep-all-the-time (aka The Edlinger), it can provide beta consistent with that. If your style is to square up and pull (otherwise known as The American), it can provide beta consistent with that instead.
> It is very interesting that since the AI climber is trained on actual climbers, it could, in principle, provide beta to climb consistent with your own style. If you train the bot exclusively on footage of yourself, it would return movement based on your style. If your style is finessy-all-backstep-all-the-time (aka The Edlinger), it can provide beta consistent with that. If your style is to square up and pull (otherwise known as The American), it can provide beta consistent with that instead.
I would think this is actually a Bad Thing. It's very easy to get stuck trying to make a sequence fit your style of climbing. The better approach (especially for long term skill acquisition) is a willingness to learn new styles. That's to say that every sequence is only solvable via one particular style, but I think long term development is hindered if you approach every crux with the one thing you are good at.
> So, in terms of solving complicated beta faster, I see real utility to this.
I can agree with this. But, to the point that others have made, I do wonder what this and the availability of beta videos for many, many routes and blocs does to climbing skill overall. Perhaps I'm just a grumpy old man, but, particularly when bouldering, sorting out the beta should be part of the journey toward eventually sending. Last fall, I visited Hueco Tanks after a six year absence. I suppose I was a bit disappointed to see so many people watching YouTube beta videos of nearly every problem they tried.
That's a fair concern. That said, there are certain sequences that I'm physically incapable of doing without a dedicated multi-month program of stretching beforehand. Turns out falling on a thumbs-down jam is close enough to a shoulder dislocation that I maybe should've done some PT about it. The extreme is, obviously, the Edlinger vs. American example, but I think the middle ground actually addresses people's peculiar body geometry and/or range of motion. Alex Honnold's exact hip or elbow position might not be as meaningful for Ashima, even if they're on the exact same route.
Such nuances have made a difference in some cases (the one that springs to mind is Todd Skinner's observation that Steve Petro's hips sagged just a little mid-crux on Fiddler on the Roof, which, until corrected, had prevented Petro from nabbing the first ascent of one of the hardest cracks ever climbed to that point). Probably a net-bad for folks projecting Midnight Lightning or similar, but definitely useful for somebody looking to repeat Silence or whatever.
Of the full distribution of possible video qualities one can take on a modern phone camera, the vast majority of video qualities will be fine for the AI to understand fine details. Obviously, if you somehow or for some reason, take a video with really bad quality, it will not give you what you want.
Same explanation goes for the walls. If you take a video of just a really dark wall with really bad holds, it is probably won't give you what you want either.
The point is to try to solve it yourself, then compare with an expert solution, and therefore learn how to improve.
If you're just blindly trying to problem-solve through trial and error without ever comparing against expert feedback, you're going to learn climbling extremely slowly...
It's an essential part of the sport - the satisfaction of using one's own body and mind to overcome the seemingly impossible.
If you never get stuck, how could you possibly experience it?
When faced with a challenge, you want to figure out as much as you reasonably can, and then learn what you were missing.
Struggling at the same problem for several sessons, or god forbid years, sounds like misery to me. I'd rather use that time productively learning, rather than struggling for the sake of it, because I refuse to learn from others.
Echoing some of the other comments, I think it would be more interesting to not just see a single avatar climbing the problem, but to see many possible approaches to the same problem, even for climbers with the same body type. This way, even skilled climbers that are stuck on a problem can consider alternate beta/techniques that they may find easier to execute.
I do BJJ and wonder how it could AI models could help. I find watching tape to help, not only myself but everyone who I know has recorded their matches has found it helpful.
Does it just tell you how it would climb things you can already climb?
I have definitely discovered in my life that I learn probably 10x faster in situations where I can try something, and then immediately compare it with an expert solution. Rather than just trying and trying and trying on my own.
And once you're experienced in climbing (like you probably are), then of course you don't need to compare anymore. The joy is in the solving. But expert solutions are really helpful to get to the point where you can consider yourself experienced, so much more quickly.
This is about wall climbing where there's lots of choice for moves.
Bouldering arguably has a larger set of moves than "wall climbing", but again the physicality of this sport is the crux for most people. One can study and maybe drill all the "advanced" moves, but without the flexibility and strength to execute on microholds, that knowledge is useless.
It could be interesting to combine something like this with an analysis tool. Analyse a climbers attempts or successes and compare it to the beta figured out by the model. Then offer tips on body positioning or technique based on your weight/height/strength.
- Like all of these things, your training data matters and the internet is awash with videos of people climbing badly. A lot of people specifically post "I can't climb this, what am I doing wrong?" videos. World cup climbers are, by the nature of the competition, extremely talented and technically proficient climbers. Even when they fail, they fail in smart interesting ways.
- There's lots of high quality video footage out there. Heck, the problems are even set with visual clarity in mind which would help when parsing that footage. There's potentially enough video to train instances on individual climbers. You could run side by sides like "How would Tamoa climb this and how would Janja climb this?".
- World cup problems are stylistically distinct. They involve lots of moves "typical" climbers will never ever encounter. Many climbers will look at a typical gym problem and think "I have an idea of how to climb this" but will look at a world cup problem and just think "????????". An app that told you how a problem like that should be climbed might be useful.
There are drawbacks too.
- World cup climbers are outliers, whose physical ability (strength, flexibility, etc.) give them access to kinds of movement that other climbers just don't have. No amount of "knowing the sequence" will get me up a climb that requires a full bat hang (look it up) because I just don't have the ankle strength to do the movement.
- World cup "style" is only commonly used at high level comps and in very large commercial gyms. It's probably not extremely relevant to a typical climbing session.
- World cup problems are very hard. Mostly v10 and up? It would be hilarious to watch a model trained on genetic monsters crushing the world's hardest boulder problems try to tell a doughy office worker (me) how to climb v2.
One also learns to just recalibrate according to regions. E.g. climbing 11d at one gym, versus 12a at another.
It is ok that it is a solution looking for a problem. There is obviously no 'business' or 'product' here. It isn't like there is a payment link on the page or anything.
What I'd like to see the comments focus on is that we should just be happy that someone is making the effort to learn more about AI and building tooling around it. Experimentation is king.
They've put their work out into the open (soon to be open sourced even!), not to be criticized over whether it is useful or not, but just that they created something that could spawn other interesting things that solve real world use cases.
Huge kudos for doing this work.
The route setters would effectively do what you're saying, while they were setting the route. They knew exactly what would work and what wouldn't. The holds/wall are really what dictate things and if you're not able to climb it... it isn't really their problem.
As far as I can tell, while this project is cool, it really has nothing to do with climbing or route setting. It is an AI project where the developer just kind of made up some task and had AI follow it.
Just like movie dorks will happily spend hours explaining how an individual shot in a movie is actually an insider reference to another movie, and as a result a statement of intent for the movie as a whole, professional route setters will talk your ear off about the way one of their problems embraces or rejects specific kinds of movement trends of the last 6 months.
That intentional rejection is interesting. Many route setters, especially for competitions, are in constant search of novelty. One kind of perfect problem is something that looks confusing and impossible, up until you see it done, at which point it seems almost obvious. It's the feeling of solving a sudoku. But critically, they want climbers to be initially confused.
I wonder if AI might actually be better than humans at sequencing these kinds of problems. Humans bring so much context and experience and expectation to the process that we are easily tricked. AI just looks through a few terabytes of video and says "What about this?".
Probably a deeply unpopular take here, but without knowing anything about climbing routes, I'm gonna say no. I'm not saying that they won't have excellent quality output that might even solve problems that human output can't, but the process of creating something is meaningful, even commercially. Surely this will be useful in some respects, but I just don't buy the idea that humanity is destined to passively consume automated algorithm-generated utility products-- especially creative ones-- no matter how smooth, cheap, and clever they might be.
That's true, but why does it mean that the answer to the more or less objective question "will AI actually be better than humans at sequencing these kinds of problems?" (As stated it's not really objective, but one could easily come up with metrics like, say, total time to a correct solution, or time spent observing the route or other climbers, or ….) One can imagine other, related questions that are less objective (like "will it be a good idea to integrate this AI assistance into climbing competitions?"), but, to me, the answer to the (implicit) original question has nothing to do with whether or not the activity is meaningful, or with humans' destiny one way or the other.
1. You don’t like what that would mean about the destiny for humanity.
2. A human making it makes it inherently better.
3. If it’s lower quality, it’s lower quality.
I get why this would lead to strong beliefs. But these arguments aren’t very convincing.
Embedded in your comment is the idea that AI might create boulder problems or routes in climbing gyms, and the human (or eventually robot) just follows that plan in bolting the holds to the wall. I expect that for a long time, AI generated climbing routes would rarely be good, but would consistently be physiologically impossible, feature uninteresting movement, or be too easy.
Its easy enough to shotgun holds up onto the wall based on some imagined sequence, the real skill of route setting is to (as the GP pointed out) figure out what's physically possible and also fun and challenging.
Ah, I see.
> The AI just generates potential solutions to the problem once the holds are found/placed
Yeah and I think that's really going to be the sweet spot for generative tools for the forseeable future.
> Its easy enough to shotgun holds up onto the wall based on some imagined sequence, the real skill of route setting is to (as the GP pointed out) figure out what's physically possible and also fun and challenging.
Right right. I have a feeling that making a more convincing substitute is primarily a matter of having less access to data than say, paintings and photography which are certainly not less nuanced than this creative task. But as I said, a lot of people care about how something was made, too. I'll bet that's going to be a much bigger factor, at least in marketing, than many realize in the near future.
This would follow the exact path image GenAI evolved through.
Step 1: Teach a model to recognize objects from noisy data.
Step 2: Reverse-feed that model random noise and force it to hallucinate that noise back into likely objects.
As there's probably physics simulation at some point in this particular scenario, there'd probably also be step 3 of simulating a climb through the generated path to validate feasibility / specific qualities.
It doesn't sound impossible.
What's really wild to me is how somebody would recognize a mid-tier poster from a website I thought effectively defunct for nearly 10 years now.
...then this would be very valuable data.
Eventually you could train AI to generate routes, and then you could fire all of your master setters!
Climb setters should unionize and copyright their art before it's too late (like for painters and software developers and musicians)
Figuring a route for the first time is by far the most rewarding part of climbing (although I love all of it). Same with kids - if they grok something on their own, what a reaction and reward compared to being told how to do it.
Edit: Unsurprised to see the demo on a gym route instead of actual rock.
If you never feel like you need help from watching someone else climb, then I think you are not trying hard enough problems.
I believe this is consistent with most elite (or elite-aspiring) athletes from many sports.
While I personally enjoy the problem-solving aspect of climbing (when I rarely do get out), I absolutely see the value in this project (and other climbing apps that thrive on beta sharing)