HN Explorer

Show HN: Run TRELLIS.2 Image-to-3D generation natively on Apple Silicon (github.com)

202 points by shivampkumar ↗ HN
I ported Microsoft's TRELLIS.2 (4B parameter image-to-3D model) to run on Apple Silicon via PyTorch MPS. The original requires CUDA with flash_attn, nvdiffrast, and custom sparse convolution kernels: none of which work on Mac.

I replaced the CUDA-specific ops with pure-PyTorch alternatives: a gather-scatter sparse 3D convolution, SDPA attention for sparse transformers, and a Python-based mesh extraction replacing CUDA hashmap operations. Total changes are a few hundred lines across 9 files.

Generates ~400K vertex meshes from single photos in about 3.5 minutes on M4 Pro (24GB). Not as fast as H100 (where it takes seconds), but it works offline with no cloud dependency.

https://github.com/shivampkumar/trellis-mac

28 comments

[ 2.0 ms ] story [ 55.5 ms ] thread
villgax
That’s always been possible with MPS backend, the reason people choose to omit it in HF spaces/demos is that HF doesn’t offer an MPS backend. People would rather have the thing work at best speeds than 10x worse speeds just for compatibility.
gondar
Nice work. Although this model is not very good, I tried a lot of different image-to-3d models, the one from meshy.ai is the best, trellis is in the useless tier, really hope there could be some good open source models in this domain.
isoprophlex
Meshy is indeed great but I am terminally put off by their alltogether terrible, sleazy, gamified, opaque web UI. It's like aliexpress and a lootbox game had a baby that's into mesh generation. Ugh.
kennyloginz
So much effort, but no examples in the landing page.
post-it
How much RAM does this use? Only sitting on 8 GB right now, I'm trying to figure out if I should buy 24 GB when it's time for a replacement or spring for 32.
petargyurov
This is fantastic, great work. I will attempt to run it on my 16GB M1 but I doubt it'll run.

Out of curiosity, how did you go about replacing the CUDA specific ops? Any resources you relied on or just experience? Would love to learn more.

sebakubisz
This is the kind of porting work I always hope for when I see a CUDA-only release. Have you thought about publishing the gather-scatter sparse 3D convolution and SDPA attention swaps as a standalone toolkit or writeup? A lot of folks running models locally on Apple Silicon hit the same wall with flash_attn, nvdiffrast, and custom sparse kernels and end up redoing the same work.
sergiopreira
Most 'runs on Mac' ports are a wrapper around a cloud call or a quantized shell of the original model. Going after the CUDA-specific kernels with pure-PyTorch alternatives is the kind of work that ages well, because the next CUDA-locked research release is three weeks away. One question: how much of the gather-scatter sparse conv is reusable for other TRELLIS-like architectures, or is it bespoke to this one?
shivampkumar
The gather-scatter sparse conv should be fairly generic. Any model using 3x3x3 or 5x5x5 sparse convolutions on voxel grids could use it directly.

The main thing that's TRELLIS-specific is the neighbor cache key format, but that's a few lines to adapt.

The SDPA attention swap is even more reusable - it's just padding variable-length sequences into batches and calling torch.nn.functional.scaled_dot_product_attention.

antirez
Great. Potentially can go much faster rewriting it in terms of Metal shaders.
shivampkumar
Agreed...I've been adding some like mtlgemm, mtldiffrast from other contributors already
drbscl
Does it support multi-view input?
shivampkumar
Not currently - TRELLIS.2 is single-image input only AFAIK
drbscl
Ah, it’s still very useful though, thanks for the port!
strimoza
Cool project. I've been working on something similar in spirit — a personal video cloud (strimoza.com) — and the hardest part was also getting local playback to work reliably without internet. How are you handling memory pressure on the M-series chips with larger models?
paolatauru
solid port. the sdpa swap for sparse attention — did you notice a meaningful quality difference, or is it basically equivalent to the cuda version? curious if the pure-pytorch path added any noticeable latency hit on the m3 max
[deleted]
(comment deleted)