HN Explorer

Ask HN: Any truly multi-modal transformer architectures?

3 points by prats226 ↗ HN
Most of the multi-modal architectures consume images as tokens in same dimension. Any architectures which look at text and images as first class citizens and also produce image tokens interleaved with text?

4 comments

[ 9.2 ms ] story [ 42.7 ms ] thread
kadushka
What do you mean? We want images and text to live in the same latent space, and be represented by similar vectors if the two correlate. How else would you want to do it?
prats226
In most multimodal archiectures, images and text is fed as input and they live in same latent space, however major difference is most networks are trained to decode latent vector into text but never an image (Atleast in same network). This makes the network more biased towards text I feel and treats image embeddings as second class citizen?
kadushka
Here's a good starting point if you want to know more about this topic: https://arxiv.org/abs/2103.00020
prats226
This is CLIP. Here if you would see, the model is pretrained to ingest both images and text. However if you would see the prediction mode, its basically text generation only. None of the multi-modal architectures think of both image and text as first class citizens