Thanks! I should probably read the gato paper. If only there were enough hours in the day.
I can’t see where in the paper Data2Vec is reported to have contextual embeddings that work across modalities. Can you reference the section in the paper?
They are using CNNs to encode the input. But… Data2Vec uses a “standard” transformer architecture.[1] In Data2Vec2 the transformer architecture forms the “bulk of the model weights”.[2] The tweaks I eluded to should…
I’ve only scanned the blog but have read OG Data2Vec paper. Data2Vec presents an architecture that performs well across the main benchmarks for vision, speech and text. The architecture is a variation on the transformer…
Thanks! I should probably read the gato paper. If only there were enough hours in the day.
I can’t see where in the paper Data2Vec is reported to have contextual embeddings that work across modalities. Can you reference the section in the paper?
They are using CNNs to encode the input. But… Data2Vec uses a “standard” transformer architecture.[1] In Data2Vec2 the transformer architecture forms the “bulk of the model weights”.[2] The tweaks I eluded to should…
I’ve only scanned the blog but have read OG Data2Vec paper. Data2Vec presents an architecture that performs well across the main benchmarks for vision, speech and text. The architecture is a variation on the transformer…