> At Prodia, we've started to investigate building safety systems by checking if the input prompts are within a distance threshold of known adult or illegal concepts.
This is why we can’t have nice AI things.
After my experience with RAG across a dozen models and god knows how many experiments against parts of Libgen’s archive in topics I’m familiar with, I’m not sure embeddings are actually useful for anything requiring any kind of accuracy. They’re great for low stakes purposes or as a step in a human driven workflow but like LLMs they’re a very fuzzy and often times inaccurate tool.
I've had some luck with using embeddings for categorization and for fingerprinting authors. But yeah for retrieval alone the results have been mixed. I get better results there from smaller sized text comparisons and the results haven't been terrible, just not perfect.
Both came into play, so not only what was talked about but how it was talked about. I ran through HN comments and when ranking authors by how similar they were to me, I found that the account I used years ago was within the top 10 or so closest users out of ~60k
I don't see why AI integrations shouldn't cater to a user's wants: there are a few things I don't want to see when generating images day-to-day. Using embeddings for safety filters lets you have uncensored models available in multiple modes depending on audience.
Embeddings mean that when we have a thought police they can now be more targeted and effective than before. Any thought you express can be objectively measured "using the euclidean distance or cosine similarity" for illegal concepts, and censored, corrected or punished accordingly. I imagine that this will come early for comment sections on the web.
if I invest a lot of money to make a nice forum like HN or a social media site, don't I get to determine the right policies to keep a nice private space?
doesn't it infringe my rights if people use my site and my money to harass people or to spread stuff that is against my economic interest, religion, values, etc. or that I just don't like and didn't intend?
if people are going to run around using AI to spread deepfake ragebait memes, shouldn't I get to enforce policies using the same technology they use to pollute the space?
This feels like a blend of reasonable and bad-faith arguments, like the motte just missing its bailey. Maybe strolling around the latent space would help disentangle mottes and baileys in general discourse. But then I fear it'll only push the conflict to a meta level.
that feels like obfuscation fallacy, because I have no idea what you are proposing in practice.
deciding when free speech crosses a line into a nexus to criminal activity like fraud, libel, conspiracy, incitement, disturbing the peace is hard. deciding when it crosses a line into simply violating community standards designed to further the goals of the community and forum owners, and violates the right or expectation of others for civil discourse is also hard.
it's a bias-variance problem, you can ban the n-word but people find other ways to make people unwelcome, a demagogue can say they are just asking questions when they are pretty plainly trying to start a riot and get people harassed and swatted and feel like they will suffer violence if they disagree.
I don't see anything inherently wrong with using AI tools to help make decisions like that. all that AI that controls what shows up in your feed is already implicitly doing that. inherently when one guy thinks he's making a stand for free speech, someone else is going to think he's coddling extremists who are strategically flooding the zone with bullshit, driving out civil discourse and worse.
Embeddings aren't objective, their point of view is some kind of complicated aggregate of the training data, which is itself (as of now) mostly human written text.
I'd honestly love to see AI fending off the eternal winter on message boards.
Is question redundant and basic? Direct user to a specialized AI that can explain the topic well and save the regulars from having to have the same discussion as two days ago.
But it is certainly the case that stronger machine text understanding can be used for censorship and oppression. As pretty much all powerful, general tools can be used for nefarious purposes. But it can also be used for a wide range of great purposes as well.
I'm not a mathematician but on their site it says - "Cosine similarity and Euclidean distance will result in the identical rankings" so I am assuming that would have to be the case?
When you have unit vectors, cosine similarity and euclidean distance always result in identical rankings, even when those unit vectors are assigned at random and have no semantic structure at all. That's because the euclidean distance |a - b|² = ⟨a - b, a - b⟩ = ⟨a, a⟩ - 2⟨a, b⟩ + ⟨b, b⟩ = |a|² - 2|a||b|cos ɸ + |b|² for unit vectors with |a| = |b| = 1 only depends on the cosine of the angle ɸ between the two vectors.
When I was first reading about Word2vec I thought it was absolutely wonderful that the vector delta (king - queen) was similar to the vector delta (man - woman). That captures relationships in such a fascinating way!
It is not obvious to me why word2vec's training objective yields this. word2vec ensures similarity of related words, but why can you then perform linear algebra on unrelated words?
The training objective does not have much to do with it, the bigger reason is that these neural networks themselves use linear algebra (dot products/projections and whatnot) to manipulate embeddings.
Really inspirational project. Does anyone know if there's an analog of this to the LLM space? I.e. can you take the embedding of two sentences and get a 'combined' sentence? Either by searching a corpus for the closest match or by feeding that combined embedding into an LLM that generates it.
A recent paper [1] shows that what you're describing is possible to some degree - you can reproduce text from its embeddings. The paper provides the code implementing the reversal process, so you could quickly hack together a prototype :)
I also recommend a video by Yannic Kilcher [2], explaining the paper.
isn't that how translation works in tranformers? encoder produces embedding then decoder generates the sentence. do it again with reverse translation and you get your text back in original language. combining two sentences can be done by using model for summary. making a cross-breed of two sentences is a different problem.
It seems to me like a lot of what LLMs and Image-generators do is find the interpolation two points in concept-space. So that's not vector-addition but vector-averaging. It's still arithmetic.
Perhaps I'm missing something but this looks like a heavy case of what's old is new again - the original "king - man + woman = queen" paper is nearly a decade old:
I find it so fascinating that at the end of the article the author alludes to something I've started becoming aware of:
There is a zone of illegal thoughts, that becomes definable by model-training. A physical boundary in n-dimensional concept-space. An "aligned" or "safe" AI system knows where this boundary is and does not reach inside it. Vectors (embeddings) that would probe it should instead intersect the surface like a ray-trace in graphics, and return the embedded concept at minimum distance to the safe-idea-boundary.
Intuitively, we all know what this zone is. It's the difference between being a wild barbarian and a gentleman. Or being chill vs antisocial. Seeing it in pure math is pretty awesome.
What do you mean by "illegal thoughts"? Do you think there is some authority that punishes you for having wrong thoughts? Do you suppose that gentleman are chill because they manage to not think wrong things? Is there some objective measure for what the wrong thoughts are across cultures and the political spectrum?
I'm taking issue with this comment because the way it's phrased (illegal thoughts) is worrisome. It brings up the idea that AI alignment is going down the path of controlling human thought, determined of course by certain corporate and political interests.
38 comments
[ 0.22 ms ] story [ 75.2 ms ] threadThis is why we can’t have nice AI things.
After my experience with RAG across a dozen models and god knows how many experiments against parts of Libgen’s archive in topics I’m familiar with, I’m not sure embeddings are actually useful for anything requiring any kind of accuracy. They’re great for low stakes purposes or as a step in a human driven workflow but like LLMs they’re a very fuzzy and often times inaccurate tool.
Embeddings mean that when we have a thought police they can now be more targeted and effective than before. Any thought you express can be objectively measured "using the euclidean distance or cosine similarity" for illegal concepts, and censored, corrected or punished accordingly. I imagine that this will come early for comment sections on the web.
doesn't it infringe my rights if people use my site and my money to harass people or to spread stuff that is against my economic interest, religion, values, etc. or that I just don't like and didn't intend?
if people are going to run around using AI to spread deepfake ragebait memes, shouldn't I get to enforce policies using the same technology they use to pollute the space?
deciding when free speech crosses a line into a nexus to criminal activity like fraud, libel, conspiracy, incitement, disturbing the peace is hard. deciding when it crosses a line into simply violating community standards designed to further the goals of the community and forum owners, and violates the right or expectation of others for civil discourse is also hard.
it's a bias-variance problem, you can ban the n-word but people find other ways to make people unwelcome, a demagogue can say they are just asking questions when they are pretty plainly trying to start a riot and get people harassed and swatted and feel like they will suffer violence if they disagree.
I don't see anything inherently wrong with using AI tools to help make decisions like that. all that AI that controls what shows up in your feed is already implicitly doing that. inherently when one guy thinks he's making a stand for free speech, someone else is going to think he's coddling extremists who are strategically flooding the zone with bullshit, driving out civil discourse and worse.
I'd honestly love to see AI fending off the eternal winter on message boards.
Is question redundant and basic? Direct user to a specialized AI that can explain the topic well and save the regulars from having to have the same discussion as two days ago.
But it is certainly the case that stronger machine text understanding can be used for censorship and oppression. As pretty much all powerful, general tools can be used for nefarious purposes. But it can also be used for a wide range of great purposes as well.
https://www.barneyhill.com/pages/emoji/index.html
I also recommend a video by Yannic Kilcher [2], explaining the paper.
[1] https://arxiv.org/abs/2310.06816
[2] https://www.youtube.com/watch?v=FY5j3P9tCeA
https://arxiv.org/abs/1509.01692
There is a zone of illegal thoughts, that becomes definable by model-training. A physical boundary in n-dimensional concept-space. An "aligned" or "safe" AI system knows where this boundary is and does not reach inside it. Vectors (embeddings) that would probe it should instead intersect the surface like a ray-trace in graphics, and return the embedded concept at minimum distance to the safe-idea-boundary.
Intuitively, we all know what this zone is. It's the difference between being a wild barbarian and a gentleman. Or being chill vs antisocial. Seeing it in pure math is pretty awesome.
I'm taking issue with this comment because the way it's phrased (illegal thoughts) is worrisome. It brings up the idea that AI alignment is going down the path of controlling human thought, determined of course by certain corporate and political interests.