I was kind of excited reading the headline, but this seems mostly to be a pretty straight forward application of existing homoglyph attacks that have been common for decades at this point (albeit maybe not common against these types of programs previously).
I thought it was going to be adding imperceptible changes that allow you to choose the interpretation (sort of like the visual adversarial examples) not just substitute a homoglyph and the binary classifier no longer works.
Don't get me wrong, this is still important work, i just got a bit too excited.
I think they should write their next paper about adversarial attacks on ‘eval()’.
Models aren’t designed to understand Unicode. It’s the tokenizers job to chop that up and feed it to the model properly. What this person found is a step missing from the preprocessing pipelines of these models, or, more likely, something that should be part of the steps that come way before this data gets anywhere near an ML model. So, I think saying this is an adversarial attack on the machine learning model it’s self is a bit disingenuous, because for the input they were given, they did pretty well.
A similar example to this for a computer vision model would be something like inserting some kind of malformed payload in the headers of an image. The data is already bad long before it gets to the model.
If you're interested in adversarial NLP, I also recommend reading this blog post on adversarial attacks on GPT2 with universal triggers (e.g. adding "nobody" as prefix for all inputs causes all entailments to be predicted as contradiction).
6 comments
[ 2.9 ms ] story [ 25.2 ms ] threadI thought it was going to be adding imperceptible changes that allow you to choose the interpretation (sort of like the visual adversarial examples) not just substitute a homoglyph and the binary classifier no longer works.
Don't get me wrong, this is still important work, i just got a bit too excited.
Models aren’t designed to understand Unicode. It’s the tokenizers job to chop that up and feed it to the model properly. What this person found is a step missing from the preprocessing pipelines of these models, or, more likely, something that should be part of the steps that come way before this data gets anywhere near an ML model. So, I think saying this is an adversarial attack on the machine learning model it’s self is a bit disingenuous, because for the input they were given, they did pretty well.
A similar example to this for a computer vision model would be something like inserting some kind of malformed payload in the headers of an image. The data is already bad long before it gets to the model.
If you're interested in adversarial NLP, I also recommend reading this blog post on adversarial attacks on GPT2 with universal triggers (e.g. adding "nobody" as prefix for all inputs causes all entailments to be predicted as contradiction).
https://lingojam.com/