I think the animation makes it pretty clear how this works… But I do wonder how the baseline coordinates are established?
In more realistic applications you’d have to deal with things like text that is not exactly aligned with pixel boundaries, and different anti-aliasing methods.
With such heavy pixellation being overcome, you'd have to have a seriously different font to avoid the general patterns arising from basic letter shapes. I think simply guessing "times" vs "helvetica" brings you close enough to guess the letters. Optimize score by shuffling the baselines around a bit and go through all the letters to find the right ones and you should be good. Work in blocks of 3-5 letters while shuffling them around and stretching the spacing, you might even have a fairly good guess.
The obvious problem is the relative scale of the pixelation to the underlying redacted material. The demo works because the scale encodes enough data of each character and its relationship to its neighbours. My guess would be even a 25% larger pixelation block would scrub too much for it to be at all reliable.
Then you move to guessing by syllables or whole words. Probably an order of magnitude or larger brute force job that way, but it should be possible. Especially if you target common words in the sentence first.
> once the pixels become larger it might be required to cycle through entire words.
That's a clever idea for speeding up this process actually. Instead of guessing a-z for every position, only guess valid word completions. If it doesn't end up giving a good score or the human judges it to be nonsensical, it can fall back to a-z guessing for that word.
A bit similar to my hangman solver (https://lucgommans.nl/p/hangman-solver/), which looks for the only words still possible with the given letters already known, but simpler because you only need prefix matching.
I know you meant it for very strong blurs, and there it is not actually an advantage because you need to go through thousands of words before guessing one (instead of 26×5≃130 guesses, assuming an average word length of ~5), but yeah there you'd have no other choice.
First of all, very cool little tool. I always suspected you could do this but it's really neat to watch it go.
But now just in response to the title... does anybody ever use pixelation as a redaction technique?
I feel like I've only ever seen pixelation for censoring nudity or a brand name or face or something. Actual redaction of text truly meant to be kept secret, I've only ever seen as black bars.
Are there notable cases where people have genuinely tried to redact something secret using pixelation?
Bear in mind that "redaction" just means any effort to protect sensitive information by obscuring text. I don't think I've seen pixelation used to redact text in any corporate/governmental context, but I've definitely seen people use it on sites like Reddit or Twitter to try to hide their name or other info.
I've also seen people try to redact sensitive information by poorly scribbling it out using the pencil tool on their phone, leaving enough parts of letters visible to guess what was originally there, or try to "black out" text using a brush that wasn't fully opaque, allowing it to be revealed by adjusting the contrast. Basically, a lot of people don't know how to safely redact stuff.
I don't think pixelation is commonly used because most people probably don't know how to use it, but my Samsung phone's built-in image editor also has a pixelation feature, so I honestly expect to see it pop up more often in the future.
I've seen once or twice people admitting to doing this deliberately to get around brigading/doxxing rules when they actually do want to share the username or whatever - not sure if it's allowed because "they tried" or because it's difficult enough to read that still stops most of the brigading (in many cases you can probably just search for the content to get the username anyway, if you care enough)
I've often seen pixelated text in AWS documentation screenshots and third-party tutorials. Not for anything truly critical like secret keys, but mostly for things like account ID numbers, where the exposure of it could potentially have a small impact.
This also seems like something a [convolutional] neural net should be able to learn pretty well and generating training data could be fully automated. Throw everything at it - different fonts, different sizes, bold, italic, different colors, different anti-aliasing methods, sub pixel offsets, pixelation sizes and offsets, JPEG noise, rotations, ... - and maybe also give it some language model to better reason about plausible letter combinations. I would love to see, how good this could be.
It’s not, though. It seems to be a very shallow understanding of the subject (or just insufficient consideration?) stated confidently (as so often happens these days).
As a sibling comment to yours also points out, there are plenty of examples to the contrary - models taught to recognise text from indistinct images. We’ve all been contributing to this for years via Captchas, and now we’ve got to the stage where our iPhones apply the concept locally to text found in our photo libraries.
It was my understanding that anything trying to recover text is still hard to do well, but I could be wrong - I have not worked with image models in awhile. Classification and OCR is truly well solved though.
You are probably thinking of diffusion generation models. Models in general work absolutely fine with text. Just about the first convolution neural network examples you come across are basic character classification examples.
A neural net was my first guess as to how this would work but was pleasantly surprised at the actual technique used, which makes a lot of sense. The animated gif does a great job of explaining it on its own.
Looking closely, this only works if you know the font name, size, and weight used, or at least can guess it, manually, before feeding the pixelated version into the tool? Still quite fun, but not as scary as the headline made it sound...
Guessing is actually easy. For the kinds of files that end up as redacted pdfs (legal, government, etc), there's probably 5-8 font options that make up 98% of documents. Sizes and weights are immediately recognizable to the slightly trained eye. I'm pretty sure I could guess all 3 attributes at a glance.
It's also a proof of concept. Slap a couple more for() loops in there to iterate through different font options and try a range of alignments and you could have it fully automatic.
If you're nearsighted taking your glasses off can blur the pixelation of a face so that you can get a pretty good idea of the person's appearance, especially if you are just trying to guess if it's really a specific person you are already familiar with.
I worry this often leads to a sort of observation bias, where people assign their expectation to the situation. If a video is blurry, it’s not hard to convince an audience that the face in the video is a particular person. Human brains are really great at filling in details, whether they’re accurate or not.
We have this advent calendar at work where one person on the team posts a pixelated scene from a christmas movie and we guess which one it is. Very hard, but when they later post the non-pixelated version it looks so obvious. Then if I look at the pixelated version again I can now "clearly" see what it is.
Might be fun to use diffusion to make adversarial/false answers to the pixelated scenes, depending on how pixelated they are. If they're quite pixelated you could probably come up with some crazy alternatives.
You could probably do the same for Guassian blur too, right? At least if you could get a reasonable guess for the parameters of the blur. Any case of obfuscation where plaintext and ciphertext characters have a 1:1 relationship should be trivial to undo.
Is there any approach to guessing a single redacted word/phrase where the length of the word is revealed? (Such as in a PDF where the rectangle is automatically drawn around the characters.)
I've wanted to build something that would if nothing else run through a list of guesses (assuming the font is the same as surrounding text) and see if any of them could match size-wise, but not sure of an easy way to deal with the PDF part of it.
> not sure of an easy way to deal with the PDF part of it.
If you aren't doing this so often that you need automation, you could sidestep that issue by just taking screenshots at 400% zoom or so and accurately measure how many pixels each letter (a-z) takes, as well as how many pixels a space is (the censored part might be into the spacing on each side of the word), and measure how many pixels the gap is between the words surrounding the censored part, then
for word in wordlist do
# Start by accounting for the spaces
wordsize = charwidths[' '] * 2
for char in word do
wordsize += charwidths[char]
done
if wordsize == gap_size then
print("Possible word: " + word)
endif
done
Probably want to do the gap_size +/- 1 or so, but that's how I'd approach this for a given document. A starting point for a wordlist on many linux systems could be `/usr/share/dict/words`.
Same. And since it's pixelated, you don't even need to match the font. Just pick the same color, make it roughly the correct size and pixelate. They'll be none the wiser.
This works if done right but it's surprisingly easy to mess up. A lot of people have fallen in to the trap where the tool they use didn't put a perfectly black rectangle in and the text can be extracted out. And on PDFs you can very easily end up just adding a layer over the top which can be removed later.
Let’s say you redact with a pure black rectangle, on a JPEG. Is there enough information in the remaining compressed image to leak some information about what was redacted?
I ask because what was there would have affected the original compression algorithm. So there might be some signal left.
No? If it were to be visible it would be because your editing software failed to blend the layers together properly during compositing and rasterization. JPEG encoding comes after this.
I think they mean the original compression of the original text file (if in jpeg) would have crafted specific DCT coefficients that reach across to the secret text. When that is later decoded to pixels and re-encoded, new DCTs are burned into the image, but only on top of the previous.
That's not how it works. it is "on top" of the previous, sure, but the same way as if I add 8 to 2 and give you 10, you can't possibly get the original number is 2.
Let’s say you’re covering a red pixel with a black pixel. There’s no way that red pixel will in any way “survive JPEG compression” because the compression happens on the raw pixel data; the raw pixel data no longer has a red pixel.
If the red pixel affected the encoding of nearby pixels in the compressed original, which you've now neglected to redact, then the data very well could persist.
They're saying that before encoding the red dot could have been X pixels wide, and to the naked eye it would seem that after encoding it's still only X pixels wide... but due to compression artifacts it may now actually be affecting an area X+Y pixels wide. So only covering X pixels could leak Y pixels of information.
It honestly sounds a bit paranoid to me, but it's definitely possible.
> Let’s say you’re covering a red pixel with a black pixel. There’s no way that red pixel will in any way “survive JPEG compression” because the compression happens on the raw pixel data; the raw pixel data no longer has a red pixel.
This is trivially untrue for JPEG; JPEG often stores chroma info as 4:2:0, that is at a quarter the resolution of luminance info. In this common case, covering a single red pixel in the decoded form only removes a quarter of the chroma information from its presence (and this is easy to not notice if, for example, the other three pixels that share its chrominance are nearly black).
Except maybe your 2 was previously stored in the LossyNumber(TM) format which can only approximate integers, and it turned into 2.033. You add 8 to it, send it to me in a less lossy encoding, and I see 10.033. My knowledge of LossyNumber artifacts lets me know you started with 2!
JPEG format normally compresses pictures in 8x8 blocks, and these blocks are aligned so the first one starts at top-left corner of the image. I think these leaks you’re talking about are limited to the inside of the 8x8 blocks.
So, if you want to redact a rectangle between pixels [ 12, 4 ] – [ 34, 18 ], redact [ 8, 0 ] – [ 40, 24 ] instead, this way the black rectangle will cover complete blocks.
I don’t think what you’re saying makes sense. If you covered all information, no information is left. The blocks are irrelevant since 100% of the information is gone. There’s no “history leak” in JPEG, which is basically what’s being asked.
The only issue I see is meta tags and embedded thumbnails, which some cameras do. Stripping EXIF is always required.
> If you covered all information, no information is left.
JPEG compression causes ringing artifacts that can extend out past the pixels you're trying to redact, up to the next block boundary. This is a problem if what you're trying to redact is already a JPEG, rather than a pristine uncompressed image.
> JPEG compression causes ringing artifacts that can extend out past the pixels you're trying to redact, up to the next block boundary.
This.
I made a response to the parent question that goes into the practical side of this in a bit more detail, but this is a much better one-sentence summary of /how/ the information leaks.
(Monochrome pictures in 8x8 blocks, 4:2:0-downsampled colour images in 16x16 blocks; 4:2:2 and non-downsampled 4:4:4 colour images are technically possible but I’ve never seen one.)
Is another low tech way simply print screening the redacted image + cropping? Naturally you will lose resolution if the OS/app is downsampling the image to render but could be good enough and a good hedge against the image format somehow encoding history. AFAIK, some law firms still redact, print, scan for this reason.
No there is no information passing through if the opacity is 100%. I have seen people attempting to redact using the pen or paintbrush tool, however, and opacity is often not 100% when doing that.
I have also seen people try to redact by drawing a black block on a new layer in Photoshop, for example, and then exporting as PDF. They're unaware that the layers are being exported in the PDF and the block can be removed. Always flatten your document completely before export.
If it its text, then it leaks the length of the text that was redacted. Something to keep in mind. So you may want to actually move the remaining text spacing a bit one way or the other.
That’s trivial to get. Once we know the pixel word length and do some frequency analysis for estimates, along with an analysis of subtle compression artifacts that typically form in the white space surrounding various characters or combinations of characters, it’s game over for you, we will find the redacted content with reasonable certainty.
This a very valid concern. Yes, indeed, the compression might leak out certain bits out to the neighboring pixels. Especially in this sequence: raw -> lossy (jpeg) -> redaction -> less lossy or lossess (png).
Whether that's enough to decode the redacted information will depend on how many pixels were redacted and other intricacies of JPEG.
This is not too dissimilar to compression oracle attacks, such as CRIME (https://en.wikipedia.org/wiki/CRIME), an infamous vulnerability in HTTPS.
Well it's sure smarter than some government agencies 5-10 years ago that thought they could redact things by drawing black boxes over them in a PDF editing application and saving it, blithely not knowing all they were doing was creating new "block of black area" vector overlays that could be easily removed.
It was before, I remember it made it to the news because of the documents regarding the death in Iraq of Nicola Calipari, that was early 2005, so some 17 years ago:
Yes, information leaks this way, if you redact a previously-JPEG-compressed image with a non-block-aligned black rectangle.
Consider the following case: You have limited previously knowledge of the unredacted document; for example, you know that it was a pure black-and-white document that was compressed as a standard color JPEG (not too unusual). Suppose further that in the block extending from (16, 16) to (24, 24) (exclusive), you wish to redact the single pixel at 23, 23, presumedly as the upper left corner of a larger black bar. Finally, suppose that the post-redacted image is losslessly compressed; this simplifies the argument, but is not strictly necessary.
In the way we've phrased this problem, there are only two values for the original color of the redacted pixel; it was black, or white. The rest of the pixels in the block are known in their encoded form (unredacted), but also known in their pre-encoded form (pure black or pure white, while JPEG compression will put them somewhere into dark gray or light gray, respectively). With all known pixels set to their pre-encoded form, each possible value of the redacted pixel can be considered, using (known, deduced, or guessed) JPEG compression parameters to regenerate the compressed block. Whichever value of the redacted pixel leads to the observed encoded values of the unredacted pixels is the correct value of the redacted pixel.
(In practice, the problem might be less constrained; and you might have multiple but less than a block of pixels redacted; these make it harder to make use of the leaked information, but do not change that some information has indeed leaked.)
Random guess: the writer doesn't know how to pronounce that name correctly.
(Of course, you don't actually have to know how to pronounce something in order to write it! But if you decide what to write by saying it to yourself, you might find yourself instinctively reaching for some alternative turn of phrase.)
I've always thought something could be done with faces when you blur them out in video. The fact that the grid moves and the colors change make me intuitively feel as if I could guess their face. I bet the 3 letter agencies already have something like that.
There are AI algorithms that 'enhance' blurred faces, however, the output 'looks' like real human face, but in all likelihood, different from the ground truth (i.e. the real face that got blurred). Therefore caution must be exercised in advocating such algorithms (e.g. in court of law)
Whereas this algorithm is 'just' enumerating the text, generating blur and finding the closest match to the original blur.
Sure, but you could create physical models, rotate those physical models on top of the video, and see if certain faces are a better match. Obviously it helps if there's more pixels and it doesn't need to be proof in a court of law to be useful in some way, or at the very least an interesting experiment. Maybe another algorithm could do the trick. It does seem like a lot of information about a face is leaked when you have multiple samples, a moving grid sampling different parts of the face, etc. I mean, just by watching certain pixels get extra dark when they pass over parts of the face you can figure out approximately where the eyes are. That's probably enough to narrow down a large pool of suspects to a small one.
I don't have a link at hand but yes, I believe I've seen this done fairly well somewhat recently. Modeling the actual faces might give false positives as someone else said, but depending on how strongly the blurring/pixelation is performed, what the movement of the camera is, and how many frames you have access to, there's a lot of information in those blotches and you can reconstruct arbitrary images from underneath.
I agree. Each frame is new information causing new quantisation decisions. Especially if there is a limited pool of potential people and you can get good photographs of them from various angles (which these days is virtually everyone if you have an agency budget), and you can estimate the head position from torso movement, then try out and see which match the best. TV programs love blurred faces and voices, and they should really use a stand-in for the actual person for interview shots.
I doubt the US TLAs have it, because the software they have is often rubbish generated by a cartel of low skill providers, and frequently 10 years behind the times. But I bet the Chinese Ministry of State Security has it. So so many dissidents to keep track of, to identify after they have moved overseas and apply pressure to.
Here's a simplified example of your scenario. Imagine a video of a pattern of black dots on a white surface, moving around. The video is pixelated in an attempt to hide the precise layout of the pattern. But the nearest-neighbour pixelation would make it easy to see when a black dot moves into different pixel. Therefore you'd be able to reconstruct the pattern precisely.
>One of the first things you might notice is that it has a curious bit of coloring in it. What gives?
>I’m actually not 100% sure why this happens
It's subpixel rendering which exploits the way pixels are laid out physically on a monitor (1 logical pixel is actually three physical subpixels: red, green, blue). It produces crisper texts.
144 comments
[ 3.5 ms ] story [ 206 ms ] threadIn more realistic applications you’d have to deal with things like text that is not exactly aligned with pixel boundaries, and different anti-aliasing methods.
E.g. a page to do fold recognition on, or even to figure out what software (e.g. Gmail) and to look up what font the software uses.
Photo identification is done with a limited search space of “tiles” that the image is decomposed into, for convolutional NNs.
What you mention would be a random permutation, which seems vialirt ot vercore given its small search space.
Also the example seems to go through one letter at a time, once the pixels become larger it might be required to cycle through entire words.
That's a clever idea for speeding up this process actually. Instead of guessing a-z for every position, only guess valid word completions. If it doesn't end up giving a good score or the human judges it to be nonsensical, it can fall back to a-z guessing for that word.
A bit similar to my hangman solver (https://lucgommans.nl/p/hangman-solver/), which looks for the only words still possible with the given letters already known, but simpler because you only need prefix matching.
I know you meant it for very strong blurs, and there it is not actually an advantage because you need to go through thousands of words before guessing one (instead of 26×5≃130 guesses, assuming an average word length of ~5), but yeah there you'd have no other choice.
But now just in response to the title... does anybody ever use pixelation as a redaction technique?
I feel like I've only ever seen pixelation for censoring nudity or a brand name or face or something. Actual redaction of text truly meant to be kept secret, I've only ever seen as black bars.
Are there notable cases where people have genuinely tried to redact something secret using pixelation?
I've also seen people try to redact sensitive information by poorly scribbling it out using the pencil tool on their phone, leaving enough parts of letters visible to guess what was originally there, or try to "black out" text using a brush that wasn't fully opaque, allowing it to be revealed by adjusting the contrast. Basically, a lot of people don't know how to safely redact stuff.
I don't think pixelation is commonly used because most people probably don't know how to use it, but my Samsung phone's built-in image editor also has a pixelation feature, so I honestly expect to see it pop up more often in the future.
A colored square is easy — and destructively removes what was there, by setting that part of the image to a chosen color.
As a sibling comment to yours also points out, there are plenty of examples to the contrary - models taught to recognise text from indistinct images. We’ve all been contributing to this for years via Captchas, and now we’ve got to the stage where our iPhones apply the concept locally to text found in our photo libraries.
Don't use text pixelation to redact sensitive information - https://news.ycombinator.com/item?id=30350626 - Feb 2022 (163 comments)
If you're nearsighted taking your glasses off can blur the pixelation of a face so that you can get a pretty good idea of the person's appearance, especially if you are just trying to guess if it's really a specific person you are already familiar with.
Now, pair that with a Prosopagnosia disorder... Right back where you started?
If you need redaction, black it out completely.
http://smartdeblur.net/
before the above the author made a version which was free (but old link is now dead):
https://github.com/Y-Vladimir/SmartDeblur/downloads
LEO's have access to a Commercial tool called Amped Five that is said to be very good at it:
https://ampedsoftware.com/five
I've wanted to build something that would if nothing else run through a list of guesses (assuming the font is the same as surrounding text) and see if any of them could match size-wise, but not sure of an easy way to deal with the PDF part of it.
If you aren't doing this so often that you need automation, you could sidestep that issue by just taking screenshots at 400% zoom or so and accurately measure how many pixels each letter (a-z) takes, as well as how many pixels a space is (the censored part might be into the spacing on each side of the word), and measure how many pixels the gap is between the words surrounding the censored part, then
Probably want to do the gap_size +/- 1 or so, but that's how I'd approach this for a given document. A starting point for a wordlist on many linux systems could be `/usr/share/dict/words`.which provokes ye olde zen koan: if a joke is made in a pixelated font whose typeface will never be known, is it still funny? :)
idk, is it? :^) https://imgur.com/1OvGeAF
If it's secure enough for intelligence agencies, it's secure enough for me.
I ask because what was there would have affected the original compression algorithm. So there might be some signal left.
Let’s say you’re covering a red pixel with a black pixel. There’s no way that red pixel will in any way “survive JPEG compression” because the compression happens on the raw pixel data; the raw pixel data no longer has a red pixel.
They're saying that before encoding the red dot could have been X pixels wide, and to the naked eye it would seem that after encoding it's still only X pixels wide... but due to compression artifacts it may now actually be affecting an area X+Y pixels wide. So only covering X pixels could leak Y pixels of information.
It honestly sounds a bit paranoid to me, but it's definitely possible.
This is trivially untrue for JPEG; JPEG often stores chroma info as 4:2:0, that is at a quarter the resolution of luminance info. In this common case, covering a single red pixel in the decoded form only removes a quarter of the chroma information from its presence (and this is easy to not notice if, for example, the other three pixels that share its chrominance are nearly black).
So, if you want to redact a rectangle between pixels [ 12, 4 ] – [ 34, 18 ], redact [ 8, 0 ] – [ 40, 24 ] instead, this way the black rectangle will cover complete blocks.
The only issue I see is meta tags and embedded thumbnails, which some cameras do. Stripping EXIF is always required.
JPEG compression causes ringing artifacts that can extend out past the pixels you're trying to redact, up to the next block boundary. This is a problem if what you're trying to redact is already a JPEG, rather than a pristine uncompressed image.
This.
I made a response to the parent question that goes into the practical side of this in a bit more detail, but this is a much better one-sentence summary of /how/ the information leaks.
I have also seen people try to redact by drawing a black block on a new layer in Photoshop, for example, and then exporting as PDF. They're unaware that the layers are being exported in the PDF and the block can be removed. Always flatten your document completely before export.
Whether that's enough to decode the redacted information will depend on how many pixels were redacted and other intricacies of JPEG.
This is not too dissimilar to compression oracle attacks, such as CRIME (https://en.wikipedia.org/wiki/CRIME), an infamous vulnerability in HTTPS.
https://en.wikipedia.org/wiki/Rescue_of_Giuliana_Sgrena
https://en.wikipedia.org/wiki/Rescue_of_Giuliana_Sgrena#Rele...
Consider the following case: You have limited previously knowledge of the unredacted document; for example, you know that it was a pure black-and-white document that was compressed as a standard color JPEG (not too unusual). Suppose further that in the block extending from (16, 16) to (24, 24) (exclusive), you wish to redact the single pixel at 23, 23, presumedly as the upper left corner of a larger black bar. Finally, suppose that the post-redacted image is losslessly compressed; this simplifies the argument, but is not strictly necessary.
In the way we've phrased this problem, there are only two values for the original color of the redacted pixel; it was black, or white. The rest of the pixels in the block are known in their encoded form (unredacted), but also known in their pre-encoded form (pure black or pure white, while JPEG compression will put them somewhere into dark gray or light gray, respectively). With all known pixels set to their pre-encoded form, each possible value of the redacted pixel can be considered, using (known, deduced, or guessed) JPEG compression parameters to regenerate the compressed block. Whichever value of the redacted pixel leads to the observed encoded values of the unredacted pixels is the correct value of the redacted pixel.
(In practice, the problem might be less constrained; and you might have multiple but less than a block of pixels redacted; these make it harder to make use of the leaked information, but do not change that some information has indeed leaked.)
Programs that appear to black out a section of image, but actually leak information.
(Of course, you don't actually have to know how to pronounce something in order to write it! But if you decide what to write by saying it to yourself, you might find yourself instinctively reaching for some alternative turn of phrase.)
There are AI algorithms that 'enhance' blurred faces, however, the output 'looks' like real human face, but in all likelihood, different from the ground truth (i.e. the real face that got blurred). Therefore caution must be exercised in advocating such algorithms (e.g. in court of law)
Whereas this algorithm is 'just' enumerating the text, generating blur and finding the closest match to the original blur.
I doubt the US TLAs have it, because the software they have is often rubbish generated by a cartel of low skill providers, and frequently 10 years behind the times. But I bet the Chinese Ministry of State Security has it. So so many dissidents to keep track of, to identify after they have moved overseas and apply pressure to.
>One of the first things you might notice is that it has a curious bit of coloring in it. What gives?
>I’m actually not 100% sure why this happens
It's subpixel rendering which exploits the way pixels are laid out physically on a monitor (1 logical pixel is actually three physical subpixels: red, green, blue). It produces crisper texts.
https://en.m.wikipedia.org/wiki/Subpixel_rendering
[0] https://en.wikipedia.org/wiki/Christopher_Paul_Neil
But just imagine the face of someone getting Rickrolled in this fashion.