afaik those are user IDs of Chinese(?) messengers. Devs or others there like to use those as usernames because the are (of course) mostly unique and also enable others to easily contact them.
I don't think in this case it'll be messenger IDs (and if it is definitely not chinese) as GCHQ is the UK's equivalent of the NSA.
They're a pretty secretive bunch, when they present in person they don't use real names, and if you go to their headquarters you have to leave all electronics at the door (did a talk there once and had to buy myself an analog watch for the day!)
Chinese messenger ID-as-username mostly starts from q(QQ - counterpart FB Messenger) and s(Sina Weibo - counterpart FB/Twitter). Others(a - admob, b - bilibili etc.) are quite rare tbh so I don't think that's the case here.
I have one of its predecessors (2GB USB 2.0 flash drives). Costed around $60 back then and it was worth it. There's hardly a day i don't use it.
After a very long time of daily usage one of the knifes broke. Totally my own fault. I sent Victorinox an email to thank them for their excellent product that lasted a long time of abuse. They replied to mail the thing to them so they could get it repaired for free. Everything was replaced, except the flash drive. Awesome thing, awesome service!
It's amazing that some companies can still do that. But then you get things like L.L. Bean ending policies because assholes are buying at Goodwill and then sending them in to be refurbed for free.
I'm a big fan of CyberChef. One of its most useful features is "magic" and turning on "intensive mode". This will automatically detect the encoding used and can often detect 2-3 levels of encoding.
If you like that, try FTFY https://ftfy.readthedocs.io/en/latest/ which can automatically repair a huge range of ways that Unicode text can be broken by re-encoding. This would be a great addition to CyberChef if they could reproduce it in JS.
Classic example from the docs:
> ftfy can fix multiple layers of mojibake simultaneously:
>>> ftfy.fix_text('The Mona Lisa doesn’t have eyebrows.')
"The Mona Lisa doesn't have eyebrows."
It's a static website. If you want to use it as an application you can download the compiled page (https://gchq.github.io/CyberChef/CyberChef_v9.32.3.zip) and create a shortcut to the 'Cyberchef.html' file contained within.
Flutter was a promising choice as it'd give me Windows, Mac OS and Linux build from a single codebase - and even the possibility of orgs running this internally as a self-hosted webapp. But one of the ideas I had was to bundle CyberChef with my app and open in a webview. Turns out, Flutter doesn't support webviews on desktop platforms at all. https://github.com/nileshtrivedi/devtoolbox/issues/4
You should check out the WebViewX plugin, for simple websites it works without any drawbacks, even though it's pretty hard to find by just searching for it
This is actually very useful. I've been spending the past few weeks working on a cross-platform, native dev assistant app like this written with GTK3 and Rust. It's been a really good exercise in designing meaningful GUIs and, well, usable code. I don't have anything to show right now, but when I get it to an MVP state I'll be sure to share it with everyone!
I'm curious to hear what things people want to see out of an app like this. What utilities are you constantly Googling for that you'd rather have on your desktop?
Something I run into a lot is a JSON-like blob of text I'd like to be formatted as JSON (new lines and indentation). Most JSON formatters choke on improper JSON (understandably). It would be great to have a tool that was more lax. Like browser support for terrible HTML lax haha. If I could paste JSON-like strings into a text area and have it fix and format it as best as can be, that would be great. Some examples of non-JSON syntax to handle would be like single quotes instead of double, arbitrary JSON nodes (not necessarily wrapped in `{}`), some pre or post text (some non-JSON text at the beginning or end), comments amongst the JSON, etc. Another JSON aspect that would be useful is something to escape/unescape JSON (specifically double quotes). I deal a lot with JSON that includes escaped JSON in values and it would be great to have some better way to visualize and process those blobs. Sublime Text has a nice plugin to handle some of the escaping/unescaping[1].
I feel like I am forever having to hack together things to parse json to csv. It feels like there is never an easy path (indeed jq is frustrating, I usually end up just solving in python)
ht-editor was a fantastic editor[1], similar to hiew[2] on windows. Unfortunately ht-editor codebase is a bit hard to extend and it's based on really old modified binutils headers. I was trying to update it, but it's probably better to just write it from scratch, it's still much more straightforward to use that most modern cli hex editors.
What is with russians and their love for advanced windows cli tools? IDA Pro, hiew and far manager[3] come to mind.
Not a fan, I have tried say formatting a large JSON API results (with no formatting) and it just runs out of memory because:
1) the string is huge like 64mb+
2) it tries to do syntax highlighting and blows up big time on the resulting formatted value.
For a utility tool it's awful finnicky on real world data and I think tries to be pretty at times rather than useful, or perhaps a case where a browser makes a poor Editor.
We might need to try fine tuning some limits. If CyberChef thinks it will have an issue rendering some text, we'll stop trying to display it and offer the user the ability to render only a part of the result or download the file.
Here's an example of me trying to format a 100mb file.
It is very useful. I have issues with how both GCHQ and NSA operate, but I also really like the tools they provide like Cyberchef and Ghidra as they automate a lot of workflows.
The 'magic' command in cyberchef is pretty magic, especially for obscure (to me) utf and language encodings, though I have certainly run down more than a few rabbit holes because it detects the file magic for 'inflate' compression in pretty much everything, which would be perfect for hiding embedded files because of the number of false positives that appear when you're looking for them, and it's just the algorithm someone of a certain vintage who was serious about hiding something from everyone but someone else of that era would use.
The image analysis stuff is great. I use palette randomization for detecting embeds, and the entropy analyzer/visualizer is great fast method for detecting encrypted and zipped payloads. I like that it's fast, and it's there on the web so I can use it on anything.
Any hints on challenge #5? FromHex returns something that looks like it has the bz2 compression header. Trying to decompress w/ bz2 doesn't seem to work though.
42 5a 68 39 31 41 59 26 53 59 34 3d 45 44 3d 31 37 3d 44 45 3d 30 30 3d 30 30 28 3d 44 46 3d 38 30 3d 30 30 3d 31 30 68 3d 30 37 3d 46 37 3d 46 30 3d 30 43 3d 30 30 66 3d 30 30 3d 33 46 3d 45 46 3d 44 46 3d 46 30 30 3d 30 30 3d 0d 0a 3d 44 41 3d 38 31 48 3d 43 34 3d 43 38 3d 44 30 68 3d 30 33 43 3d 30 38 3d 30 30 3d 30 30 3d 43 30 3d 43 38 32 3d 30 30 3d 30 43 46 3d 38 33 21 3d 39 30 3d 31 38 35 4f 3d 31 34 3d 46 34 46 3d 39 34 3d 46 43 52 6f 42 3d 0d 0a 3d 39 45 3d 41 37 3d 41 36 6a 46 3d 39 41 63 3d 31 32 6d 46 3d 30 34 3a 32 3d 43 38 52 5d 52 7a 68 3d 31 30 3d 31 31 3d 30 31 3d 41 41 3d 31 46 3d 38 36 3d 31 45 3d 42 30 52 3d 42 41 3d 30 42 5e 3d 31 36 25 3d 41 46 3d 30 37 3d 0d 0a 3d 46 31 3d 44 37 38 74 3d 43 46 3d 42 37 3d 31 36 3d 39 35 3d 38 42 3d 30 35 3d 45 41 3d 39 34 30 44 3d 46 31 3d 31 34 3d 43 33 3d 32 32 62 6a 3d 32 32 3d 38 43 6d 3d 41 41 3d 45 35 3d 41 34 3d 44 45 3d 39 34 53 3d 42 34 2e 3d 0d 0a 3d 41 36 3d 42 33 3d 41 43 3d 31 42 3d 38 30 3d 30 38 3d 33 46 46 3d 44 41 2c 49 3d 43 35 3d 42 38 09 28 3d 41 32 72 35 3d 30 38 40 3d 46 41 2a 3d 31 45 61 3d 44 45 5e 3d 39 45 3d 44 42 3d 31 41 3d 43 33 3d 38 42 26 3d 0d 0a 3d 43 44 3d 39 32 3d 44 30 3d 43 32 3d 31 32 3d 41 41 2a 3d 30 33 48 3d 46 38 45 3d 45 38 30 3d 41 35 3d 43 35 40 3d 44 45 3d 46 30 3d 31 37 3d 43 38 3d 44 34 3d 41 30 3d 39 46 3d 43 45 3d 46 34 20 76 3d 30 36 2c 3d 0d 0a 43 6e 3d 45 37 3d 39 30 3d 44 39 3d 43 31 64 3d 30 35 3d 42 39 3d 30 30 58 3d 44 32 48 3d 46 41 3d 41 46 3d 41 44 3d 38 38 3d 38 30 30 3d 31 38 3d 46 37 3d 39 30 3d 31 32 4d 3c 3e 76 3d 30 45 6b 3d 38 42 5b 3d 42 46 3d 0d 0a 3d 45 36 3d 31 30 3d 38 46 3d 44 31 3d 44 31 3d 45 41 3d 45 42 3d 39 44 44 3d 44 30 3d 44 31 3d 43 34 74 3d 38 43 2b 3d 41 38 5e 3d 43 42 31 3a 50 3d 42 37 3d 30 38 3d 42 43 09 3d 0d 0a 3d 41 32 3d 31 37 3d 41 37 3d 31 46 3d 46 31 77 24 53 3d 38 35 09 3d 30 33 4e 3d 44 31 7d 3d 45 30
I just got through this one and was having the same block as you, took me a while to connect the dots. There's a step missing, take a look at some possible encodings before you decompress. Hope that helps without giving too much away, good luck!
I always wondered if any of the online tools like JSONlint or any other tools that manipulate data would save it or try to extract data as it is often used to indent or decode potentially sensitive data.
55 comments
[ 2.4 ms ] story [ 117 ms ] threadYou can find more contributors here:
https://github.com/gchq/CyberChef/graphs/contributors
Among others :)They're a pretty secretive bunch, when they present in person they don't use real names, and if you go to their headquarters you have to leave all electronics at the door (did a talk there once and had to buy myself an analog watch for the day!)
[1]: https://github.com/gchq/CyberChef/commit/c423de545fd0d27aabe...
After a very long time of daily usage one of the knifes broke. Totally my own fault. I sent Victorinox an email to thank them for their excellent product that lasted a long time of abuse. They replied to mail the thing to them so they could get it repaired for free. Everything was replaced, except the flash drive. Awesome thing, awesome service!
After a recommendation on HN, I carry one of these on my keychain: https://www.amazon.com/Samsung-BAR-Plus-128GB-MUF-128BE3/dp/... . Essentially indestructible, as everything is solid metal casing.
Classic example from the docs:
> ftfy can fix multiple layers of mojibake simultaneously:
https://news.ycombinator.com/item?id=20767183
https://news.ycombinator.com/item?id=13056254
CyberChef – Cyber Swiss Army Knife - https://news.ycombinator.com/item?id=20767183 - Aug 2019 (59 comments)
CyberChef - The Cyber Swiss Army Knife - https://news.ycombinator.com/item?id=20543810 - July 2019 (1 comment)
CyberChef – The Cyber Swiss Army Knife - https://news.ycombinator.com/item?id=13099687 - Dec 2016 (1 comment)
CyberChef – A Cyber Swiss Army Knife - https://news.ycombinator.com/item?id=13056254 - Nov 2016 (139 comments)
Flutter was a promising choice as it'd give me Windows, Mac OS and Linux build from a single codebase - and even the possibility of orgs running this internally as a self-hosted webapp. But one of the ideas I had was to bundle CyberChef with my app and open in a webview. Turns out, Flutter doesn't support webviews on desktop platforms at all. https://github.com/nileshtrivedi/devtoolbox/issues/4
https://github.com/abhimanyu003/sttr
Since it's written in GO, you can run it pretty much anywhere you like :) ( cross platform )
Yes, it support file inputs as well.
I'm curious to hear what things people want to see out of an app like this. What utilities are you constantly Googling for that you'd rather have on your desktop?
[1] https://github.com/Nadock/json_stringify
I'm still looking for my ultimate hex editor (TUI/GUI) for offline use. Not been impressed with any of the many options I've tried on Linux.
What is with russians and their love for advanced windows cli tools? IDA Pro, hiew and far manager[3] come to mind.
[1] http://hte.sourceforge.net/
[2] https://www.hiew.ru/
[3] https://www.farmanager.com/
https://github.com/gchq/CyberChef
For a utility tool it's awful finnicky on real world data and I think tries to be pretty at times rather than useful, or perhaps a case where a browser makes a poor Editor.
Here's an example of me trying to format a 100mb file.
[0] https://imgur.com/a/3DtqnuL
The 'magic' command in cyberchef is pretty magic, especially for obscure (to me) utf and language encodings, though I have certainly run down more than a few rabbit holes because it detects the file magic for 'inflate' compression in pretty much everything, which would be perfect for hiding embedded files because of the number of false positives that appear when you're looking for them, and it's just the algorithm someone of a certain vintage who was serious about hiding something from everyone but someone else of that era would use.
The image analysis stuff is great. I use palette randomization for detecting embeds, and the entropy analyzer/visualizer is great fast method for detecting encrypted and zipped payloads. I like that it's fast, and it's there on the web so I can use it on anything.
I appreciate the magnitude of cyberchefs operations, but having to switch to the browser for these tasks can be cumbersome at times.
[0] https://chepy.readthedocs.io/en/latest/
Edit: I had missed that cyberchef offers a node API exposing most of its operations: https://github.com/gchq/CyberChef/wiki/Node-API
43 6f 6e 67 72 61 74 75 6c 61 74 69 6f 6e 73 2c 20 79 6f 75 20 68 61 76 65 20 63 6f 6d 70 6c 65 74 65 64 20 43 79 62 65 72 43 68 65 66 20 63 68 61 6c 6c 65 6e 67 65 20 23 31 21 0a 0a 54 68 69 73 20 63 68 61 6c 6c 65 6e 67 65 20 65 78 70 6c 6f 72 65 64 20 68 65 78 61 64 65 63 69 6d 61 6c 20 65 6e 63 6f 64 69 6e 67 2e 20 54 6f 20 6c 65 61 72 6e 20 6d 6f 72 65 2c 20 76 69 73 69 74 20 77 69 6b 69 70 65 64 69 61 2e 6f 72 67 2f 77 69 6b 69 2f 48 65 78 61 64 65 63 69 6d 61 6c 2e 0a 0a 54 68 65 20 63 6f 64 65 20 66 6f 72 20 74 68 69 73 20 63 68 61 6c 6c 65 6e 67 65 20 69 73 20 39 64 34 63 62 63 65 66 2d 62 65 35 32 2d 34 37 35 31 2d 61 32 62 32 2d 38 33 33 38 65 36 34 30 39 34 31 36 20 28 6b 65 65 70 20 74 68 69 73 20 70 72 69 76 61 74 65 29 2e 0a 0a 54 68 65 20 6e 65 78 74 20 63 68 61 6c 6c 65 6e 67 65 20 63 61 6e 20 62 65 20 66 6f 75 6e 64 20 61 74 20 68 74 74 70 73 3a 2f 2f 70 61 73 74 65 62 69 6e 2e 63 6f 6d 2f 47 53 6e 54 41 6d 6b 56 2e
42 5a 68 39 31 41 59 26 53 59 34 3d 45 44 3d 31 37 3d 44 45 3d 30 30 3d 30 30 28 3d 44 46 3d 38 30 3d 30 30 3d 31 30 68 3d 30 37 3d 46 37 3d 46 30 3d 30 43 3d 30 30 66 3d 30 30 3d 33 46 3d 45 46 3d 44 46 3d 46 30 30 3d 30 30 3d 0d 0a 3d 44 41 3d 38 31 48 3d 43 34 3d 43 38 3d 44 30 68 3d 30 33 43 3d 30 38 3d 30 30 3d 30 30 3d 43 30 3d 43 38 32 3d 30 30 3d 30 43 46 3d 38 33 21 3d 39 30 3d 31 38 35 4f 3d 31 34 3d 46 34 46 3d 39 34 3d 46 43 52 6f 42 3d 0d 0a 3d 39 45 3d 41 37 3d 41 36 6a 46 3d 39 41 63 3d 31 32 6d 46 3d 30 34 3a 32 3d 43 38 52 5d 52 7a 68 3d 31 30 3d 31 31 3d 30 31 3d 41 41 3d 31 46 3d 38 36 3d 31 45 3d 42 30 52 3d 42 41 3d 30 42 5e 3d 31 36 25 3d 41 46 3d 30 37 3d 0d 0a 3d 46 31 3d 44 37 38 74 3d 43 46 3d 42 37 3d 31 36 3d 39 35 3d 38 42 3d 30 35 3d 45 41 3d 39 34 30 44 3d 46 31 3d 31 34 3d 43 33 3d 32 32 62 6a 3d 32 32 3d 38 43 6d 3d 41 41 3d 45 35 3d 41 34 3d 44 45 3d 39 34 53 3d 42 34 2e 3d 0d 0a 3d 41 36 3d 42 33 3d 41 43 3d 31 42 3d 38 30 3d 30 38 3d 33 46 46 3d 44 41 2c 49 3d 43 35 3d 42 38 09 28 3d 41 32 72 35 3d 30 38 40 3d 46 41 2a 3d 31 45 61 3d 44 45 5e 3d 39 45 3d 44 42 3d 31 41 3d 43 33 3d 38 42 26 3d 0d 0a 3d 43 44 3d 39 32 3d 44 30 3d 43 32 3d 31 32 3d 41 41 2a 3d 30 33 48 3d 46 38 45 3d 45 38 30 3d 41 35 3d 43 35 40 3d 44 45 3d 46 30 3d 31 37 3d 43 38 3d 44 34 3d 41 30 3d 39 46 3d 43 45 3d 46 34 20 76 3d 30 36 2c 3d 0d 0a 43 6e 3d 45 37 3d 39 30 3d 44 39 3d 43 31 64 3d 30 35 3d 42 39 3d 30 30 58 3d 44 32 48 3d 46 41 3d 41 46 3d 41 44 3d 38 38 3d 38 30 30 3d 31 38 3d 46 37 3d 39 30 3d 31 32 4d 3c 3e 76 3d 30 45 6b 3d 38 42 5b 3d 42 46 3d 0d 0a 3d 45 36 3d 31 30 3d 38 46 3d 44 31 3d 44 31 3d 45 41 3d 45 42 3d 39 44 44 3d 44 30 3d 44 31 3d 43 34 74 3d 38 43 2b 3d 41 38 5e 3d 43 42 31 3a 50 3d 42 37 3d 30 38 3d 42 43 09 3d 0d 0a 3d 41 32 3d 31 37 3d 41 37 3d 31 46 3d 46 31 77 24 53 3d 38 35 09 3d 30 33 4e 3d 44 31 7d 3d 45 30