Show HN: Nosey Parker, a fast and low-noise secrets detector for textual data (github.com)
The big idea: textual content in, hardcoded secrets out. These include things like API keys and passwords.
It should do a reasonable job on any textual input. It will recursively scan any directories it is pointed at. It also has special support for scanning the complete contents of Git repositories.
The default rules in Nosey Parker have been carefully chosen to minimize false positives. Many API tokens these days have well-specified formats that are amenable to precise matching with regular expressions, and these are the kind of things that the default rules detect. Additionally, its findings are deduplicated. Together, these give much higher signal-to-noise compared to similar tools.
Nosey Parker is fast: it can scan 100GB of Linux kernel commit history in just over a minute on my laptop. This speed comes from several factors, but most significantly from using the amazing Hyperscan library for simultaneous matching of all regex rules in a single pass. In comparison with similar tools on large inputs, Nosey Parker is usually 1-3 orders of magnitude faster.
Nosey Parker was originally created to help construct a labeled dataset of secrets for machine learning purposes. But it proved surprisingly useful on its own. In the past year, an internal, proprietary version (with added machine learning capabilities) has been regularly used in security engagements at Praetorian.
In late 2022, Nosey Parker was reimplemented in Rust, released as open-source, and presented at Black Hat Arsenal.
It now supports enumeration and scanning of GitHub repositories by providing just usernames or organization names. It also recently got support for SARIF output, which several other tools understand.
6 comments
[ 0.23 ms ] story [ 26.2 ms ] threadThe development so far has focused on security-related uses, especially finding hardcoded credentials in source code and log files. For the serial number and license spelunking you describe, Nosey Parker would need a couple more pieces to work well:
(1) Regex rules that would match serial numbers. I think it would be tough to write a regex for this that would match precisely and with good recall, i.e., producing few false positives and few false negatives. Not a showstopper, but not ideal.
(2) Built-in unarchiving support. Nosey Parker will handle textual files on disk just fine, but won't do useful things for many binary file types such as zip files. Your old external drives probably have tons of non-textual files like that. I would like to eventually add built-in unarchiving support to Nosey Parker; this would allow a way for textual content to be extracted from binary files and then scanned like the rest.
On the one hand, Nosey Parker is effectively a special-purpose `grep` with a bunch of security-relevant patterns built-in, including one for PEM-encoded keys: <https://github.com/praetorian-inc/noseyparker/blob/main/data...>
On the other hand, to naively run the check you describe, you would need access to a copy of all of GitHub, which isn't feasible.
What you can do with Nosey Parker is use its GitHub enumeration features to specify your GitHub organization and a list of GitHub usernames you are interested in, and scan against just those. This will implicitly list all the relevant public repositories, clone them, and scan their entire history.
For your use case, another thing you could do is use the new GitHub code search (<https://cs.github.com>) to regex search for particular keys or tokens. That new search seems to cover lots of the public content available on GitHub.
Also, to put some color on this use case: in offensive security engagements (aka "red team" engagements) at Praetorian, we frequently find leaked credentials or tokens on GitHub or elsewhere, which allow us deeper access into the client's systems. It's a significant problem.