Show HN: Pfuzz, a web fuzzer following the Unix philosophy (github.com)
However, I disliked that the existing fuzzers were monoliths where I had no easy way of creating custom behavior or analyses. They commonly do a multitude of things: Create multiple requests using one or more wordlist, sending the request, possibly with rate limiting, displaying progress, applying filters to the received responses and storing the output. If you want something different from the offered features, for example custom delays between requests or a new filter for the responses, your only option is to dig into a moderately large code base and try to adapt it to your needs.
I am a fan of the UNIX philosophy and felt like it could help out here. If there was a common format for communicating HTTP requests and responses, an ecosystem of small, specialized tools could use it to work together and fulfill tasks like fuzzing, while allowing the user to easily create custom behavior by combining the existing tools in different ways or adding small, quick to write tools to the ecosystem.
This is what I've attempted with the httpipe format [2]. It is a line based JSON format for exchanging HTTP requests and responses. I have also built some first tools using this format, namely pfuzz [3] for creating HTTP requests from wordlists, preq [4] for sending HTTP requests and receiving their responses and hpstat [5] for filtering the responses by their HTTP status codes. Since it's a line based format, many UNIX tools can be used with it as well and since each line is JSON, jq can also be used for manipulation, filtering and displaying.
[1] https://github.com/tomnomnom
[2] https://github.com/codesoap/httpipe
[3] https://github.com/codesoap/pfuzz
26 comments
[ 4.2 ms ] story [ 65.0 ms ] threadI think would call this a formatter, as it doesn't seem to do any mutations or executions itself. I would call AFL a fuzzer.
I would think at the minimum a fuzzer would more or less copy the mutation aspect, because to me that's what makes fuzzers uniquely useful.
Curious to hear your thoughts, or if mutations are a planned feature I missed
I see how reusing a term for a different concept is bothersome, but I feel like "fuzzer" is the term that people learning about bug bounty hunting are familiar with.
[1] https://github.com/ffuf/ffuf
[2] https://wfuzz.readthedocs.io/en/latest/
So things that look like reasonably complex HTTP requests but have deficiencies or small variations?
The last few API outages we had in my group were due to JSON payload edge-cases (either malformed or incorrectly structured) that weren't caught by what we thought were pretty extensive E2E tests and validation.
[1]: https://gitlab.com/akihe/radamsa
I like the idea - I think I have at least two formats for storing expected request / responses and probably more.
But standardising - as in not just my ball of twine tools uses but every uses is great.
I just think it already exists ?
1. PcapNG as well as the other file formats look like they are storing packets, which is a lower level than HTTP requests and unnecessarily verbose for my intended purposes.
2. They are binary formats, which makes them less suitable for printing to stdout. This also means, that they are not line based, which means UNIX tools, like grep, cannot be used effectively.
3. They are not designed for streaming. The httpipe format is line-based and contains no header/global fields. Thus it is trivial to, for example, build a filtering program: it would just read one line at a time and print it again, if it matches filter criteria; the output would automatically be valid httpipe again.
4. Lastly, parsing and composing JSON is something most developers have done before and basically every programming language has libraries for it. This makes it easy for the ecosystem to grow and enables users to build custom tools without too much initial effort.
[1] https://wiki.wireshark.org/FileFormatReference
[2] https://pcapng.com/
It also has the benefit of being single-lined, since newline characters are encoded, which is necessary for a line based format. This, in turn, allows the use of many UNIX tools, like grep.
However, it certainly is not the most compact format, when encoding large amounts of binary data. Gzipping for storage will alleviate the overhead somewhat. Overall, dealing with large amounts of binary data in web application testing seems like a less common use case, so I felt like I shouldn't put too much focus on it.
I like the idea! I'll see if I get a chance to play around with it at work next week.
It's a visual Unix pipes environment and has great support for all things Web. The advantage is constructing pipelines visually and not via the command line.
Node-RED is NodeJs based which unfortunately might make it harder to integrate Go code but it does support executing command lines tools using an exec node.
[0] https://nodered.org
However, I was also thinking about specifying an optional/informational "url" field or something similar, which also includes the path; "https://test.net/src" for your example. This field would be ignored by tools, but one could more easily copy things over into curl or a browser for further investigations.