Launch HN: Codeparrot (YC W23) – Automated API testing using production traffic
Here’s a short video that shows how it works: https://www.loom.com/share/dd6c12e23ceb43f587814a2fbc165c1f .
As managers of engineering teams (I was CTO of an ed-tech startup, Vedant was the founding engineer of a unicorn company) both of us faced challenges in enforcing high test coverage. We ended up relying a lot on manual testing but it became hard to scale, and led to reduced velocity and higher production bugs. This motivated us to build CodeParrot.
How it works: we auto-instrument backend services to capture production traffic. Requests and responses coming to your backend service, as well as the downstream calls made by it like DB calls are stored. As part of your CI pipeline, we replay the captured requests whenever your service is updated. The responses are compared with the responses from production env and regressions are highlighted to the developers. To ensure that the same codebase gives the same response in CI environment and production, we mock all downstream calls with the values from production.
Most tools to record and replay production traffic for the purpose of testing capture traffic on the network layer (as sidecar or through load balancer), CodeParrot instead relies on an instrumentation agent (built on top of OpenTelemetry) to capture traffic, enabling us to capture downstream request/response like database responses which are otherwise encrypted on network layer. This helps us mock downstream calls and compare the response from CI environment vs production environment. Additionally, this helps us sample requests based on code flow and downstream responses which provide better test coverage compared to just relying on API headers & parameters.
Our self-serve product will be out in a few weeks. Meanwhile, we can help you integrate CodeParrot, please reach out at royal@codeparrot.ai or you can choose a slot here - https://tidycal.com/royal1/schedule-demo. We’ll be selling CodeParrot via a subscription model but the details are TBD. In addition, we will be open sourcing the project soon.
If you’ve already tried or are thinking of using tools in this space, we’d love to hear your experience and what you care about most. We look forward to everyone’s comments!
65 comments
[ 3.6 ms ] story [ 170 ms ] thread> How does CodeParrot work
This part without a question mark is making me feel like I’m left hanging. If you don’t want a question mark, I would much prefer this to be “How CodeParrot works”.
> CodeParrot … What’s more? …
There’s no question mark where there should be one, and where I don’t expect one I do see one. Here it can be a simple comma instead of a question mark. This flow of language works if you’re speaking live to an audience but on paper it feels awkward.
It bothers me more than it should, mostly because of my reading cadence. Hopefully others can chime in and let me know if I’m off base here or not.
To reduce false positives - we run the same request twice to eliminate flaky fields in response like current timestamp, mock the downstream dependencies as they behaved in prod env and are providing options to ignore / modify the sampled requests
To make integration easier - we are building on top of opentelemetry which has seen remarkable increase in support across languages / frameworks, which makes it easier for us to support different tech stacks.
Regression testing - our primary goal is to provide regression tests. We have come across two type of teams where this makes sense - companies with low test coverage and companies which high number of micro-services as they find it hard to cover every production scenario in tests
- Downstream calls (think DB, 3rd party API calls, kafka) are not handled.
Did you manage to play it back manually and/or via tests? I ran into unexpected challenges while doing this.
[DataTestMethod] [DynamicData(nameof(GetAllHttpTests), DynamicDataSourceType.Method, DynamicDataDisplayName = nameof(GetTestDisplayName))] public async Task RunHttpTests(string endpoint, string method, string data){...}
To be clear, this is integration testing, not unit. Testing that certain HTTP requests work, not necessarily that they are correct.
I don't know that there's necessarily a wrong answer here (well, there probably are, but wrong only in the sense that a given solution might be prohibited by the regulation), just want to see how y'all have thought through the prompt.
HAR can already be recorded in middleware (e.g. loadmill/har-recorder) and replayed in multiple CI compatible ways.
> What do you mean by "live, regulated data into non-prod environments" exactly? Could you provide some examples?
Credit card numbers, card verification codes, protected health data/electronic medical records... list goes on.
Every environment that has live data in it functionally increases the valuable attack surface for most adversaries. I.e why bother attacking production when they can slurp up production data from test environments that are less likely to be well protected?
As for the "AI," I think the op was just commenting on the TLD used by the startup.
We also provide configuration option to specify additional fields are needed to be anonymised
1. what is the type of service instrumentation needed to capture the data? Wonder why this is needed when typically the data is already captured in an APM log? The instrumentation might add performance and security concerns.
2. what is the sampling logic to capture the traffic? It might compromise the fidelity of the test data and give a false sense of test accuracy.
3. what is the duration of data capture? Is it a week's or month's or quarterly data? Meeting 90% coverage on a week's production sample data will provide a false metric.
4. can it faithfully handle data privacy and customer anonymization? This is critical for API's dealing with PCI and other sensitive data.
Implementation is very similar to an APM log. So the same performance and security concerns apply. We are working on giving both at the same time (Automated tests, and APM), to reduce overhead.
> 2. what is the sampling logic to capture the traffic? It might compromise the fidelity of the test data and give a false sense of test accuracy.
It is random sampling. I feel, 1M or 10M randomly sampled requests should cover all cases.
> 3. what is the duration of data capture? Is it a week's or month's or quarterly data? Meeting 90% coverage on a week's production sample data will provide a false metric.
I was thinking 1 week should be enough. Maybe we will have to add some custom sampling logic for lesser frequency calls (like monthly crons).
> 4. can it faithfully handle data privacy and customer anonymization? This is critical for API's dealing with PCI and other sensitive data.
Yes. Additionally, for compliance, we offer a self-hosted solution- Our code runs on your servers and no data ever leaves your cloud / on-prem.
> It is random sampling. I feel, 1M or 10M randomly sampled requests should cover all cases.
1. I suggest providing alternate approaches to sampling: The input itself may have bias towards a single use case. If 70% of the input exercises the same code path, there's no benefit to having a uniform sample. Ideally it would be stratified amongst customers, or perhaps on other dimensions to allow for covering the most surface area.
2. Requests don't happen in a vacuum. They likely have data dependencies on prior requests. I recommend some way of sampling sessions rather than individual requests. Replaying the 3rd request in a series of 6 is likely just going to be exercising failure paths.
3. Behaviors may vary between requests with respect to time. If requests were sampled over a number of days but replayed within a short time period, there are behaviors that could differ from what actually occurs in production.
I didn't see any explanation on how results are determined. I think it's important to surface those types of details on the website. I'm not going to watch the video on it in hopes of learning.
This is so true. I ran into this problem when I was trying to implement a kind of sampling that limited total RPS. Thanks for reminding. I am definitely looking into more sophisticated methods of sampling.
> I didn't see any explanation on how results are determined. I think it's important to surface those types of details on the website. I'm not going to watch the video on it in hopes of learning.
Good feedback! :)
It's a neat idea. These kinds of systems often require lots of care and grooming. Since it's used to retroactively test features after they're in production, there's a repeating process of discovering we're saving data we shouldn't, scrubbing, filtering, anonymization, etc. In most cases, I've watched them eventually get replaced by fuzzers. Still, having a central service used by lots of companies may allow this solution to scale up, develop necessary features to solve these problems and function well. I hope it works out!
[1] https://speedscale.com/
Quick question, how do you (plan to) deal with schema/API change? Or the tool is more intended for regression testing?
Its simply a matter of using a baseline of: `production` (usually for regression tests), `staging` or `local`.
For example, suppose a startup has an Express app that made downstream requests to Postgres, Redis, Kafka, other REST APIs, etc. How do the outgoing requests, which may follow different protocols, with varying serialization formats and handshakes, all get intercepted, recorded, and matched to the same outgoing calls when the session is replayed? How are the outgoing requests indexed and then evaluated for sameness when replayed?
It definitely seems possible to implement something like this as a series of high-level one-off integrations, for example, a middleware package for Django apps. But if the goal is a universal and automatic downstream service mocking system, it's not clear to me where this sort of middleware would even sit in the application stack. It seems like it would need to be pretty low-level, but not too low-level because the data needs to be decrypted first.
Anyway, if you guys have figured out a good way to do this, I'm definitely interested in hearing how you managed it. I'm in no way a great BE engineer, so there could be something elegant or obvious that I missed.
And besides network calls network-bound requests, what about reading from sqlite, files, and interprocess communication.
What about ZeroMQ over UDP? A remote time service. Pulling from a git repo.
No really, I'm asking because I'm trying to figure out how to deal with all this in my tests right now, and I honestly have no idea. But I doubt Codeparrot is going to be able to intercept and intelligently mock everything.
In theory, *everything* can be recorded and replayed; that's the magic of patching-in at the application layer. *Any* function call can be recorded, and, replayed.
In practice, we support whatever is demanded by our customers. If people are ready to pay for something, we are happy to build support for it.
I definitely agree with your approach of auto mocking the database and/or third party services too. That's what I did with my home rolled solution.
Happy to see more projects in the space. Generating tests by snapshot including the DB and service calls always seemed like the obvious way to go for me.
As a dev I'm not keen on using services which I could easily replicate so this would have to be both substantial and cheap/free for smaller teams.
- Support for high number of languages, downstream dependencies - Intelligent sampling to choose requests with high coverage and auto update them over time - Performance, safety and data compliance guarantees
If the latter, I'm wondering how it might compare to tooling with a similar intent, e.g. https://www.microsoft.com/en-us/research/publication/restler...?
In my experience, fuzzy testing is more helpful from Dast / security testing perspective and we were thinking of adding these later.
Especially for rewrites or code refactorings.
I've built this system at many companies myself. Never thought of doing this as a service for others.
New sessions - What is the use case for it? Its definitely possible.
> I've built this system at many companies myself.
Interesting! Would I know any of those companies/products?
Could you elaborate on how this is achieved? Eg, say I have a lambda endpoint and the code in it is querying one or more databases. Are you somehow automatically hooking into those function calls, recording their return values, and then mocking the functions in the replays? Or are you doing something else entirely?
1. Statefulness: simple example, api call that returns a query from the DB that has a high transaction load
2. Non determinism: for example returns a random number, guid or a time
3. Privacy requirements for certifications or legislation that mean private data cannot be used in a test environment.
4. GDPR laws
5. Authentication
I presume this is more microservice friendly and monolith unfriendly. But that is probably a reason on the pro side of smaller, bounded services.
We natively handle API and DB calls.
> 2. Non determinism: for example returns a random number, guid or a time
This is a bit tricky. We run the same test multiple times, and ignore the changing fields.
> 3. Privacy requirements for certifications or legislation that mean private data cannot be used in a test environment. > > 4. GDPR laws
We anonymise *any* private data before storing it for replay.
> 5. Authentication
We capture auth headers, and mock auth server responses.
> I presume this is more microservice friendly and monolith unfriendly. But that is probably a reason on the pro side of smaller, bounded services.
Currently yes. APIs can be easily tested. Tests for function calls will require user configuration.
I'm maintainer of GoReplay https://github.com/buger/goreplay and work in this area for the last 10 years.
It is quite hard problem to solve, because you have to deal with state difference between test and production environments. Love your approach to mocking dependencies, and leveraging OpenTelementry. It potentially can solve some of state issues. But still require modifying user code. I wonder if it can be done purely using OpenTelementry (e.g. you depend on typical OTel setup), and then read the data directly from OTel DB.
Cheers!
If anyone in this thread wants to build a product based on GoReplay technology (capture network traffic directly, via AWS Traffic Mirroring or k8s), sent me message :)
Typical Otel implementation don’t capture some request data esp parameters and replay part is missing among few other issues, so we need to extend it.
OTel for go requires user code changes. Languages that allow monkey-patching (java, js, python, etc.).
> I wonder if it can be done purely using OpenTelementry (e.g. you depend on typical OTel setup), and then read the data directly from OTel DB.
OTel doesn't work out of the box. OTel usually doesn't collect request or response for any network or db call. 90% of my time is spent on extending the individual agents' code; so that they can collect additional required information, and perform "replay".
BTW, good folks at https://www.hypertest.co are doing something on same lines here in India.
Cheers!
We have come across Hypertest, seems pretty cool and useful.