Ask HN: Do you test in production?

I think it depends on how your application works. If you have the concept of customers, then you can have a test customer in production with test data that doesn't affect real customers for example. You can reset the test customer data each time you want to test.

Anytime you need to talk to a third party API, you need to test in prod.

Some people have sandbox apis. They are generally broken and not worth it. See eBay for super in depth sandbox API that never works.

You can read the docs 100 times over. At the end of the day, the API is going to work like it works. So you kind of “have to” test in prod for these guys.

Yes

Just because you have staging doesn't mean you don't need unit tests. Similarly, test in stage, then test in prod. Ideally in a way isolated from real prod users (eg, in an insurance system we had fake dealer accounts for testing)

We run canaries in Prod, it isn’t as extensive as our integration tests that run in our test stages but it still tests happy paths for most of our APIs.

Feature flags.

In electronic trading, most new systems are tested in production by running with smaller capital allocation first. It is hard to flatten out all bugs unless you are on the real market with real money and real effects (of course, simulations testing and unit testing are heavily employed too).

It's more about handling production error quickly, than testing in production. Feature flag is a good way.

Lots of ways to test in production. IMO the way you are suggesting – injecting synthetic data into prod – is the worst of both worlds. You aren't actually testing real world use cases, and end up polluting your prod environment.

Some common ways to go about it:

- Feature flags: every new change goes into your codebase behind a flag. You can flip the flag for a limited set of users and do a broader rollout when ready.

- Staged rollouts: have staging/canary etc. environments and roll out new deployments to them first. Observe metrics and alerts to check if something is wrong.

- Beta releases: have a group of internal/external power users test your features before they go out to the world.

I work for a B2E company that has a structure similar to Salesforce. We test in production all the time even for our secure environments where the data is highly sensitive.

Re: data, it’s a somewhat common practice to notionalize data (think isomorphically faking data). We regularly do this and will often designate rows as notional to hide them from users who aren’t admins. I’ve found this to work exceptionally well; we do this 1-2 times a week, ensure there’s a closed circuit for notional data, and for more critical systems we’ll inform our customers that testing will occur.

I’m sure there are more complex and automated solutions but when it comes to testing, simple and flexible is often the way to go.

Note: if you plan on accurate financial planning and metrics (esp. if going public), you need to be able to separate your test prod stats from the real prod stats for reporting.

A/B tests and feature flags are basically testing in prod. And yes, some of those features sometimes run as a "well, it should work, but we're not entirely sure until we get a significant number of users using the system". It could be an edge case failing or scalability requirements being wrong.

Another variation on the same theme is rewriting systems when you run production data through both systems. Quite often that's the only way of doin migrations to a new platform, or a new database, or yes, a newly re-written system.

> Is it common practice to inject fabricated data into a prod system to run such tests? What's the best practice or prior art on doing this well?

A very common practice is to run a snapshot of prod data (e.g. last hour, or last 24 hours, or even a week/month/year) through a system in staging (or cooking, or pre-cooking, or whatever name you give the system that's just about to be released). However, doing it properly may not be easy, and depends on the systems involved.

Sometimes one cannot get the exact same specs on test hardware versus production, yet a rollout depends on simulating system load to shake out issues.

Performance testing needs a schedule, visibility, timebox, known scope, backout plan, data revert plan, pre- and post-graphs.

  Schedule. Folks are clearly tagged in a
            table with times down the side.

  Visibility. Folks who should know know
              when it's going to happen,
              are invited to the session, 
              and are mentioned in the 
              distributed schedule.

  Timebox.    It's going to start at a
              defined time and end on a
              defined time.

  Known scope. Is it going to fulfill an
               order? How many accounts
               created?

  Backout plan. DBA and DevOps on standby
                for stopping the test.

  Data revert plan. We know what rows to
                    delete or update after
                    testing.

  Pretty pictures.  You want to show graphs
                    during the test, so
                    that you know what to
                    improve and everyone's
                    time wasn't wasted.

Reference: observing successful runs that didn't result in problems later.

yes you can do so with a canary tier . assuming your code is well instrumented to distinguish performance and quality regressions , a canary tier served to customers will catch more regressions than synthetic testing

Depends a lot on your application and how big the changes are. IF you're an online store and you're pushing out incremental changes to a subset of users its a good strategy. If its aircraft auto-pilot not so much.

Everyone tests in production. Some people also test before production!

Some people try to NOT test in production, but everyone does test in prod in a very real sense because dependencies and environments are different in prod.

I think the question was "Do you INTENTIONALLY test in production"

I see a lot of suggestions in the comments for feature flags -- we've been using these from the beginning, to very good effect.

However flags turn on/off code, not data, and my main area of interest here is how to deal with the test data problem in prod.

In a multi tennant system one of the accounts can be a test account. Within that you can run integration tests. You might need special cases: test payment accounts and credit cards, test pricing plans and so on.

Some basic ping tests and other checks before swapping (as in preparing, initiating, and pointing the load balancer) to a new version into production would be smart.

Good testing is an exercise in pushing I/O to the fringes, as that's what has stateful side-effects. (Some might even argue that anything that tests I/O is an integration test. The term "integration test" is not well defined and not worth getting hung up over IME.)

Once you're into testing I/O, which is ultimately unavoidable no matter how hard you try not to, you either need cooperative third parties who can give you truly representative test systems (rare) or a certain amount of test-in-prod.

Testing database stuff remains hard. You either wrap things in a some kind of layer you can mock out, or dupe prod or some subset of it into a staging environment with a daily snapshot or similar and hope any differences (scale, normally) aren't too bad.

Copy-on-write systems or those with time-travel and/or immutability help immensely with test-in-prod, especially if you can effectively branch your data. If it's your own systems you are testing against, things like lakefs.io look pretty useful in this regard.

And yes, feature flags, good metrics, and load balancers that you can send a small percentage of traffic through a new version (if your traffic/system allows such things) all help.

With certain kinds of reporting/BI tools, I've generally found it's not that risky to test in production, provided certain conditions apply, and it comes with a number of advantages where the QA environments don't truly mimic what happens in production (or the time for updates in QA is way too slow, so you don't see varied output cases appearing fast enough to give a good test).

A common dev concern (usually raised by people who have no idea how users actually use stuff!) is that someone might pick up the report and then do something awful based on it, which would be awful^2 - I then explain that users can't find/use the updated reports till we tell them where they are and grant access permissions etc etc, so it's going to be fine and there's no need to panic, which calms them down till they forget by the time this comes up again!

On a side note, these people seem to get much more wound up about principal based worries (it would be bad to test in Prod being a prime example) compared to concerns based on their own weaknesses (ie they rush, forget a whole section of requirements, make mistakes, can't spot obvious bugs) which they seem to imagine are way less likely to cause problems than experience demonstrates.

As others have said, injecting fabricated data into prod wont give you any value. The only reason to test in prod these theys are to try your new feature on data with the breadth and level of detail that prod data has that you can never fabricate. (Hardware differences between prod and other envs really should not be a problem these days).

In almost every case you cannot test new functionality on actual prod data, at least not anything thats not strictly “read only”-functionality. If you have a new feature to send automated mail to someone foreclosing on their property you just do not test that on a real live system.

What you can do is setup a staging environment that is as close to prod config as possible, and then copy the prod database to the staging env. Do your tests in staging. It doesnt matter if data in stg is messed ud. There may well be legal or company policy or security restrictions preventing you from doing this, but its the only way to test on real life data without the risk of f**ing up data in the live system.

Then there are integration tests - to other systems that is, which are a much harder problem

Is there another way?

The taboo is when you only test in production. At the very least you should manually try out your app after deploying a change. As far as automated integration tests in production, it is simple as identifying which tests are prod safe, and marking them. That really depends on the app, but in a web app it generally means all the GET requests, plus some of the others.

Hi! I wrote the referenced Segment post! Happy to answer any questions.

The way we did it safely is just as you say: creating fabricated users/organizations/configurations with data generators injecting into the system.

Faking data to look realistic is always challenging, but we used this cool library written by an early segment engineer: https://github.com/yields/phony

Not perfect but works well enough! And it's super simple. :-)

Anyone with CI/CD is testing each deployment in production, right?

What I've done in the past is to write a test that runs every five minutes in production, accessing the APIs like a user for the most common app flows. It provided a great way to be sure the app was genuinely working.

That did require having multi-tenancy support, and there was a need to suppress some security features by whitelisting the IP of the test app.