Tell HN: By default, New Relic will start collecting production log data on 5/3
I was surprised (and annoyed) to receive the email below from New Relic, stating that on 5/3 they will start ingesting all production log data, by default.
To make matters worse, if you provisioned New Relic through Heroku, you can only opt out (by enabling High Security Mode) if you contact support. And if you're on the free plan, you can't open support tickets so have to ask on the community forum.
> New Relic APM agents will collect log data starting 5/3!
We’ve been hard at work making improvements to our APM and logging capabilities, and when you update certain APM agents starting May 3, 2022, logs will be automatically collected and sent to New Relic One. Logs are a critical telemetry type for observability and this new feature will help you troubleshoot your applications faster.
You probably have a few questions—including how to customize your logs ingest—so we’re including a FAQ below.
FAQ: Q: Why did you make this change? A: Logs are a critical telemetry data type but they are messy. This improvement allows users to send contextualized log data to New Relic without any additional setup. Relevant log data is now surfaced and correlated with other application telemetry automatically, reducing the need to switch context or run log queries when troubleshooting your applications.
Q: Which APM agents will have automatic logs collection and ingest upon upgrade? A: Starting May 3, 2022, when users upgrade to the latest version of the Java, Ruby, and .NET agent, log ingest will be enabled by default, unless High Security Mode is enabled or you have enabled the logs toggle for your accounts (more information on this below). We expect to enable application logs for Node.js, Python, and Go by July and PHP by September.
Q: I have already implemented logs in context. What should I do? A: We recommend only using manual OR automatic log forwarding. For more information, check out this documentation.
Q: I already use a third-party log forwarder, or forward logs via the New Relic infrastructure agent. What should I do? A: To avoid duplicating log data, consult this documentation.
Q: What does this mean for my New Relic bill? A: Collecting application logs means that more data will be ingested into the platform, at your standard ingest rate. The APM agent samples logs to ensure optimal agent performance. You can increase or decrease an application’s log volume as desired. Learn more here.
Q: I am concerned about sensitive log data being sent to New Relic. What should I do? A: No logs of any kind will be collected if High Security Mode is enabled on the agent, even after the agent is upgraded. If you do want to use New Relic Logs, it is also possible to configure drop filters to ensure sensitive data is not stored in New Relic. If you have not enabled High Security Mode, but still do not want to send logs to New Relic, see the next question.
Q: I do not want New Relic to collect or ingest logs, even after I upgrade my agents. What should I do? A: You can either configure the agent config file locally on a machine to disable it, or you can disable logs ingest for APM agents at the account level with a toggle in the New Relic data management hub. The toggle can be flipped before ever setting up an APM agent that forwards log data.
Q: Where can I learn more? A: Check out our documentation, read the Explorer’s Hub post, reach out to your account team, or contact New Relic Support.
104 comments
[ 3.9 ms ] story [ 106 ms ] thread> New Relic services are designed to receive and process telemetry data on the performance of applications, systems, and infrastructure, which typically do not contain any personal data. Customers generally send very little additional personal data to our platform.
https://newrelic.com/blog/how-to-relic/how-demise-of-privacy...
On the topic of privacy and GDPR, I find the previous link a good summary of the arguments used by American companies that cannot respect GDPR. Microsoft or Google say the same things in a lot more words.
Has any company ever been driven into bankruptcy because of GDPR violations? Google, Amazon, Vodafone, Yoigo, The Municipality of Bergen, Orange Espagne, Malagatrom, the Bulgarian National Revenue Agency, Enel Energia, Deutsche Wohnen, and Facebook have been fined over and over again, but it hasn't hurt them. If you can make more money abusing people's data than you'd pay in fines, and that's only assuming you're caught, it seems like possible GDPR issues are just another cost of doing business.
https://web.archive.org/web/20200813235643/http://slawsonand...
> Article 3(2), a new feature of the GDPR, creates extraterritorial jurisdiction over companies that have nothing but an internet presence in the EU and offer goods or services to EU residents[1]. While the GDPR requires these companies[2] to follow its data processing rules, it leaves the question of enforcement unanswered. Regulations that cannot be enforced do little to protect the personal data of EU citizens.
> This article discusses how U.S. law affects the enforcement of Article 3(2). In reality, enforcing the GDPR on U.S. companies may be almost impossible. First, the U.S. prohibits enforcing of foreign-country fines. Thus, the EU enforcement power of fines for noncompliance is negligible. Second, enforcing the GDPR through the designated representative can be easily circumvented. Finally, a private lawsuit brought by in the EU may be impossible to enforce under U.S. law.
[snip]
> Currently, there is a hole in the GDPR wall that protects European Union personal data. Even with extraterritorial jurisdiction over U.S. companies with only an internet presence in the EU, the GDPR gives little in the way of tools to enforce it. Fines from supervisory authorities would be stopped by the prohibition on enforcing foreign fines. The company can evade enforcement through a representative simply by not designating one. Finally, private actions may be stalled on issues of personal jurisdiction. If a U.S. company completely disregards the GDPR while targeting customers in the EU, it can use the personal data of EU citizens without much fear of the consequences. While the extraterritorial jurisdiction created by Article 3(2) may have seemed like a good way to solve the problem of foreign companies who do not have a physical presence in the EU, it turns out to be practically useless.
I'm not a lawyer, and I know nothing about New Relic specifically. But the worst offenders of the spirit of GDPR really are vulnerable to EU enforcement.
[0] https://www.theguardian.com/world/2021/jun/03/microsoft-iris...
I'm talking about from the perspective of a small US company with no assets outside the US.
Likelyhood of that happening? 1/1mil? Who knows. EU GDPR enforcement is only getting better, and the scenario you have asked about is sure to be in the spotlight sooner or later.
That's interesting: Sanctioning you by going after business partners. That could be messy, and could spark laws about that here were a European court to actually try it. How much power do we want foreign courts to have over our citizens?
These EU/EEA laws are for consumer protection. To operate in EU is way easier for digital companies than it is for other companies. But as an EU resident you expect uniform application of consumer rights, and you definitely don't want to be surprised along the way.
You have the option of not opening up your services to EU residents, and when the GDPR became effective I've seen websites that did just that. On the other hand GDPR and very similar regulation is going to become more prevalent. You have the CCPA, which is a distilled version of the GDPR, you have the Brazillian LGPD which is a derivation of the GDPR, and other countries are in the works of setting up their own.
In the end, I think that GDPR, thus far, is the most strict of them all, and if you implement universally what the GDPR requires, on a technical level (needs to be emphasized), you'll comply with any other privacy regulation.
The US has a tighter track record on political/economical pressure, even in EU. All the copyright acts and regulation that almost succeed all the time in EU, and happily thus far got rejected, didn't come from within EU, but from the EU-US trade partnership.
My opinion is that it's better to just check of compliance marks on a list, and move on with ones life. Politics are **, either way.
For instance:
https://en.wikipedia.org/wiki/David_Carruthers
Was a pretty high profile case like that.
You may of course disagree with that but I think that if you intend to do business in a country that you should abide by the laws of that country, even if there is no direct way to enforce them. Sooner or later you may find that nation states have pretty long reach.
For instance, let’s imagine a US business decides to make surprise changes to its T&CO and put penalties on some random behavior, but only applicable to EU users. It would proceed to take exorbitant amounts from EU cards, and refuse chargebacks pointing at the T&CO.
Would it be completely shielded from any fines and lawsuit from EU users ?
My gut instinct would be that even if the US company doesn’t have a EU presence, its service providers do, and a gov. could request any of them to recover legal fines on behalf of the country.
The reason may be logs can contribute to lot of ingestion volume compared APM, RUM, Infra products. They make lot of money in this case.
Just thinking out loud, I'm personally not familiar with this service.
Can you please elaborate on what these features are? I used to work in the agent team and we were cautious not to have that happen.
- some integrations have options to collect more timeseries. If the volume is too high, these will be marked in the documentation as billed as custom metrics
- custom integrations (including custom prometheus/openmetrics instances) are billed as custom metrics, as if they were sent through statsd
- container monitoring (enabled by default in the docker image, but not the other agent installations) is included up to 10 concurrent containers per host ; additional containers are billed on-demand
When installing for logging purposes this is not expected or desired. It wasn’t at the point that I encountered it, but I don’t believe the documentation notes this anywhere still.
Normally this wouldn’t be an issue and would be more cautious in production, but was using the Agent on some Dev machines to ship logs and was surprised to see Host metrics from them unintentionally along with the bill.
Automatic, opt-out bill increases come 5/3?
I wonder what they'd do if someone rigged their install to feed their servers junk information. Is that possible? I honestly don't know how the New Relic stuff works.
I'd like to think folks would be willing to pay to do it, but unless a large number of people do they probably wouldn't care.
Companies who are collecting data for profit don't really care how accurate any of it is. As long as it looks real enough they can still sell it to data brokers (oh sorry, I mean the "third-party service providers that support our business and operations"). If it doesn't look real enough it's probably easy to filter out.
All three of these sites have a large amount of "promoted" jobs that show up. I'd recommend filtering them out with a script or user style.
I wasn't looking for a start up role, but have heard good things about AngelList[2] for those that are.
[1] https://builtin.com/tech-hubs
[2] https://angel.co/
NEW_RELIC_APPLICATION_LOGGING_ENABLED=false NEW_RELIC_APPLICATION_LOGGING_FORWARDING_ENABLED=false
Amusingly, despite our account being closed, I received this same email this morning, so somebody there isn't exactly on top of keeping their mail list clean.
Query language is a bit of a PITA compared to some of the other options, but you get used to it.
Cortex for Prometheus, and tempo also all have s3 storage capabilities iirc
When they changed their billing to be per user our bill went up 20x for no additional value. Unfortunately I haven't found a better APM product that ticks all the boxes yet. I don't care about anything else but they don't understand that. And APM hasn't seen a single new feature in the 5 years I have used it. Stop taking my money to develop features I don't care about.
There are other options out there, but surprisingly few open source ones. I know APM is hard, but...
In a high-throughput distributed system, traces start being mostly noise due to random latency spikes which are usually not very interesting.
NewRelic is a lot more focused on the average but it breaks it down nicely so that you know where to start optimizing code. I rarely look at the p95 traces to do that.
New relic collects a lot of data but it really is a lot of noise. We dropped new relic because it kept modifying requests in one of apis that caused the result api to be corrupted and new relic ghosted us when unable to fix it.
A real APM cares more about profiling program execution than it does HTTP requests. I need to see function calls, not just HTTP requests.
(disclosure: I work for Sentry)
Edit: I looked at the docs, Sentry still has a sample rate parameter so this product can't give you the same results. With NewRelic I can have 100M transaction per day and then I have a single different transaction once every day and I can be sure that the agent will pick it up - always. With sampling you don't get that. Tracing gets you more depth but I seldom need that, I want APM and I only need tracing for a very very low percentage of transactions.
“Glad we could help! Just remember to continue making regular payments.”
edit: We had to resort to using APM on a portion of the farm but this allowed us to use it on all of it and finding issues was way faster. Incidents by level 1 were resolved faster as well and that was huge for global teams SLA.
APM is the product which got them off the ground, now they hardly care about it. Yet they have no problem raising prices beyond 10x for certain usage patterns. And its not really negotiable either. I understand that they want to make money, but we are paying hundreds of thousands more, not just a few thousands. And we get exactly the same product we had 5 years ago.
People are equally annoyed about the Datadog pricing model but at least there, I don't pay for what I don't use and I only pay for data ingested. NewRelic has the per user pricing model for one single reason: They want to upsell everything else on their platform because you are already paying for the users. Ingesting logs by default is just another way of costumer unfriendly upselling.
A fancy APM UI isn’t worth it to tolerate these dark business patterns which they try (or just do) all the time IMO.
We didn’t regret the decision.
Was really happy to leave it. But a few years later we tested the new Dynatrace and that shit is really shiny. Sadly, cost is expensive unless you are working in enterprice level, so cost is not an issue. But with the massive fuck up of newrelic/DD/Splunk with their skyrocket licenses.. is the cheaper option.
Fully-anonymized PHI can be analyzed on anything but the anonymization process must also be certified to have a reasonably low risk of re-identification when combined with other (arbitrary) datasets.
(source: I used to implement and certify PHI data collection platforms.)
No, it's not. (There are compliance reauirements, both direct and implicit on the HITECH definition of secured PHI, but “certified for PHI” is not one of them, despite marketing myths spread by the private HIPAA certification industry, which pedals certifications without legal meaning.)
> a good way to get sued for all of your dollars x10,
HIPAA penalties are capped at a penalty of $1.5 million per year for violations of any single provision, regardless of number of violations (and $50k per violation).
We already had logging disabled, but changing the default behavior in their favor is shady.
New Relic is already one of the most expensive items in our budget. Last year, I had to tell a producer "yeah, our SaaS that monitors our servers is more expensive than the servers themselves". That changed this year (started ramping up for scale), but it's still in the among the top of services we pay for.
They use Vitess but I think that is limited to information like sharding keys, account configs and other metadata.
We tried the log ingest but even for 1 VM, it was several GB per day data. 99% of the log data was irrelevant and trying to figure out how to filter it via New Relic was extremely difficult to figure out. We turned it off.
I guarantee that if they turn this on on 5/3, then 90% of their free accounts will shoot past the 100GB free tier. They hope people will convert to paid plans.
If they provided clear and easy to understand examples on how to filter the data before sending it into their environment, it would allow people to decide what data to send and then only have to pay for the data that is relevant instead of EVERYTHING.
Removal of development mode & support. The 'improved' price model. Their new UI is buggy as hell and they refuse to fix it. Every month orso you get a new interface that is objectively worse then what it is supposed to replace. Really helpful when shit is on fire and you need to figure out what is going on.
But. It is still very useful too me & finding an alternative is a time investment I haven't been able to make so far. So, for now, we'll continue to take it and deal with their bullshit as it comes. But as soon as I have some time to look for alternatives I'm done
All I do when I use NR to figure out issues or get insight is run some queries by some key name and it shows me a nice graph. I feel like we could have just collected this data into a system we build in a few days and run the analysis ourselves. As an end user I don’t feel like it’s doing anything that fancy besides the graph.