If Truly you Are In Need Of A PROFESSIONAL LEGIT HACKER Who Will Get Your Job Done Efficiently With Swift Response, Congratulations, You Have Met the Right HACKERS.
WHO ARE COMPOSITE HACKS???
• We are a Team Of Professional HACKERS , a product of the coming together of Legit Hackers from the Dark-Web, (pentaguard,CyberBerkut, RedHack , Black Hat, White Hack ) we have been existing for over 12years, our system is a very Strong and decentralized command structure that operates on ideas and directives.
JOB GUARANTEE:
• Frankly speaking, I always give a 100% guarantee on any job we are been asked to do, because we have always been successful in Almost all our jobs for over 12years and our clients can testify to that . To hack any thing needs time though, but we can provide a swift response to your job depending on how fast and urgent you need it. Time also depends on what exactly you want to hack and how serious you are. Enough time with social engineering is required for hacking. So if you want to bind us in a short time, then just don't contact us because We can't hack within 30minutes,sorry. Basically, time depends on your luck. If its good luck, then it is possible to hack within 30minutes but, if it is in the other way round, it would take few hours. I have seen FAKE HACKERS claiming they can hack in 30min, 20min , but there is no REAL HACKER who can say this (AVOID THEM).
Please Note : we have only one contact email : compositehacks@gmail.com
We will be happy to have you join over 2000 satisfied clients around the world to use our services.
OUR HACKING CAPABILITIES:
There are so many Reasons why people need to hire a hacker, It might be to Hack a Websites to deface , retrieve information, edit information or give you admin access Some people might need us for Hacking any smart phone giving you access to all activities on the phone like , text messages , call logs , Social media Apps and other informations. Some might need to Hack a Facebook , gmail, yahoomail, Instagram , twitter and every other social network Accounts, Some might need to Hack into Court's Database to Clear criminal records. However we can also Hack into school's websites (server) to change grades without any trace, Also Some Individuals might want to Track someone else's Location probably for investigation cases.
All these Are what we can get Done withing few hours.
SOME OTHER SPECIAL SERVICES WE OFFER:
Hacking and sales of Programmed credit cards & CREDIT CARD TOP-UP
Sales of untraceable phones (even the pentagon can not track our phones)
Sales of Tutorial packs for Beginner Ethical Hackers.
You can also contact us for other Cyber Attacks And Hijackings, we do almost All.
Contact Us for Your Desired Service: compositehacks@gmail.com
I had the opposite reaction. If you're in a big city in the US, $240K is the cost including overhead of one, maybe two engineers. If running this in-house takes up a total of a full-time engineering load (i.e., if the team maintaining this wants to add headcount to run this and maintain prior commitments), you haven't saved very much and you might be net negative.
(The math may be different in different job markets, in which case, yes, it makes sense not to pay a startup that's paying SF wages to its own employees.)
Yeah. It's definitely not a clear-cut net benefit. To be fair though, they are utilizing hosted services (so they don't need to hire kubernetes experts) and the company likely already has a team of engineers maintaining N services. Maybe maintaining N+1 services is within their engineering capacity.
It's not uncommon at all in the expensive markets (I'm thinking SFBA and NYC) for an engineer who isn't straight out of college. But I'm counting overhead - recruiting, benefits, employer-paid taxes, space, equipment, amortized cost of just becoming a bigger company (helpdesk, possibly management, etc.). Depending on how you count, that could add anywhere from 25% to over 100% to the employee's take-home compensation. http://web.mit.edu/e-club/hadzima/how-much-does-an-employee-...
$192K ($240k/1.25) is definitely common; $120K ($240k/2) is low in these markets for someone who knows what they're doing.
Five weeks of, say, three engineers with $150k salaries ($200k fully loaded) is $60k. People say that the majority of software is after initial development. Let's say 20% is development [0]. That means this project costs $300k total and lives for 5 years before the team decides to rewrite it. That brings us to $60k / year.
This is good savings! My general guidance to folks thinking about this sort of project is that you should implement a vendor and see how you like it. The reason being, MixPanel does a LOT more than this tool, and buying a vendor can help you figure out the subset of features that you actually need. Once you have a clear requirement list, you have a clearly scoped project and can hammer it out relatively easily to get that cost savings.
Beware of thinking that you can do this for $X,000 / year on your own, or get it right on the first shot. There's a reason MixPanel (and any other vendor) has so many engineers. You're paying them to have already made the mistakes that you're about to make.
The whole thing is a big ball of mud. If you commit to a vendor you risk vendor-lock in. If you commit to hand-roll you risk runaway deadlines, feature creep, and a slew of other threats. Once you pick one or the other and get it to a business functional level you lose all momentum and willpower to pivot to the best solution now that you have a "good enough" approach.
Software is always 'Pay me now, or pay me later'. In the end I believe you usually always end up paying pretty close to the same, it just depends on if you are happy with the solution in 18-36 months or hate it.
The tricky bit is when opportunity costs come in... "Pay me now" means sacrificing clear opportunities in the visible horizon, "pay me later" does the same thing at an less-controlled future point sacrificing presently unknown opportunities.
Personally I much prefer 'pay me now', because 9 times out of 10, IME, some form of technical debt underlies the reasoning about why a team can't do something. 'It always pays to take the pain up front', basically.
It's good to be conscious of vendor-lock-in as you implement a tool or library. If their product is good, at the worst you can borrow their API design and slowly reimplement portions of the back-end provided with your own implementations to liberate yourself. For a product/business that makes money, most often using a stable external tool is a better first-step.
If things flip around again and you find a new library that solves your needs better than your homegrown lib, you can drop it in forking your lib and turning it into a wrapper on the external lib.
At least this is what I have learned implementing high flexibility cross-platform mobile video playback solutions in our app over the last 5 years, where the landscape of libraries (and our own tools) has kept changing.
Sometimes "good enough" is, well, good enough. How accurate good is good enough? 80%? 95%? It's different for every business, and the needs change over time. 60% accurate for an eagerly stage bootstrapped company? Gold! 60% accuracy on a growth-stage business where you should know what to do? Terrible.
Be limber enough to eschew the tools that aren't working. But "good enough" is, well, good enough for most folks. Perfection the enemy of progress and all that.
> There's a reason MixPanel (and any other vendor) has so many engineers. You're paying them to have already made the mistakes that you're about to make.
The real reason Mixpanel (et al) have so many engineers is:
A) They need to scale their service to reliably provide to N corporations which is frequently non-trivial.
B) They need to implement the 80% of features that aren't critical path for Company A because some subset of them are for Company B,C,D,E,F,G,H,I,J,K,etc.
C) I've seen software vendors make some pretty terrible and painfully obvious mistakes.
Idk, maybe I'm crazy but I have very little faith in this idea that more engineers = better decisions were made. I've just never found quantity has an impact on quality of the end product, only the speed at which its produced.
The number of engineers they have is irrelevant to the point. The point is you pay someone to worry about a specific thing so you don't have to and (hopefully) their expertise leads to a better product than you could have built.
> The number of engineers they have is irrelevant to the point. The point is you pay someone to worry about a specific thing so you don't have to and (hopefully) their expertise leads to a better product than you could have built.
Yeah, and in my experience, that frequently fails to materialize which why I mentioned point C.
Now, that could be arrogance on my part but it doesn't change my experience on the quality of the vast majority of outside software vendors.
We went for a vendor to replace one of our internal tools. It looked too costly to pay team of 4 engineers a year to support it.
Unfortunately, the part "implement a vendor and see how you like it" went only with the first half. Complaints of developers supposed to be using the tool were brushed aside as a simple resistance of change. In the end it was implemented by decree from higher management.
Only after all the paperwork was signed, we started to see that the vendor doesnt support our usecases, that their hardware is underpowered for the load we were expecting, e.t.c ...
In the end, I have heard at least 20 various senior and beyond senior engineers from various teams complaining that they spend 20 to 50 % of time working around the vendors bugs, around once a month bringing down vendors infrastructure, because our use-case mandated uploading more data than the infra could handle.
I definitely agree that clear requirement list and clearly scoped project and implementing a vendor to see how you like it is usually better than in-house
On the other hand in-house is usually better than implementing a random vendor because management feels like it ;)
Built a similar thing last year -- nginx (openresty) + lua -> NSQ -> python streaming -> BQ
Using Metabase for visualization.
The other key benefit is you can warehouse the rest of your data in BQ as well -- where it can be easily linked to your non-event data; marketing data from other systems, etc.
This article epitomizes the old adage: software engineers are terrible at estimating. I have absolutely no stake in Mixpanel, and it has its own flaws, but seeing something as misleading as this on the front page of HN means I have to write a clarifying, if not somewhat edifying, comment =/
1. Right off the bat, there's the cost of building all of this + maintaining it. Something like Mixpanel at scale requires at least two, if not three, engineers: one for client libraries, one for infrastructure (which the OP blogs about), and another for the web app (dashboard, real-time user stream, etc.) To be sure, not everyone needs all features of Mixpanel. To be really sure, nobody needs all features of any SaaS tools. But if you really want to compare apples to apples, then you have to account for these. To hire competent software engineers who can collaborate closely and maintain a complex piece of analytics software requires at least $100k per year, if not twice that based on locale. That's at least $300k and more like $600k right there.
2. The whole point of something like Mixpanel is to make analytics accessible to non-engineers. In their case, it's primarily product people and secondarily marketing/customer success. In any case, building an analytics/data product consumable by non-technical people is hard and takes way more than assembling a couple of cloud infrastructure together. If there's one reason Mixpanel is still in business, that's because of this.
3. Finally, the OP has a valid point which should have been highlighted more, if not to make their own biases more clear: Mixpanel's diminishing differentiation is dev shops/consultancies' opportunity. It is indeed incredible that a dev shop can build even a third of Mixpanel's functionality by leveraging GCP components. Mixpanel had to build a lot of its core backend systems from the ground up, including its original key-value store. Just this year, they fully migrated to Google Cloud Platform themselves, suggesting that there's really little room for differentiation among analytics vendors at the level of infrastructure components (Mixpanel's arch-nemesis, Amplitude, leverages Apache Kafka and various AWS components, most notably Amazon Redshift, heavily)
With all of this being said, one thing remains true: the most expensive cost of any software is people running them and the dependencies created around them. These may not show up as line items, but they sure are deeply embedded in your total cost.
Having built something like this myself the components you actually need to write are so small and simple the scope for security issues is minimal. All the infra scales automatically.
Also, this post only addresses data flowing in one direction into BQ. Presumably they're then using something like Tableau or the Google/AWS offerings to build dashboards based on this data. There aren't many places for bugs to hide unless the BQ perms are wide open.
I agree with all three of your points concerning Mixpanel and what it would take to replace it with something similar. But I think you're being harsh when you say "This article epitomizes the old adage: software engineers are terrible at estimating".
I don't see anywhere that the article says they have replaced all of Mixpanel. It only says that they were able to replicate the functionality their client actually needed. I don't see them making claims that the solution they built is anywhere comparable to the full solution from Mixpanel. Only that it provides all the features their client needed.
Having looked into doing a very similar migration, I generally agree with you. There are performance, control, and speed gains on the back end to be sure. It's replicating the flexible, intuitive UI where the cost argument always breaks down.
That last part is the most important. It's why you're really using Mixpanel/Amplitude/other vendors. To say "we replaced Mixpanel" without addressing the front end feels a bit disingenuous to me.
GCE is Google Compute Engine for running VMs. GKE was Google Container Engine but makes more sense now as Google Kubernetes Engine. GKE runs on top of GCE VMs.
I dunno. The way I look at it, whatever you could have been working on to advance your company when Mixpanel took care of things, you weren't when you were building this and can't when you have to maintain it.
This did not mention the front end at all. In my opinion getting a front end anywhere near as slick and powerful as Mixpanel's is an even bigger challenge than the data pipeline. The Google Reference Architecture diagram shows spreadsheets, which would require a bunch of handroll queries, and BI tools like Tableau which would be another cost and another big custom query integration job. I've seen PMs spend hours in mixpanel quickly generating custom report after custom report to really understand their data, none of these other front ends would come close to that flexibility for non-technical users.
The biggest advantage of systems like this isn't even the money. You can use the same data warehouse/SQL/BI tool stack to do sales reporting, analyze marketing spend, track support KPIs---everything in your business. My company (Fivetran) does the data-pipeline part of that, we event have a Mixpanel connector for people who want to operate both systems in parallel or migrate over time.
Our goals were to get high volume web data (pageviews, clicks, etc.) alongside application data already saved in our Firebase DB and synced to BigQuery.
Unless you have restrictions about how your data moves and where it is stored, or need to have a trusted computing base with no externally developed software, or have very strict requirements that no available service implements (unlikely), you are better off just using an open source solution or paying for a service.
# Real cost
Even if you have to pay for a commercial service, in comparison, the TCO of developing a similar solution in-house is very large.
When you create a system in-house, you are paying for: design, implementation, testing, maintenance, deployment, infrastructure, security audits, training for users, documentation, and sometimes costs go beyond engineering, e.g: UX and graphic design and such.
After you are done spending all that money, you end up with a custom built service that is far from the actual main activity that supports your business.
# Quality
If you are to authorize a team to do something like this, audit their code constantly and impose a higher quality standard than you do for the rest of your applications. This is because this system will be a dependency for all your applications.
Even in large companies, it is unlikely that you have enough resources to have a large dedicated working on something like this. Because of this, requirements will need to be deprioritized or just neglected.
Because of this, you can lose all hope of selling this solution externally.
# Users
Then, since you are committing significant resources to your internal tool, it is likely that every single team will be forced to use it. That in itself is also a problem. What is better?
a) Learning how to use MixPanel, and put it in your resume (a skill that has market value and can be traded).
b) Learning how to use a proprietary system that is only used internally within a company. A skill that cannot be traded in the job market.
If I am a user, it is against my own self interest to push for an internally developed tool.
To be fair though, it seams to me that they moved from one service that was providing 150% of what they need to another one, Google cloud, that is providing 90% of what they need plus blueprint exemples on how to implement the 10% that are missing.
If I'm not wrong this is totally reasonable, especially if it gives them the flexibility to mine this data the way they want, since it's really close to the core of their business.
Building a self-hosted copy of MixPanel is surprisingly easy. I too did it for a desktop app I'm working on [1]. Took a few days, not more, and is much more flexible. Another advantage is that I retain control of the data.
Does anyone have any experience of negotiating your costs down with Mixpanel? Years ago I remember my CTO ringing up our hosting provider and in the space of a phone call bringing some of our costs down 90%.
Definitely can negotiate, but it’s not going to be 90%. MixPanel is pretty solidly enterprise. At this level of spend they’re sending a team of a couple implementation people onsite and price is def negotiated.
I do some of this but in a slightly different, maybe simpler way:
1. App runs in App Engine (flex). It includes a simple /collect endpoint for mobile events that just logs to /var/log/app_engine/custom_logs/track.json which automatically gets picked up by Google's Stackdriver. App code can just directly log to that file as well to record server-side events. (no network involved in recording events).
2. Stackdriver stores logs in Cloud Storage and also dispatches a Cloud PubSub push to a configured topic.
3. Another mini App Engine flex app receives the message, does the ETL, and inserts the row into BigQuery.
"500 events per second" is approximately zero. We go to 3rd providers, not for "tolerance" but because getting milk off-the-shelf in the supermarket is easier than feeding and milking your own cow (unless you need raw milk or something).
For me, the issue is not cost. It's the data. You're paying for what's essentially a data warehouse and BI tool... but you have little to no control over it. Sure, it's fully managed, but you want to change X? You want to evolve your schema? You need to change some data in history (I know)? You want to build real-time systems based on these streams of data? You want Mixpanel UI for other datasets in your company?
All of these are issues (that I have faced). From what I've seen, people just keep downloading Mixpanel data and uploading to their DWH, sort of voiding the reason to implement Mixpanel in the first place.
The tool is lovely, their export policy is great, but there's something about actually owning your data.
Honestly you don't even need some kind of fancy google architecture, query engine, or columnar data store.
I recently implemented 99% of what mixpanel provides using S3 and lambda. I stored the events on S3 (actually get requests to an S3 bucket with logging turned on and what I wanted to track seeialized as JSON in the query string of the request.
From here it's as easy as writing lambda functions to process the logs and output the results in some location where a static web app can visualize them.
The "backend" would have been a perfect problem to be solved by Google's Cloud Functions (i.e. serverless), rather than a Kubernetes cluster with Node and nginx.
There's one other "big hammer" for saving money at MixPanel: "send fewer events".
We saved 60% (which was no small amount of money) by realizing that sending more than 1 of an event per user per hour was:
A) Not necessary
B) Not even something you could query in MixPanel.
That's a great domain name and seems like a neat product. One of those things you never think of then when you see it you think "of course that's a thing, it makes perfect sense."
69 comments
[ 3.0 ms ] story [ 130 ms ] threadIf Truly you Are In Need Of A PROFESSIONAL LEGIT HACKER Who Will Get Your Job Done Efficiently With Swift Response, Congratulations, You Have Met the Right HACKERS.
WHO ARE COMPOSITE HACKS??? • We are a Team Of Professional HACKERS , a product of the coming together of Legit Hackers from the Dark-Web, (pentaguard,CyberBerkut, RedHack , Black Hat, White Hack ) we have been existing for over 12years, our system is a very Strong and decentralized command structure that operates on ideas and directives.
JOB GUARANTEE: • Frankly speaking, I always give a 100% guarantee on any job we are been asked to do, because we have always been successful in Almost all our jobs for over 12years and our clients can testify to that . To hack any thing needs time though, but we can provide a swift response to your job depending on how fast and urgent you need it. Time also depends on what exactly you want to hack and how serious you are. Enough time with social engineering is required for hacking. So if you want to bind us in a short time, then just don't contact us because We can't hack within 30minutes,sorry. Basically, time depends on your luck. If its good luck, then it is possible to hack within 30minutes but, if it is in the other way round, it would take few hours. I have seen FAKE HACKERS claiming they can hack in 30min, 20min , but there is no REAL HACKER who can say this (AVOID THEM). Please Note : we have only one contact email : compositehacks@gmail.com
We will be happy to have you join over 2000 satisfied clients around the world to use our services.
OUR HACKING CAPABILITIES: There are so many Reasons why people need to hire a hacker, It might be to Hack a Websites to deface , retrieve information, edit information or give you admin access Some people might need us for Hacking any smart phone giving you access to all activities on the phone like , text messages , call logs , Social media Apps and other informations. Some might need to Hack a Facebook , gmail, yahoomail, Instagram , twitter and every other social network Accounts, Some might need to Hack into Court's Database to Clear criminal records. However we can also Hack into school's websites (server) to change grades without any trace, Also Some Individuals might want to Track someone else's Location probably for investigation cases. All these Are what we can get Done withing few hours.
SOME OTHER SPECIAL SERVICES WE OFFER: Hacking and sales of Programmed credit cards & CREDIT CARD TOP-UP Sales of untraceable phones (even the pentagon can not track our phones) Sales of Tutorial packs for Beginner Ethical Hackers.
You can also contact us for other Cyber Attacks And Hijackings, we do almost All.
Contact Us for Your Desired Service: compositehacks@gmail.com
* FREE * 999 a year * Contact us
That number 3 must be really spooky!
(The math may be different in different job markets, in which case, yes, it makes sense not to pay a startup that's paying SF wages to its own employees.)
$192K ($240k/1.25) is definitely common; $120K ($240k/2) is low in these markets for someone who knows what they're doing.
Five weeks of, say, three engineers with $150k salaries ($200k fully loaded) is $60k. People say that the majority of software is after initial development. Let's say 20% is development [0]. That means this project costs $300k total and lives for 5 years before the team decides to rewrite it. That brings us to $60k / year.
This is good savings! My general guidance to folks thinking about this sort of project is that you should implement a vendor and see how you like it. The reason being, MixPanel does a LOT more than this tool, and buying a vendor can help you figure out the subset of features that you actually need. Once you have a clear requirement list, you have a clearly scoped project and can hammer it out relatively easily to get that cost savings.
Beware of thinking that you can do this for $X,000 / year on your own, or get it right on the first shot. There's a reason MixPanel (and any other vendor) has so many engineers. You're paying them to have already made the mistakes that you're about to make.
[0] https://stackoverflow.com/questions/3477706/development-cost...
Personally I much prefer 'pay me now', because 9 times out of 10, IME, some form of technical debt underlies the reasoning about why a team can't do something. 'It always pays to take the pain up front', basically.
If things flip around again and you find a new library that solves your needs better than your homegrown lib, you can drop it in forking your lib and turning it into a wrapper on the external lib.
At least this is what I have learned implementing high flexibility cross-platform mobile video playback solutions in our app over the last 5 years, where the landscape of libraries (and our own tools) has kept changing.
Be limber enough to eschew the tools that aren't working. But "good enough" is, well, good enough for most folks. Perfection the enemy of progress and all that.
The real reason Mixpanel (et al) have so many engineers is:
A) They need to scale their service to reliably provide to N corporations which is frequently non-trivial.
B) They need to implement the 80% of features that aren't critical path for Company A because some subset of them are for Company B,C,D,E,F,G,H,I,J,K,etc.
C) I've seen software vendors make some pretty terrible and painfully obvious mistakes.
Idk, maybe I'm crazy but I have very little faith in this idea that more engineers = better decisions were made. I've just never found quantity has an impact on quality of the end product, only the speed at which its produced.
Yeah, and in my experience, that frequently fails to materialize which why I mentioned point C.
Now, that could be arrogance on my part but it doesn't change my experience on the quality of the vast majority of outside software vendors.
Unfortunately, the part "implement a vendor and see how you like it" went only with the first half. Complaints of developers supposed to be using the tool were brushed aside as a simple resistance of change. In the end it was implemented by decree from higher management.
Only after all the paperwork was signed, we started to see that the vendor doesnt support our usecases, that their hardware is underpowered for the load we were expecting, e.t.c ...
In the end, I have heard at least 20 various senior and beyond senior engineers from various teams complaining that they spend 20 to 50 % of time working around the vendors bugs, around once a month bringing down vendors infrastructure, because our use-case mandated uploading more data than the infra could handle.
I definitely agree that clear requirement list and clearly scoped project and implementing a vendor to see how you like it is usually better than in-house
On the other hand in-house is usually better than implementing a random vendor because management feels like it ;)
Using Metabase for visualization.
The other key benefit is you can warehouse the rest of your data in BQ as well -- where it can be easily linked to your non-event data; marketing data from other systems, etc.
1. Right off the bat, there's the cost of building all of this + maintaining it. Something like Mixpanel at scale requires at least two, if not three, engineers: one for client libraries, one for infrastructure (which the OP blogs about), and another for the web app (dashboard, real-time user stream, etc.) To be sure, not everyone needs all features of Mixpanel. To be really sure, nobody needs all features of any SaaS tools. But if you really want to compare apples to apples, then you have to account for these. To hire competent software engineers who can collaborate closely and maintain a complex piece of analytics software requires at least $100k per year, if not twice that based on locale. That's at least $300k and more like $600k right there.
2. The whole point of something like Mixpanel is to make analytics accessible to non-engineers. In their case, it's primarily product people and secondarily marketing/customer success. In any case, building an analytics/data product consumable by non-technical people is hard and takes way more than assembling a couple of cloud infrastructure together. If there's one reason Mixpanel is still in business, that's because of this.
3. Finally, the OP has a valid point which should have been highlighted more, if not to make their own biases more clear: Mixpanel's diminishing differentiation is dev shops/consultancies' opportunity. It is indeed incredible that a dev shop can build even a third of Mixpanel's functionality by leveraging GCP components. Mixpanel had to build a lot of its core backend systems from the ground up, including its original key-value store. Just this year, they fully migrated to Google Cloud Platform themselves, suggesting that there's really little room for differentiation among analytics vendors at the level of infrastructure components (Mixpanel's arch-nemesis, Amplitude, leverages Apache Kafka and various AWS components, most notably Amazon Redshift, heavily)
With all of this being said, one thing remains true: the most expensive cost of any software is people running them and the dependencies created around them. These may not show up as line items, but they sure are deeply embedded in your total cost.
What happens if the system collecting all of your data has vulnerability? suddenly all of your apps and data are owned.
What if it's inefficient? then you have DDoSed yourself.
Non-functional requirements cannot be solved with a functional requirement mindset.
What if you are losing data? what if you aggregate it incorrectly? what if data is ingested in the wrong order? ... you get the idea.
It's not a 3 people problem. When you take on a hard problem like this you have 10 different things to cover and you need to pick all of them.
Also, this post only addresses data flowing in one direction into BQ. Presumably they're then using something like Tableau or the Google/AWS offerings to build dashboards based on this data. There aren't many places for bugs to hide unless the BQ perms are wide open.
I don't see anywhere that the article says they have replaced all of Mixpanel. It only says that they were able to replicate the functionality their client actually needed. I don't see them making claims that the solution they built is anywhere comparable to the full solution from Mixpanel. Only that it provides all the features their client needed.
That last part is the most important. It's why you're really using Mixpanel/Amplitude/other vendors. To say "we replaced Mixpanel" without addressing the front end feels a bit disingenuous to me.
I was thinking GKE = Kubernetes, and GCE = was their "run container on a VM" thing, but I think GKE and GCE are the same thing...
One of my goto none coding interview questions is to have a candidate design a mobile analytics solution.
From the article, I've learned Google has a reference architecture: https://cloud.google.com/solutions/mobile/mobile-gaming-anal...
MixPanel costs $X per month. That cost only increases. Self-based solution costs $Y to build and $Z to operate.
delta = $Z x n - ($Y + $Z x n)
Solve for that delta.
Our goals were to get high volume web data (pageviews, clicks, etc.) alongside application data already saved in our Firebase DB and synced to BigQuery.
We picked Keen because it has an open source web tracking lib https://github.com/keen/keen-tracking.js/ that easily plugs into our React/Redux stack.
They also have built-in streaming to BigQuery: https://keen.io/docs/integrations/google-bigquery/
Keen pricing is about 10% of mixpanel, so for our limited needs it has been working well.
Long term if our volumes really grew the original post looks like a good option, but figured we'd pass along this lower dev approach.
Unless you have restrictions about how your data moves and where it is stored, or need to have a trusted computing base with no externally developed software, or have very strict requirements that no available service implements (unlikely), you are better off just using an open source solution or paying for a service.
# Real cost
Even if you have to pay for a commercial service, in comparison, the TCO of developing a similar solution in-house is very large.
When you create a system in-house, you are paying for: design, implementation, testing, maintenance, deployment, infrastructure, security audits, training for users, documentation, and sometimes costs go beyond engineering, e.g: UX and graphic design and such.
After you are done spending all that money, you end up with a custom built service that is far from the actual main activity that supports your business.
# Quality
If you are to authorize a team to do something like this, audit their code constantly and impose a higher quality standard than you do for the rest of your applications. This is because this system will be a dependency for all your applications.
Even in large companies, it is unlikely that you have enough resources to have a large dedicated working on something like this. Because of this, requirements will need to be deprioritized or just neglected.
Because of this, you can lose all hope of selling this solution externally.
# Users
Then, since you are committing significant resources to your internal tool, it is likely that every single team will be forced to use it. That in itself is also a problem. What is better?
a) Learning how to use MixPanel, and put it in your resume (a skill that has market value and can be traded).
b) Learning how to use a proprietary system that is only used internally within a company. A skill that cannot be traded in the job market.
If I am a user, it is against my own self interest to push for an internally developed tool.
[1]: https://fman.io/docs/metrics
May I ask why? Seems like exactly the kind of data where some loss is fine.
https://discourse.snowplowanalytics.com/t/porting-snowplow-t...
Is 500 events per second considered high throughput over here, or do other people feel they really just needed fault tolerance from Google cloud?
All of these are issues (that I have faced). From what I've seen, people just keep downloading Mixpanel data and uploading to their DWH, sort of voiding the reason to implement Mixpanel in the first place.
The tool is lovely, their export policy is great, but there's something about actually owning your data.
I recently implemented 99% of what mixpanel provides using S3 and lambda. I stored the events on S3 (actually get requests to an S3 bucket with logging turned on and what I wanted to track seeialized as JSON in the query string of the request.
From here it's as easy as writing lambda functions to process the logs and output the results in some location where a static web app can visualize them.
We saved 60% (which was no small amount of money) by realizing that sending more than 1 of an event per user per hour was: A) Not necessary B) Not even something you could query in MixPanel.
more thoughts in: https://blog.ratelim.it/blog/how-to-save-money-on-event-trac...