Ask HN: Is there server-side software that we are missing in 2018?

It feels rather like we have too much software.

I'm still waiting for a rock-solid scalable open source graph database in the mold of the Freebase database engine or the amazing graph database that Facebook have built for themselves. I'm very excited about dgraph as an option here but I think it's still an area that is very open for new entrants.

Static website generators.

> Nowadays it seems like there's nothing missing.

There's lots of things missing. There's just nothing missing in established categories. Why would there be?

Start your own category, create software for it, then convince everyone that the category is important.

There might be something exciting to build to help implement ludicrously fast Google-style autocomplete / typeahead Search. I've tried using MySQL, elasticsearch, PostgreSQL with trigram indices... they can be made to work, but I've never felt that I'm anywhere near the quality of whatever it is Google are doing here.

Some of the biggest advancements in recent years have come less from technological breakthroughs, and more from improvements in designs and abstractions. Tools like Ruby on Rails, GraphQL, and even AWS, while impressive technically, are breakthroughs because they improved developer efficiency. They also weren't "needed" until they were built, and have allowed many developers to work on broader parts of the stack.

Also, today's tech solves today's problems.

Before storing data we should have a good do not track / privacy respecting software for the data we store.

An open-source platform for self-hosting function-as-a-service - something that provides the tooling for easily saying "deploy this function, auto scale it, route HTTP traffic to it, now atomically replace it with this new version" without having to lock yourself in to Google/AWS/Azure.

There's two ways to "fill a gap", either you create a gap that needs filling or you make an existing solution better. You don't see anything missing because you aren't looking at things from this perspective.

I want a distributed (not cloud) file system that handles many files and WAN latency (i.e. not HDFS). It might be a cross between Git and BitTorrent.

It feels like deb repos, PyPI, NPM, CPAN, CRAN, etc. should be put in there, with the addition of binaries for popular architectures. And probably Docker-like images, although I think if they are not opaque blobs, it would be better for rsync-like differential compression (which Git implicitly provides).

There will be some small files and some big files. I want it to be like Git so I can clone locally, not just go to the cloud. The way that Git is trivial to set up and clone through SSH is nice too.

It probably has to have a notion of "user", like a local file system. (This makes the problem a lot harder; git doesn't really have permissions.)

BTW Julia's package manager just used git, but I watched a talk that said this ended up being a really bad idea, especially on Windows.

As far as I understand IPFS has some of these properties. Has anybody used it? Could I use it for the package repository use case?

I don't know much about it, but Project Atomic sounds similar too: https://www.projectatomic.io/

Any other projects that seem like a close fit?

BTW I think this would also be useful in the data center, as some companies like Twitter apparently use BitTorrent in data centers to start large jobs quickly (i.e replicate the same 500 MB binary to 1000 machines).

Serving machine learning models in production is still something that appears not to have an obvious correct solution.

10 years ago, I learned about the idea of elastic infrastructure. Services with load balancers in front of them, hosts that come and go easily, a structured way to communicate between services.

At the time, that was nginx + manual autoscaling, then it was ELBs and autoscaling groups, now it's kubernetes and containers, maybe hosted. It's still not there.

I'm excited about the software that makes that operable at scale. It seems like a service mesh is a good idea. It seems like mutual security between services is a good idea. It seems like storing routing configuration in a separate control plane that is executed in a data plane like Envoy is a good idea.

There's a bag of software at CNCF that's loosely organized around this, but I don't think the "just deploy some code, it can scale and you can have tons of services doing that with good visibility and operability" is quite there. I'm really exicted about Envoy, but I don't think there's a good control plane for it. I'm part of a company that's working on a commercial implementation (turbinelabs.io), and Istio is in a similar space.

There's still work to be done!

Machine learning model management and serving service.

I haven't seen a decent open source framework that makes it easy to package a trained model with prepocessing and postprocessing steps and deploy it behind an API.

Adding performance tracking and model validation on top of that would be great.

Then there are things like queuing/batching and autoscaling.

Closes thing that comes to mind right now is tensorflow serving and their k8s stuff (which it looks like they just renamed to tf-operator and moved to kubeflow org https://github.com/kubeflow/tf-operator)

I feel like we're in a consolidation phase. We have a ton of tools/software and the big developments have been in how we use those tools aka devops. I'd put things like kubernetes/docker in this category, but there's obviously huge room for growth around this tech.

I do think there's room for new tools tho, but by definition, they're on the cutting edge and not especially visible if you're not looking. Things like stream processing are already incredibly useful and only getting better. Machine learning could be in this list too. It might not be a 1000% improvement like caching, but that's part of a maturing industry.

I believe the problem has now shifted to a slightly higher level and especially when working in a microservice/containerized world. For example, given the number of specialized services you mentioned, we see more applications with polyglot persistence stacks underneath. If they form a part of the same application, how do you collectively manage them? What does it mean to take a consistent snapshot? These and a bunch of other similar questions need to be answered.

There is some work we are doing (https://kanister.io) to help with these issues but there are a lot of emerging solutions in this space. Look at CNCF's landscape for some more detail.

1. Static Website generators for non-blogs 2. Half decent wrapper libraries for tools like FFMPEG in NPM 3. Plug and Play game server with stat tracking that can be easily used for any game 4. More versatile compression formats for media, ability to serve up 360/720/1080 from the same file?

I haven't found a good authorization service (fine grain roles, acls, etc). I always find myself having build it for every new project/company.

I don't think anyone has solved the issues raised in Out of the Tar Pit in the data storage space. We still spend a ton of effort, perhaps more than ever before, wrangling incidental complexity. In 2018, I should be able to define a simple relational schema in 30 seconds and start using it with effectively zero constraints around data access patterns, transaction volume, and data scale, and with no design decisions needing to be made around these bits of incidental complexity. There should be one gigantic knob I can turn to spend more money to increase more resource consumption of the system, and it should be damned cheap.

We are so far from that reality that it seems obvious there is still a ton of work to do. You can see bits and pieces of solutions but if your product asks me to define indexes, tune queries, determine sharding keys, write against computer-centric (vs human-centric) APIs, cannot deal with change easily, requires babysitting, or falls over after hitting some incidental tipping point relative to the resources available to a single machine, you've not hit it.

I have not used Google Spanner but at 30k/ft it seems like the closest thing out there to the idealized case -- but being closed source and centralized it is not really a "solved problem" imho.

   customers = [id:int, name:str; 1 "jhon"; 2 "doe"]

   inv = [id:int, lines:Lines; 1 [id:1, qty:1, price:$10];  2 [id:1, qty:3, price:$40];]

   fun CoolIndex.get(key) -> Value

I'm a big believer in logging, and I find it hard to believe that collecting and monitoring logs needs to have a server with 32GB of RAM and massive Java and ElasticSearch back-ends like ELK, Splunk, Graylog, or similar monitoring software.

I've been searching for something self-hosted that can run in my server's spare capacity and monitor the OS and application logs with a simple searching interface (just a web interface to grep would be handy) and come up short. If I can't host it on the cheapest DO plan, it's out of my budget. I really don't want to have to build it myself but I'm just about at that point.

Given the sheer number of services on AWS, I would like an expert system that would query me for requirements and then suggest a set of possible services to use together to solve my problem.

I don't have great suggestions for you but I think software needs are ever evolving. As more of the world comes online, there are more data requirements, newer business requirements, etc. We would need even more scalable systems on various dimensions.

I would say anything in the video / 3D / imaging service platforms always have a lot of things left to be desired for.

Anything that potentially touches FFMPEG basically

I can name a few examples that I still think need improvements

- Online gif editors

- Video editing / clipping

- Background image editing / online photoshop equivalents

- Better alternatives than lucidpress / adobeIndesign for catalog page / brochure creation

- Machine learning / deepfake online tool, this is all driven client side mostly

- Pretty much anything client side thats not yet server-side is open game I would say

- PDF markup tools could be better for online-based services, especially for architectural design

- CAD-based online programs for retails so customers can DIY build their own warehouse or layout schemas is lacking

- Better online RDBMS. Currently, there's just airtable, its lacking some core features like refential integrity

- Integrating space-repetition learning in most educational based services (lynda.com,pluralsight, etc)

- Managed ecommerce cart / hosting services. Its a well understood problem that should technically be easy for a client to do.

Again, I would say, almost every profitable service touches some form FFMPEG for video / image editing / 3D is definitely still out there. There's such a huge untapped market out there combining all of theses services in one package.

I really want to simplify writing backend logic and implement everything like functions that reacts on some events and produce another one's. This is a very very looks like, Actor Model (Akka) plus Redux's Reduces: We have just a bunch of a state changing rules and they can be executed reliably and with decent performance.

Something very simple (for example for Build Server):

Build Failed (Event) -> Assign responsible user for a crash -> Send Notification -> Deliver Notification via user's configured notification systems (email, push, sms..)

This is a Event Sourcing, but event sourcing works well only for one part of the platform - write side, but reader side is sometimes too hard to implement. Sometimes there are a problems with eventual consistency and you actually need to wait while some of the reducers in chain will process this event and starting from this point everything became toooo slow to develop and you basically start to redevelop the wheel - this is just a database engine de-factor. Meh.

I want a way to use AWS Lambda/GCF but with cluster level network locality.

> It seems like there are so many choices in each category that there's nothing left to do.

Log storage and search for structured logs (e.g. JSON or CEE, not merely stringblobs). We have paid solutions (Splunk, Loggly, Papertrail), and then we have Elasticsearch, which gets worse in this use scenario with every release.

Message stream processing engine that doesn't require restarting to add a query or data sink, so you could build monitoring system around it. In fact, a monitoring system designed to allow you to easily add your custom processing or data sink.

Infrastructure inventory that can be both filled by hand and kept updated by machine and that can be queried from script or browsed by human. For that it would be useful to have a good topic maps engine, which is another missing thing.

OS updates manager that can handle more than just Red Hat/CentOS (Red Hat Satellite or Spacewalk) or just Ubuntu (Canonical Landscape), and while at it, one that doesn't try to be underdeveloped configuration management tool (like CFEngine or Puppet) and underdeveloped deployment tool (like Ansible), but can cooperate with them.

And there's much more where these came from.