Semi-related... I work in wellness and healthcare.
I don't know about you, but I despise filling out the same forms over and over again when seeing new healthcare providers. I'd love to start a service modeled after granular smartphone permissions where
(a) I check in at a new office (scan a code, they scan my code, beacon, something like that)
(b) the office then requests x, y, and z information
(c) a push is sent to my phone where I can review the information and approve or disapprove some or all permissions
(d) a final step of either entering my pin at the office, using my thumbprint on my device, or something else.
The key components would be storing the data encrypted at rest, following HIPAA and then some, having a solid auth protocol (keys, jwts, etc).
I think adoption would be helped because the public are already used to permissions like these when installing apps.
The benefits are a lack of paper trail, no one is going to not shred my SSN, my most up to date data is now available, and instead of hosting N apps/databases, I'm storing 1 and can reduce my maintenance, customer support issues because one for all, all for one.
The military is terrible for this. You are constantly filling out forms that amount to a half page of your basic information, followed by a couple of text fields that form is actually for.
I'm sure it's possible to hack together an AHK script[1], combined with Pulover's macro creator[2] to automate virtually anything repetitive on a Windows PC, or use Selenium to automate browser actions[3]. Of course then you run the risk of having to fall into the classic XKCD automation time sink[4].
Too much inertia on the provider side for this to catch on and reach critical mass - many septuagenarian sole practitioners out there using paper diaries / files, and larger organisations with some monstrosity written in COBOL (or MUMPS?) that will never change to accommodate this.
I'd suggest something much more low-tech - a website where you can punch in all your details - insurance, allergies, medical history, etc, etc... and then you can print it out (or a subset of it, for different kinds of providers) or generate a PDF that they can copy & paste into their horrible legacy system (an improvement on retyping), or, for those truly at the cutting edge - the kind of electronic transmission you speak of.
I'm probably bias because I've lived in two areas now where healthcare is one of a few, if not the, major industry in the area. They're always trying out new apps and services here.
I am on board with what you're saying; an escape hatch for non- or semi-adopters. Obviously, printing is a way to go, so maybe on the mobile app, the ability to check each piece of information required then export/email to your preferred destination.
It'd also be interesting to look to make money on conversion i/e replacing, or integrating with, the outdated monsters you're talking about.
Maybe we're not even talking about healthcare anymore, maybe just the ability to piece together PII (personally identifiable information) and deliver it to X.
>>>> on another note
This goes into a topic I've seen posts on recently, and something of interest to me, personal indexing; a better way to throw blobs against the wall and have it indexed for me, leading to a personal Google. I mean, that's already coming, really, between Facebook and Google (especially Google Photos) but currently I see nothing about piecing together information I'd like to share on a professional level.
Hmm, Google Drive does a reasonable job of that. It indexes everything (including OCR for images + PDFs), has decent search, and has per-folder permissioning and sharing.
It's actually a pretty good solution for ad-hoc "working together" with someone (a lawyer / architect / whatever) on a project, where you have lots of files you need to share and refer to during the project.
Maybe the problem is we're trying to get the wrong people to pay. Since the pain point is with patients, fix it for them and make them pay. Gets around the industry inertia.
Sell the service to the patients for some smallish fee ($5 per month) and then provide the integrations into the various provider systems for free.
Later on you could scale it up to be an add-on to employee benefits or the health plans.
I am working on a patient-driven platform which brings together all key stakeholders with support of a few good partners, and I am currently conducting user interviews for it. I would love to talk with you more about your ideas. To make scheduling painless, I have a link in my LinkedIn profile summary:
I would love to hear from anyone else with big ideas relating to or are working on driving outcomes towards holistic wellness with patient-center healthcare, patient data collection/quantified self, and patient-powered research networks. In the bigger picture, I am passionate about making the world a better place through innovation and working on what really matters for humanity.
Insurance card scanning and recognition is available. Costs $999.[1] This has apparently been around for years; there's Windows 98 support. It's been acquired by AcuFill [2]
They also offer identity document verification with facial recognition crosscheck. They want to use this to detect visa overstayers for immediate deportation.[3] That now looks like a market with potential.
We just built something like this for the health market. Users can auto-sigin to websites with one of their identities or set it to ask for each visit. Here is a lil demo I made that uses Craiglist as an example: http://tricorder.org/cl
When the user goes to your website, say wellness.com/newcustomer, there are javascript APIs to get at your standard data, that brings up a perm dialog and if the user accepts, the data is sent to the website. Send me an email (profile) if you want to talk biz, tho was planning to open source it. Auth is very solid, but its currently android only.
France has national healthcare and everyone has a smart card with vital information, and all doctors have the hardware to read it and software to process it. Or at least they did 15 years ago when I was there.
France has national healthcare and everyone has a smart card with vital information, and all doctors have the hardware to read it and software to process it. Or at least they did 15 years ago when I was there.
The more in-depth the examination and the more time you spend with the patient, the more they can charge. All those forms are "taking family history", etc. and it is free money since you have to do the work. Those are then scanned so they can be used later in an audit.
(Source: I also worked at a start-up that was trying to disrupt out patient medical systems. It's very hard and has lots of roadblocks. btw, of the top 50 EMRs in the US, only 3 have APIs and these are mostly to pull data, not push it back in).
I'm surprised none of the comments mentioned ZocDoc, which does most of this already. You can fill out forms once, schedule appointments and click to send your info to the office.
Here's an observation. HIPAA applies to health care providers, insurance companies, and other entities like that. HIPAA does NOT apply to me when I am in possession of my own personal health records. Not saying such an app should not be secure, but for me to hold my own records is regulatorily simpler than HIPAA.
Not in my industry but I'm part of an HOA and I swear 2/3 of what the people paid to administer it do seems streamlinable/automatable with lots of opportunity for making money along the way. Every time I deal with them I proclaim to myself, STARTUP! Then I forget about it..
There's BuildingLink in this space. I think this is the kind of thing where selling software to existing, backwards organizations is a big marginal, and it might be better to be a property management company with good in-house systems, instead of trying to sell good systems to individual HOAs or small management companies.
Have HOA and totally agree. So many key functions that could be improved: submit work orders and report security incidents online, pay dues and special assessments electronically, review association budget, monthly news and events, etc.
Problem is these who would benefit (residents) are often disconnected from those against it (the employees). You would need boards that really are sold on the benefit.
Homeowners Association - Might be US thing - but basically Condos / Townhouses / even neighborhoods form associations to take care of the 'commons' / enforce a look for the neighborhood, etc... There's usually a monthly payment. Repairs / work orders / disputes are settled through them. There's bylaws...
2 of our 3 rental properties have an HOA and its quite annoying getting postal mail about things then having to relay messages to tenants. Or you get some action to vote on but have no clue if its a good thing or not because you're not involved in the politics part of it.
I can confirm. HOA software is big. It's mainly managing PDFs and sending emails out. E.g. https://www.condocerts.com
It's a decent industry my friend (not after this posting) makes ~300k a year working on it. However the race may be ending as these companies lock apartment managers into long complicated contracts. Not sure how many apartment managers at scale still need.
I'm on my HOA and I imagine the hardest part of making this work would actually be sales. The sales cycle for an HOA would be multiple months between meetings and actually making a decision. Then there's the issue of rotating responsibility, retraining, no guarantees of technical skills among board volunteers.
I'd tend to agree with the other comments that the hired out management companies would be a much better sales target since they have a real profit motive to efficiency gains. At best a neighborhood HOA is looking at periodically saving some time and being a bit better organized.
I'm in the AI industry and our problem is that we have so little grasp on what intelligence actually is, that we stumble in getting our machines to mimic portions of human cognition even on the vector space level.
As a SaaS provider, one of the key indicators of a customer at risk of churn is the presence of another competitor in their account. A service which notifies you once an account signs up for a competing service would be immensely valuable in helping to target retention activities.
You could employ some sneaky tactics to ferret out if a customer's browser visits a competitor's website... without letting the competitors or the custom know.
I can think of a solution that's VERY VERY anti-consumerish.
Being a SaaS provider, create an extensions that you have to install (something consumers want...). This extension should have the permissions to read history / or urls visiting.
Have a blacklist transmitted of urls (so you don't need to transfer the user's data back to the server) and match with it. If it matches...you have to work on that consumer a lot more.
1) how about you find ways to make retention a continuous objective
2) get used to the fact that not all customers will be retained. I try stuff all the time that I have no idea if it will solve my specific problem. If you do #1 right, I would know that you want to hear about my specific problems and offer ideas on how your saas can solve them. That's the most you can hope for if your service doesn't immediately do what I initially hoped it would (or is more burdensome to implement that I thought, etc)
This sounds like a massive invasion of privacy, to me. It also sounds like a great opportunity to drive customers further away when you contact them based on that information.
Content and patch distribution for video games: Data integrity, progressive downloads, file-level patching, compression, encryption, and platform/version branching.
It's quite mind-boggling; nobody is really doing it on an industry-scale level. Every video game developer has their own way, all of which have their own problems.
It is a very hard problem. Blizzard actually came up with a very good system, but it's not in a state where it can be commercialized or open sourced.
I actually think whoever comes up with a system which solves these problems in a clean and consistent way will be sitting on a little revolution for content distribution.
For themselves! It's also not a great system, lots of legacy. More to the point though, it's not a commercial system. Steam behaves as a distribution platform and licenses the publishing rather than the distribution.
Which is not at all interesting for self-published games (be it indies who want to avoid steam, big publishers with their own systems etc).
Steam supports any game from any publisher, including self-published games. Steam is exactly what you get when you try to solve this problem, because none of the big companies want to be involved, since they see it as competition. Blizzard and EA are the only two big companies I know of that are not on Steam, and they both have direct competitors to it that are vendor-locked.
I have devoted 10 years to game content distribution, packing, compression etc. (Now not in gamedev anymore)
This is a very easy problem which usually solved by attaching fairly simple script which is aware of your file formats to any commercial installer system.
Some companies are even selling more or less standard solutions for that, but in reality from any given 1000 games 900 will have very different data formats and all have fairly good reasons to do so - using universal "patch systems" really creates more problems.
I think the "900 different data formats" problem is something that will go away as we move towards better tools which cover all the standard use cases.
Gamedev is riddled with really smart people that reinvent the wheel all the time because they found a way to micro-optimize this or that. They get to do this because until recently, there was no "good enough" solution for a wide range of games (or the "solution" was priced with enough zeroes to make bill gates cringe).
But you saw how popular Unity got, and how fast. That's the games industry in a nutshell: ripe for solutions that work for more than just one studio.
BTW Unity, with all its excellence, has really horrible data format for content and patch distribution, and had and still has huge problems with this. Perhaps the legacy of early overengineering and struggle to protect the games from easy reverse-engineering.
And compare, say, to simple incremental zips of Quake with alphabetic file loading order.. Total no-brainer to implement and use. (I have even seen zips with custom LZMA compression!)
So, if any, someone will have to solve a problem of artificially created obstacles, not a problem per se.
I used to use irrlicht and Ogre. Both have the problem of only really doing graphics and to a certain extent input. In comparison, Unity and Unreal offer the whole package: graphics, asset pipeline, audio, networking, and physics.
Speaking from experience as I'm currently making the jump to Unity for my projects, the time savings of choosing one of the all-in-one engines instead of gluing together engines is really substantial.
The path forward for games is roughly similar to where digital audio is now: Comprehensive workstation environments with an easing facade through plugins, presets, etc. The coarse elements of a rendering algorithm or a piece of game logic can be reduced to a processing graph, behavior tree, or other convenient abstractions. They can plug into each other by exposing both assets and processing as globally addressable data. Original coding for game logic will still be required for the foreseeable future, but most of the development problem is weighted towards getting assets in the game, and that can be abstracted.
This is done in bits and pieces across existing engines and third-party tools, but there's a lot of room to make it cheaper and easier.
Wharf is a protocol that enables incremental uploads and downloads to keep software up-to-date. It includes:
A diffing and patching algorithm, based on rsync
An open file format specification for patches and signature files, based on protobuf
A reference implementation in Go
A command-line tool with several commands
Butler is the commandline tool for generating patches (it can negotiate small diffs from the server without requiring a full local copy of the thing you're diffing against), uploading them and applying them back on the client.
It is used to power itch.io's Steam-like application, itch: http://itch.io/app, delivering multi-gigabyte game installs & updates.
Hey, amos here, main developer of wharf/butler, here's a quick technical summary so you don't have to do the digging yourself:
- File formats are streams of protobuf messages - efficient serialization, easy to parse from a bunch of programming languages. Most files (patches, signatures) are composed of an uncompressed header, and a brotli-compressed stream (in the reference implementation, compression format are pluggable) of other messages.
- The main diff method is based on rsync. It's slightly tuned, in that: it operates over the hashes of all files (which means rename tracking is seamless - the reference implementation detects that and handles it efficiently), and it takes into account partial blocks (at the end of files, smaller than the block size)
- The reference implementation is quite modular Go, which is nice for portability, and, like elisee mentioned, used in production at itch.io. We assume most things are streaming (so that, for example, you can apply a patch while downloading it, no temporary writes to disk needed), we actually use a virtual file system for all downloads and updates.
- The reference implementation contains support for block-based (4MB default) file delivery, which is useful for a verify/heal process (figure out which parts are missing/have been corrupted and correct them)
- The wharf repo contains the basis of a second diff method, based on rsync - for a secondary patch optimization step. The bsdiff algorithm is well-commented with references to the original paper, and there's an opt-in parallel bsdiff codepath (as in multi-core suffix sorting, not just bsdiff operating on chunks)
- A few other companies (including well-known gaming actors) have started reaching out / using parts of wharf for their own usage, I'll happily name names as soon as it's all become more public :)
It's actually exactly what you described - the documentation is very sparse on it because it's an internal thing (I'm guessing you found the CASC documentation, not the NGDP one). If you're interested, shoot me an email and I can send you some more details; but it'd simply be for intellectual curiosity, as I said it's an internal protocol.
It's exactly what you want, but this service isn't poplar. It doesn't work because everyone is on Steam. Network effects. It's like when app.net tried to replace twitter. You can get mad at users, but users are using Steam & Origin and Battle.net and they don't care.
Normally I live in China, and rarely eat western breakfast.
Recently I returned to Australia to spend some Christmas time with my extended family.
A few mornings ago, I put some real bread I cut from a sourdough loaf in a toaster. Due to its irregular size, when it popped it didn't pop out completely, resulting in a sort of "toaster is too hot to insert fingers, toast is too hot to hold, toast is ready, find metallic implement to insert in to mains-powered device to extract toast" problem.
In the US, most toaster ovens I've seen have about 2-3 times the footprint of the average toaster, with additional height. This is actually pretty reasonable as you can use the toaster oven instead of the oven or microwave oven for many tasks (it's a less specialised tool than a toaster).
Last I read, US homes are the second largest in the world, after Australia. In Asia, many people live in apartments and that space is unavailable / non-negotiable. In addition, microwaves are not nearly so widely used.
And yet, every tiny apartment I visited in Korea and Japan had a toaster oven. The toaster oven is an incredibly useful and versatile tool that can stand in for bigger appliances, like full-sized ovens, in pinched spaces.
Sure. In China we would use chopsticks. But my point was not that there is no viable workaround, more that the mechanical issue should be solved.
Perhaps it is playing too much Shenzhen I/O, but I feel like a basic IC and sensors could solve this by detecting the width/height of toast pieces and continuing to release until the toast was substantially ejected from the toaster.
I think most toasters pop up using a coiled spring. Replacing it with a linear actuator seems like the way to do this. Perhaps some high end toasters already use an actuator with a laser to detect if the toast is 'up' enough. Fancy stuff
With irregular pieces on this and other toasters I have used, part of one piece of toast can block the further vertical motion of the carriage at the top portion of its linear range.
Others have complained if the slot is too big for the piece, the piece falls one side and gets toasted in one part and not in another part. That's other people's feedback, not mine.
Some toasters have wire meshes that close in on the bread to hold it vertical, but they tend to leave marks.
Your suggestion amounts to "work around it". That's one perspective, but also consider the very real problems of irregular ends of the loaf (where cutting thinner would result in a piece too small along the other two dimensions to be useful), loaves with air pockets that need to be cut thicker to maintain structural integrity (and utility) of the resulting toast, and people who simply prefer thicker slices.
What you want, then, is more akin to a sandwich press rather than a toaster. Or perhaps a toaster oven. Mechanically, it's not impossible, but the design of a toaster (spring loaded release, heating elements on either side) can't be altered much without turning it into one of those two things (or a poor imitation of them).
Seems a tortured equation: perhaps a vertical sandwich press without the press! I was thinking it may be possible to change the spring to another form of vertical linear actuator (eg. stepper motor driven) and giving it more controlled movement, also smarter with proximity/LOS-style sensors. Perhaps the top cover could also expand and contract. The very fact there's a spring-loaded release shows this is old tech. The market can clearly tolerate a few dollars for a stepper motor, as there are 'smart' toasters selling for nearly ~$200USD and the bottom is ~undifferentiated. See http://www.brevillegroup.com.au/wp-content/uploads/2016/10/B...
Would an ordinary stepper motor cope with the temperatures? Our cheap toaster has a handle that lifts (so isn't limited by spring return); and the socket off switch is the backup for when you need to jam a form in to reach to get your crumpet out.
To be fair there shouldn't be any voltage inside the toaster basket area when the toast has popped up. Unless they did something stupid when designing the toaster. And you can always unplug it before sticking utensils in there.
Sure. To be clear the gripe is less about the potential for a shock (real or otherwise) and more about the clearly nontrivial frequency and irritation of the issue.
I've had the same thing happen with toasters, and at least as of a few years ago it was completely possible for a piece of toast (of just the wrong size) to get stuck such that it not only didn't pop up, the toaster didn't turn off. So the toast starts to burn, you grab a knife, what could possibly go wrong?
This happened pretty regularly as it was a European toaster designed for perfectly uniform extruded wonder-"toast" and I kept sticking hand-cut slices from round loaves into it.
I bought my bottom of the barrel Wal-Mart toaster for $6 about 8 years ago. It might occasionally have issues popping out a piece of bread but it cost $6 and it toasts bread consistently.
I'm mostly holding on to this thing to prove a point to the "they don't make them like they used to" and the "you have to spend at least $100+ on X or else it will break in a week" crowd. I know it isn't the best toaster out there but I'd take it over a $189 toaster even if both were offered for free just because I'd feel like an idiot for having something so ostentatious on my countertop.
1) Cleaning the data as it comes in rather than in batches so we can use it sooner, invalid data is discarded, outlier detection, normalizing inputs etc....
2) Warehousing of the data with proper indexes so you can perform some advanced queries on unstructured data
3) Some data is sent in bulk at the end of day, some of the data is streamed in fire hose style. How can we preprocess the fire hose data so that we don't have to wait until the end of the day to parse it all.
4) Oh and all of this data is unstructured and comes from 75 different sources.
Soon the average hedge fund will have more people just cleaning and managing data than they do in quantitative research, dev ops, software development and trading.
Oh and lots of the data is considered proprietary so while AWS/Azure, etc is fine, sending it to a third party to process is not.
TL/DR
Help me, I'm drowning in data. How do I get the time from when I acquire data to when I trade based on it down to a reasonable time frame, where reasonable is closer to hours rather than days/weeks.
Great question, but I don't see this much different from other industries or companies with millions or billions of dollars at stake. They don't all roll their own software inhouse, and most companies (hedgefunds or not, small or even large) simply cannot afford to for financial and commercial risk-management reasons unless they can justify the software truly being a core competitive competency.
The same line of inquiry has been evaluated for most 3rd party software that companies rely on. For this specific instance of data collection and cleaning, I'm imagining it's not going to be a much different calculus, although perhaps you'll see a higher percentage of firms choosing to roll their own if they have the chops and pockets (e.g. Two Sigma, Bridgewater, Goldman Sachs, etc.).
I will note that there are commercial mechanisms firms could try to implement to try to limit the downsides in case something like this happens: warranty & damages provisions, and insurance are two come that spring to mind. I'm sure there are numerous other considerations in the age-old "build or buy" cost-benefit analysis.
Palantir doesn't really have a product, they write tons of code to put everything together and make it look seamless from the UI but they don't have anything drop in as far as I know.
There are good open source options for each step here - is the solution you are looking for just a UI and easy install process? Or would your ideal solution make all of the decisions for you - data structure and format, which data is/isn't valid, what output options are possible, managing server resources, etc.?
I'm not very familiar with the open source options since after many years of coding this by hand, I work with what I know. I am a developer that works with data, not a Data Scientist, so I don't really know the lingo and whatever hipstery terms people are using these days. I will answer to the best of my ability, though (mostly for my sake, who knows if this will be useful):
Cleaning:
Open Refine seems to be the best product in this category. I haven't used anything but my own tools to do this before, so I can't really offer any advice.
Warehousing:
My understanding is that this is just a fancy way to talk about a database with a schema designed for analytics. There are many open source databases which do this very well, the one I use being Cassandra (and/or KairosDB), though it is also likely the one that is hardest to use. For a beginner, you might want to refer to this SO answer: http://stackoverflow.com/questions/8816429/is-there-a-powerf...
Data processing/collection:
This is something that is incredibly dependent on the data sources, so I likely can't tell you anything that will help. Most of my data sources I've worked with have been internally sourced log files, messages from ZMQ, or CSV data - you might be working with something far different though, since there are lots of public data sets and such which are common. Ideally, this would be integrated into the tools that you are using to clean the data, but I don't know if that exists.
Handling input from many different sources at different rates is not a very hard problem to solve if your system is build correctly - you could for example run a daemon for each data source which will populate the database when there is new data available, then send a message off to the processing engine, which will integrate the data into whatever reports you are running.
Specifically for a use case of a hedge fund, the reports could be triggered by a message which is sent when the new data is available, and processing could be done in parallel in Lambda or similar dependent on need to get a nearly instant return, enabling nearly real-time reporting.
I have seen local companies working for months/years to finally use his BI package but the trouble at the step 1 is big (and also, to put the data in a "nice" schema).
The problem is that enter in this space is hard. Years ago I was at a company that have a niche product (in foxpro) for this kind of task, and I have dreamed about build something like this based in my experience, but get the funding for this kind of "boring" task is hard (more in my country, Colombia).
P.D: If wanna help, we can talk. I can't give a magical solution but at least I find this kind of "boring" jobs compelling ;)
What I envision a solution to be like, would be something like an configurable/codeable OpenRefine (was Google Refine) with streaming ingestion/extraction, with a validation engine/parsing engine (something more elegant than regex, but you can drop into that if necessary) and maybe a pluggable event processor (i.e. a Spark or Flink). I would love to work on such a problem, and solve it.
Great questions. I worked on similar problems in the weather/ag space for a few years, trying to minimize the time between data was acquired and data is ready to inform a decision.
We threw every rule out the window in the name of performance _when fetching raw data from external sources_. So we had weather station networks, NOAA forecast runs and NASA satellite data in a workable schema in our shop way faster than average. Mix of C, PowerShell, Perl, and the nonstandard parts of T-SQL, highly parallelized, tricky but fast.
After the "workable schema" was established, the rules came back and we acted more responsibly. Smart instead of clever.
Ran this stuff all day long, getting every piece of data asap. Things that can only be calculated with a full day of data we poked and prodded the meteorologists to express in "partial aggregates", which to me were just like the map steps before an EOD reduce.
Took a lot of mutual understanding and iterating but worth it in the end. When the ultimate data source (satellite or radar site for us) posted its last hour of data, we were 95% done with the day's computation work. We do our last step, publish our numbers, and bam, Our ag clients have this stuff a day earlier than they are used to.
It's a native streaming platform so your data will be cleansed, processed, scanned for outliers event-by-event rather than in batches. We have dozens of streaming connectors IT/Enterprise/Web data sources. We also support initial load for your firehose data. For unstructured data, we have support for RegEx based parsers.
Shoot me a message if you have any more questions. We have many big name users in Aerospace, Banking, Device manufacturing, and Logistics industries.
Interesting. We worked in this area for many years, did the 'startup' in this area, now acquired by Intel, open sourced the product (ob plug: https://github.com/01org/hyperscan ) and I would never claim that we are anything near 'solving' regex.
More like "mitigating" or "occasionally getting regex slightly more right than some other solutions". There are many different approaches to regex and all seem to focus on different parts of functionality (RE2 focuses on quick compiles and simplicity, libpcre has 'all the functionality', we're about streaming + large scale + high performance if you can tolerate long compiles and lots of complexity). A number of new projects are trying very interesting approaches, like icgrep and the Rust regex guys.
I've been working on similar issues, and developing towards solving them. I work with mixed schemas, handle user defined processes inline and allow you to gather stats and show everything as it streams + accommodate a workflow. I'm only 1 month from launching my beta and want to give away licenses to people who can use it and give me feedback. here is the landing page for it: http://ohm.ai OH and it uploads nothing... it all runs in your browser anthony dot aragues at gmail dot com if interesed
I don't know much about Paxata but I think Trifacta are well-regarded in industry and academia. Trifacta founders worked on research / open-source-? project Data Wrangler http://vis.stanford.edu/wrangler/ and turned it into Trifacta.
One thing you don't really mention here, but it's mentioned a lot in the comments, is the data extraction piece. Is data extraction a pretty solved problem at this point, and it's really the intelligent cleaning, transforming, then warehousing / analysis that's the unsolved issue?
I've actually been working on something like this for a while now, and found your comment about proprietary data interesting. Would this mean that hosting this data in a third party server is out of the question for you? OK with NDA?
In my current job I use Informatica Cloud, which either by itself or in combination with other Informatica products can do these things. I have two main complaints about it:
1. The UX is subpar. It insists on running in only a single tab at a time, and attempts to open multiple tabs will instead override whatever it considers to be the master tab. This is a huge pain, because I often need to have a mapping workflow open in one window and some other relevant part of the application open in another. Instead I have to save, go find the thing I want, and go back. Another problem is that when working with data sources containing tons of fields, there's no easy way to search.
2. It offers an expression language to perform some computational tasks, similar to what you'd find in Excel, but it's hamstrung by a poor UI and a limited amount of functions. The built-in editor for expressions is really poor (see Tableau for an example of a great editor for a simple Excel-like language; it even has type linting) and, unless I've misunderstood something, you can't declare any variables so you end up with huge nested expressions. There aren't many functions available, so something as simple as removing whitespace ends up as lstrip(rstrip(foo)). In combination with no support for statements (or at least a let expression like in lisp) this makes any nontrivial data munging completely indecipherable.
I've looked around in this space and it seems like there are a variety of products, but the supplier of our main CRM will only support Informatica Cloud. I think that a company that can offer a product that does what you've said but makes a serious effort at UX could cause users to revolt and demand to use it! I know the joke is that Slack is just a pretty IRC with better UX... but that's exactly why it has become so successful.
In terms of data munging, take a look at Microsoft's Power BI. It's visualization software but it has a nice data munging mode that, crucially, keeps track of all the changes you make and displays them in a linear format. This is great for getting a quick idea as to what was done with the data and is essential for doing reproducible data analyses. Unfortunately, Power BI also suffers from poor UX in insisting on tiny fonts and gray-on-gray palettes that are totally unreadable to anyone over 30.
Have you looked into Snowflake? Seems like their solution satisfies most of your requirements, including native ingestion of unstructured data. The one caveat is that all source data must first be loaded to S3. https://www.snowflake.net/
Check out Holistics.io (disclaimer: I'm a cofounder). While I can't say we can solve all your listed problems above, we provide enough tooling on top of your DW to help you pre-process (clean, aggregate) them.
Omg, so many sales pitches. You should figure out which of those were automatically generated by someone who's bot is crawling HN and using NLP to find posts like this, and then hire them. There's basically 0 chance that isn't happening...
Chollida. I have been working with a startup in Seattle tackling these very issues. Super great team and great software, please get in touch and try it out! datablade.io
On a smaller scale my EasyMorph might be of help. It's a lightweight ETL and you can do with it way more than with data preparation tools http://easymorph.com
I tried to evangelize THIS very problem with my previous company (a somewhat successful managed data infra service) for a year, it was AMAZINGLY difficult to make my executives even understand the problem and its magnitude.
Take a look at Apache Merton's architecture. They seem to have dealt with a number of inherent issues in the problem putting together data processing pipelines of open source components and ensuring they work together. The project is active but incubating right now.
As a structural engineer, I see a good opportunity to make reinforced concrete design software available in a SaaS format. The competition is outdated, clunky, requires local installation and messing about with licenses. Design firms are paying $1000-$3000/year per user/seat for what amounts to a pretty basic app.
Unfortunately, there are very few people that understand both computer science and structural engineering.
In the US there are about 281,400 civil engineers [1]. I couldn't find more detailed information on structural engineers.
-Assume about 10% are practicing structural engineers who need to design concrete structures = 28140.
-Assume a company wants 1 license for every 2 engineers = 14070. (I base this off the fact that my company has 6 licenses for 12 engineers, but we may be higher than average)
-Assume we could get 10% market share = 1407 subscribers.
-Assume $1000/subscriber/year = $1,407,000 from the US market
I think this is an overoptimistic view of the size of the market. The field is highly fragmented. A large fraction of structural engineers are contractors or work for smaller firms (think wood design, rather than concrete/steel) and wouldn't be target customers for section analysis and concrete detailing software. An absolute majority of those that work for the larger firms don't require anything more than Excel spreadsheets. From my experience, a typical structural engineer is a rather savvy computer user, often times writing Excel macros or AutoLISP scripts to automate their tasks.
A large not-going-name-it software package for modeling steel and concrete structures creates alone over 100M a year. It hardly dominates the market so depending on how the market is segmented a good estimate is probably any number between 1 and 10 times this.
Single seat licences are not the only revenue model. Once a product gains traction consulting, training and providing VIP helpdesk and bugfixing services factor in as well.
Tekla Structures has dedicated reinforcement concrete design features. The licence cost is probably closer to 10k per seat.
You need computational geometry, computer graphics, and structural engineering expert level domain knowledge to implement anything. You need to create traditional 2D machine/construction design drawings from the 3D models. Then you need to sell it to corporations, whose work, most of all, must be dependable and free of guess work.
You need to know what sort of geometries you can use to model the reinforcements. Then you need to know how to design the system so it can handle very large amounts of geometry.
The worst of all is you need to deal with god awful industry standard formats- DWG, DGN, IFC, Step/Iges and so on. Maybe DWG import and export first.
To have any real chance you need a guy or two who are good with numerical code, someone who is familiar with e.g. Game engines, soemone who knows computer graphics, a structural engineer to tell how he does his job and what the thousand inconsistencies in the field are (this is not a trivial domain like housing or transport), a sales/marketing guy to connect and push the product.
And, like someone else estimated, the potential market is not gigantic - which is kinda funny because we all depend on reinforced concrete but don't need so many engineers for the design work...
> I was more leaning towards member design software, such as spColumn and S-Concrete.
The utility of your software tools will be very limited if you are restricting yourself to only member design instead of total structure solutions like ETABS. Why should engineer pay you at all if they can use spreadsheet for free to do what you do with your SaaS?
> No one I know is using the automated concrete design built into analysis programs like ETABS, Tekla, etc.
Not too sure about this because I know quite a lot of people who are using these tools. Any reason why the people you know don't use ETABS or Tekla?
> Why should engineer pay you at all if they can use spreadsheet for free to do what you do with your SaaS?
Why do businesses invest in new tech? Why pay for excel when I can use a pen and calculator? The answer is because it makes them more efficient. We have excel sheets to do the same thing, matlab code to do the same thing, and yet here we are paying for these member design tools because they are the most efficient for us. If you save an engineer even a couple of minutes for each element they are designing, you essentially pay for the software.
>Any reason why the people you know don't use ETABS or Tekla?
We do use ETABS extensively for analysis. We don't use it for design. It is foolhardy to trust the automated RC design in these software. That seemed to be the standard of practice around here, but perhaps it is different in other areas of the world.
> It is foolhardy to trust the automated RC design in these software
Do you mind if I ask why? I'm working on a sort of general approach toward designing trustworthy engineering software, and I'm trying to collect as many reasons as possible for "can't trust the software".
Since you can already do the analysis (like ETABS), and you are planning to do individual member design, why put the two and two together and do an automated RC analysis+design software? There is no reason to distrust an automated software anymore than separate analysis+design software.
There absolutely is a reason: seismic design. We end up doing a lot of data manipulation between the FEA stage and member design stage.
Its not a distrust so much as a fundamental flaw. For simple gravity design it works fine, but even then we are using spColumn because its just quicker for us.
Care to explain why you have to manually do lots of data manipulation between FEA and member design? Why not write a software that can automate these whatever data manipulation? Seems to me that an All-In-One software should have no problem doing analysis, the-whatever-data-manipulation, and the member design.
I suspect much engineering software is ripe for innovation. My wife is a Water Resources Engineer- a specialized form of Civil that focuses on "when it rains, where the hell is all this water going to go?".
The software for that kind of modeling is apparently pretty basic, pretty expensive, buggy, etc.
99% Invisible's America's Last Top Model. The Mississippi River Basin model was shut down because the computer models were cheaper and "good enough." They still use physical models today for other projects, albeit on a much smaller scale.
Fluid simulation is a very difficult problem to simulate, structures are a lot simpler.
A friend of mine was an environmental consultant, and went to startup weekend. They successfully made a SaaS app that would spit out an environmental report in minutes instead of days of manual entry. Just a really good example of small applications that have a huge benefit in old industries http://www.enterratech.com/reports/1/
I think that the models are quite advanced and where the money goes.
It does not mean that you just can't take a couple of web developers and make it usable, but as the market is small it might pass the price point with a supersonic bang...
My brother in law would agree with this, he is building an open source project for collecting all of the analysis that civil and structural engineers have to do on every project into a single repository where people can share and augment the calculations. (think if it as github for civil and structural engineers)
I would love to see that as well. I have a project with similar goals (posted above). I would like for users to be able to simply call built in functions to do the normal tedious stuff like stress in rods, beams, etc.
I have a friend who works in construction glass. (It's NYC, that's a big market). They ship millions of dollars of complex, custom molded glass every year. Everything's kept track of by emailing excel spreadsheets.
That's disgusting. I feel filthy. I replaced a system in my first client school that also used excel attachments for tracking student data, exams results, attendance, assignments, etc. I could physically feel better once I saw my own implementation replacing email attachment system. Since then email-excel has been a sensitive topic for me.
I've recently found a department for the company I'm working for needs help as they have literally reached the limits of Excel.
They have a sheet with around 30 rows and 150 columns, and they have 100 of these sheets (in a single Excel file). Some parts use formulas, but usually when somebody needs to change something they need to go through every single sheet. The issue is now when they try to add new data Excel won't let them.
I don't even want to know how they share the file or do backups.
I work in the healthcare industry. Basically we ARE the industry nowadays, and we use excel and word to keep everything "organized". There are some half-ass designed software,websites, and databases that are used as well, but it's amazing how a multi-billion dollar company can rely on this level of technology. I think they get these bids to run state government programs, and have absolutely no plan in place. And for some reason instead of just automating or updating things, the company just throws bodies at the problems and makes everything "production" based. I'm sure a lot of places are similar, but this is a white collar factory on such a massive scale, it literally sickens me. There are so many channels that approval for changes has to go through, that by the time some small minor change is implemented its already way too late, too distorted by having so many hands touch the problem, and too outdated.
Same with the electronics manufacturing industry, where inventories and BOMs are done in Excel. It is a pain in the ass, time consuming and error-prone. But that's what managers want.
Out of curiosity, what do you think of what flux.io is doing? If a SaaS structural engineering application could be integrated with that, would it address your needs?
(For what it's worth, I'm doing something similar in the transport planning space. And yes, bridging the gap between that and modern CS is a substantial piece of work.)
I am working on doing this. The project is called cadwolf. The MVP is up now and I will coming out with a full version in January. If anyone has comments, love to hear them. If you are interested in collaborating, let know too.
A fellow structural engineer here. I think that there's little room for innovation in the "calc" area. The cost of doing calcs is a small fraction of the overall budget of a structural project. Modeling/drawings is where it's at. The analysis/design toolset of a structural engineer (the FEA/design programs) hasn't changed in essence since the 80's outside of drafting/modeling, and for good reason: the marginal cost of an engineer perfecting their analysis exceeds marginal revenue. There's a sweet spot where an experienced structural engineer knows to stop refining their calcs, with the rest of the effort is spent on detailing, which sets apart good structural design from mediocre.
Where I would invest (if I were Autodesk or their competitor) is in releasing CAD tools for free in exchange for a consent to use the designs/details internally for ML purposes. Would love to contribute if anyone is working on such a product.
I might be working on something that you would be interested in. MVP is at www.cadwolf.com. I am a structural engineer with an MS from UT Austin. Website in written with php using Laravel and angular framework for JS.
The plan is to link CAD to the mathematics and then link the finite element to this as well. The system would also function as a sort of github for engineering where users can find and use functions to do most standard analysis. Email is in my profile if anyone is interested in talking.
I checked out cadwolf and as an engineer myself I find it very interesting. I am curious to understand a few points:
- Why do you center everything around documents? Is it more because people are used to it, or do you believe that they are best fit for design-tasks?
- I saw that to update multiple documents after a requirement changes, you need to open them one by one, in the order of their dependencies. Have you tested that this is still a viable approach, once you have thousands of dependencies and multiple users in a complex design?
I really like the equations and how you only allow to make formally correct equations (including units). Anxious to see how this develops.
(full disclosure: I am co-founder of a Software which tries to achieve the same aims using different concepts: www.valispace.com)
What I call "Document" are not files in the sense of a word file. They look like text files because that is what I thought engineers would be comfortable dealing with. However, they function as programs. Documents can be used as programs within other documents as well. Documents fill both the need to solve the calculations and to document them in one place. It eliminates the need to update documentation, have multiple platforms, etc.
There are places for users to upload and store data as well - datasets.
As of now, the code solves equations in javascript within the browser. This is why documents have to be opened when a requirement changes - because I have no server side of the code to solve them without the browser. It isn't a long term solution, merely a step in the building on the platform. My next step is to add a server side code that is capable of solver more complex and larger equations on the server. When that is done, changes to requirements will update documents without the need to open them manually. I plan on using python and there are several large libraries available.
This will allow me to link documents to CAD. When the math changes, the CAD will change as well. Once that is done, I will add a finite element meshing and solution system to create an engineering platform that essentially does everything.
I like your site. It's nice to see other people addressing these problems. I am also an aerospace guy. I worked on the shuttle for a while and then designed some components for the Orion. Shoot me an email if you want to talk more.
I am a fellow engineer (Satellites in my case) and we have been fed up with engineering-tools in general (specially systems engineering), which seems to only consist of Excel-Spreadsheets and document-management systems. Even in the space industry there has been practically no innovation since the 60's more than digitalization of documents.
We are working since 1.5 years with some engineers on a software to solve this: www.valispace.com
I would be curious to hear from you whether what we are building with a focus on the space-industry also applies to structural engineering.
This problem is hard to solve at the root because each MLS is independently owned and heavily influenced by the National Association of Realtors and local governments. As an active RE investor myself, I'd love to work on a problem in this space.
You hit the nail on the head. Having developed with MLS data before, my opinion is there's little incentive for MLSs to make the data easy to work with. As long as each MLS gets their data distributed on the large portals that most consumers use, there's little reward to update (read: spend money on) their systems to implement a modern format to help indie developers and startups. That's the status quo. I don't think anything is going to change unless the large portals band together and spearhead some kind of industry-wide effort to fix this problem.
As an aside, I'm working on a RE investment product that solves challenges similar to this one and I'm looking for RE investors to join me. If you're interested, shoot me an email at hello[at]myname.com.
Ugh. Realtor.ca isn't really due to standardization but monopolization and control. It's a site by CREA (the Canadian Real Estate Assocation) and has exclusive rights to data from all member associations.
disclaimer: I am affiliated with rew.ca (a competitor on the west coast)
The entire Real-Estate industry is ripe for disruption. Exactly why are we paying 6% to "agents"? What are "agents" actually doing? 10+ years ago the need for "Real Estate Agents" was there as information was not readily accessible. Things like "what kind of area is this property in", "what are schools like" "what are prospects for this area 5 years down the road" and so on. All this information (and MUCH more) is currently readily available online. I have purchased and sold 4 homes in the last 6 years and agent I have worked with has made many tens of thousands of dollars and I can tell you with utmost precision that she hasn't spend more than 10 hours total. My wife and I scouted locations, went to open houses, did research online for things that are important to us and finally called the agent and said "we want to make an offer." Other than that the only other time we called is for her to let us into a home if there was no open house soon.
10+ years ago we had other "agents" that made a lot of money, e.g. "Travel Agent" in Travel&Leisure industry. Then internet came along and now we have 1,000's of websites that have replaced what Travel Agents used to do...
For Real-Estate I think MLS is just a start a small piece of the puzzle, someone very smart is going to disrupt the whole industry and make many billions of dollars.
The real estate industry is very heavily regulated and, worse, these regulations are extremely fragmented. Any disruption is going to have to deal with that.
Also, agents for basic residential purchases where the property is about as mundane as it gets provide very little value. However good agents will do proper diligence by inspecting the public records for the property. That part may be disruptable with software and an ambitious enough records maintenance effort. But good agents who do their jobs properly are well worth 6% (and, by the way, that 6% is frequently split two ways, so each agent gets 3%). Of course my bar for "good agent" is much higher than Joe Coldwell or Jane Keller-Williams. "Good" agents know local and state regulations very well and do diligence to inspect the public record and work closely with well qualified inspectors to get the best (lowest for buyer, highest for seller) price they can. Most agents, especially for "big box" brokers do basic humdrum contract paperwork and have a very basic approach to disclosure and negotiation and don't add much value. They're ripe for disruption.
> work closely with well qualified inspectors to get the best (lowest for buyer, highest for seller) price they can.
Kind of an aside; what's the motivation for buyer's agents to get good inspectors or the lowest price possible? They're getting a percentage and they're motivated to make sure the deal falls apart. Although, as a buyer it's not like I knew good inspectors, anyway. The best I could do is read their sample reports and go off of my own (sparse) knowledge of home construction.
Australian here: on my last house sale, I did a deal with the agent where if he got a price above a quite-high threshold that he would get 3% instead of his usual rate. This was seen as a remarkably generous offer, unheard of in the industry.
I've only once seen someone use a buying agent: husband and wife were both hospital doctors with three young active kids and they figured they valued their time enough that they would pay for a real estate agent to short-list properties for them.
I wonder what is so much more difficult about real estate in the USA that it requires so much more in the way of services.
Australian who lives in the US and is looking at buying a house. What's different? Almost everything, starting from the basics like there aren't auctions, you have to submit a secret bid that the seller will read and then tell you if it was accepted or not. Factors that can affect acceptance include what percentage of your purchase will be made with cash, whether you are willing to buy it without bothering to inspect the house first, and how many days it will take for you to close the deal if accepted - and of course, everything else like whether you are the kind of nice quiet couple they think their neighbours want to live next to, whether you have the same name as their brother-in-law, and whether you are the kind of person they want to imagine living in their house once they are gone.
Agreed. There are many potential advertising opportunities to boot.
When people buy a new home and move they typically need to get new utilities, health and car insurance, doctors, dentists, furniture, appliances, etc... They need to decide where they're going to shop for groceries, which restaurants they want to explore, what activities they're going to get into.
Wife's a real estate agent so I have some insight.
Agents mainly do two things:
1. If selling they can put your house into MLS.
2. For buyers they can let you into a house.
Being able to do these things requires NAR membership, a state license, and other fees and cost real estate agents thousands per year. Also driving meetings clients at houses takes gas and time. Also your 6% commisiion is paying for all the other people that the agent spent gas and time on who did not buy a house.
Would you be willing to advertise in the news paper that your house is for sale and leave the doors unlocked so that random people could check it out?
True a buyer could be let into a home by sellers agent and even use the sellers agent to purchase the home. Agents love this! Its called getting both ends of the deal. It doubles the agents commission. Does not benefit buyer at though and actually is a bad idea.
Yeah, even with a buyers agent, the buyer doesn't get much. Both agents (if being paid out of the 6% commission) have a fiduciary responsibility to the seller.
About the only thing a buyer gets out of the whole deal is a title search to make sure there are no outstanding liens on the property - and they have to pay for that themselves!
Its worth what agents are willing to do it for. If you cant find a real estate agent willing to help you sell your house for 0.5% then it is not worth it.
Opendoor is already doing agentless open houses with access codes (smart locks) and cameras throughout the house where people can visit whenever they want without calling ahead, not that difficult
That seems to me to be fraught with all manner of peril. Suppose a burglar poses as a buyer under an assumed identity, and then robs the house while wearing a mask. What is the recourse?
Apparently Opendoor literally buys (and fixes up) the house and then re-sells it, rather than simply arranging the transaction, so they're selling an empty house.
Buyers have to go through an identity verification stage to gain access to the homes, including, I believe, the taking of a credit card. Not impossible to spoof, but far more frustrating than merely downloading an app, clicking "I agree" and getting the unlock code to burgle.
I'm not sure I understand what's stopping the burglar today? 50 year old Aunt May with the bad hip who has been selling houses for 25 years? I'd also expect the person to be checked out ahead of time similar to Airbnb or other services.
The house unlocks to a tracked verified identity from a smart phone, Not to random people in masks, would be my guess. Also empty houses are hard to steal only vandalize
I guess I'm asking how the identity is verified. It's not hard to spoof an identity on a smart phone. And empty houses are full of copper wire and electrical appliances (and in this case, apparently, security cameras) which are commonly stolen from construction sites (and cost a lot more than a smart phone nowadays).
The TLDR on this piece is that since commission is a fraction of sale price, the primary incentive for agents is to just get a sale, and fast, rather than maximise sale price, as even tens of thousands extra adds only a little to their cut, so their time is better spent pushing the property through the sales pipeline and moving on to the next one as quickly as possible.
There's a study which draws similar conclusions:
> Our central finding is that, when listings are not tied to brokerage services, a seller's use of a broker reduces the selling price of the typical home by 5.9 to 7.7 percent, which indicates that agency costs exceed the advantages of brokers' knowledge and expertise by a wide margin.
The entire Real-Estate industry is ripe for disruption.
Exactly why are we paying 6% to "agents"?
https://www.redfin.com/ is already doing this. They charge half the normal fee if you're selling and they give about half of their fee to you after purchase if you're buying.
Sure, but what you are paying 1/2 for? Imagine owning a home which sells for $1,000,000. Currently you pay roughly 6% - $60,000. You pick Redfin - they charge take the money and give you back 1/2 but you are still paying them $30,000!!
But if your house is sold for $100,000 you'd pay $6,000 or $3,000 to Redfin. The whole thing is just the biggest currently running scam that I can think of and someone will come along and totally disrupt this entire industry of scammers :-)
Exactly. Does it take ten times as much work for a real estate agent to sell a $1m house versus a $100k house? Obviously not. So what are we paying ten times as much for in the one case, and if there's no connection between effort and fee, what are we paying for, and why, even in the low end case? Totally a scam, totally ripe for disruption.
The level of service is pretty low too. Its basically handling setting up a few appointments to see houses, helping grease the skids for inspections, and setting up a closing with a settlement company/lawyer. Stuff that really doesn't take all that much expertise/knowledge.
I'm sure there are little things that go on behind the magic curtain, but definitely something that could be automated in a b2b fashion.
The worst part of it all - all that money is spent basically to protect the seller so they can wash their hands of the whole deal once the house is sold. There is very little to no buyer protection in the real estate world.
Sadly your commission on that million dollar house is actually $45k -- Redfin cuts their seller's agent commission in half (3% becomes 1.5%) but the other half goes to the buyer's agent, so you pay 4.5% rather than 6%.
Redfin isn't disrupting the industry. They set out to, and ended up pretty much recreating the existing model and competing on price.
"We’re full-service, local agents who get to know you over coffee and on home tours, and we use online tools to make you smarter and faster. More than 10,000 customers buy or sell a home with us each year."
That's pretty much what any brokerage is. The only difference is they pay their people salaries. Any agent that can actually substantially sell and earn would strike out on their own and get their brokers license.
We are actually in the process of building out a platform that gives home sellers the power to sell their home with the support, guidance, and advice of an agent without the fees. Feedback is welcome.
In less that 30 seconds, you can begin the process of listing your home with no contract or fees and includes MLS syndication.
The app auto-dispatches photography, drop-ships signs and lock-boxes, and starts the documentation management process.
At anytime you can reach out to your agent via phone, chat, or email.
When offers come in, they are displayed in "easy to understand" language and you can review then with you agent/listing account manager anytime.
When closing time comes, we dispatch a notary to the home and you exchange everything at the home. Super easy. Super convenient.
How we make money:
We retain a small fee at closing. When you go to buy your next home, we rebate that fee back to you when you use one of our vetted affiliate agents.
It is a win win. We keep the small fee and the agent "rebate" is essentially the acquisition fee for the guaranteed buyer. Agents don't waste their time working leads that will most likely not close (which is part of the reason why big commissions exist) and home sellers save thousands of dollars.
We operate in Colorado right now using brute force but V1 of our app will be released in January of 2017. We will be servicing Washington, Colorado, and Minnesota as of Februaury.
The entire Real-Estate industry is ripe for disruption. Exactly why are we paying 6% to "agents"? What are "agents" actually doing? ...
I basically agree, but I will say that when we bought our house and the (FSBO) seller turned out to be a flake and/or an outright nutcase, the realtor we were working with earned every penny of his 3%. I won't go into what he did for us because some of it may not necessarily have been legal for someone who's not an attorney to perform. Let's just say it took a lot of hustle to hold the seller to his commitments and make the deal happen according to the contract... especially when some of the flakiness turned out to be our own fault, due to the (in)actions of our mortgage lender.
Most of the other realtors I've dealt with fit your description perfectly. Nowadays, my thinking is that they're like airline pilots -- most of the time you can take them for granted and even think about how to replace them with automated processes, but when things start to go wrong, you may find yourself appreciating their work more than you thought you would.
I think a lot of the value in real estate agents is a motivated party to make sure the deal doesn't fall apart. Buying a house is emotional for both sides and petty things can cause people to walk away.
As a Brit I don't get the agent thing for buying and selling houses in the U.S. As I understand it's basically required for the buyer and seller to have an agent.
Here you can sell a house privately, and the only fees you need to pay are the solicitor for conceyancing - the fees are around the same when buying or selling, between £500 - £2000. They will just do the paperwork, check there aren't any issues with the deeds, and do minor contract negotiations, and register the sale.
The purpose of an agent here is really just to have a shop where they can advertise your sale, take clients and guide them around a house (but TBH I'd rather just look myself without an agent as they don't add any value in my experience), and do a bit of management between the buyer and seller. For this they'll usually charge 0.5% - 3% (to the seller), but if you want to do it yourself you can just advertise on sites such as Zoopla.
As someone who grew up in a family of real estate investors, landlords and brokers and has 14 years experience as an adult with it, I have to say, and I'm not picking on you because what you are espousing is a widespread belief, but you are wrong.
You are wrong about the amount of time she spent, and wrong about the industry being ripe for disruption. Spending 10 hours on four deals is not physically possible, or you have the luckiest, worst realtor in the history of the broker business.
Many have said the same thing about the industry, where are the results? Realtors are not still in business because of information asymmetry. Redfin has tried, and they have not succeeded. In fact they ended up moving towards the historical model. That tells you something.
I'll let you in on a secret, you can sell and buy any house you own or want yourself! It's called For Sale By Owner and many people do it. You can also negotiate on commission.
But be careful, because there are a lot of laws in place in RE, to protect buyers. It is heavily regulated. You could make a mistake and fail to disclose something that could end up costing you HUUUUGEEE.
What you get with a good realtor is a ton of service. Just the other day I drove 25 minutes each way to go make sure a homes thermostat was on and the house was warm when the temperature dropped. I was doing a favor for a realtor I know, that is selling a house for a family in another part of the country.
A good realtor will do things like that for you. And it's often a thankless job. The house I did that for might not even end up selling, the realtor may not even get a commission and it ends up being a lot of free work.
A good realtor knows attorneys, handy men, plumbers, electricians, landscapers and is a tough negotiator. A good realtor will market your property to a wider audience than would be easy for you to do yourself.
There are a lot of things that can go wrong and disrupt a sale, often times at the last minute. Those connections can save a deal. I've seen it countless times.
The reason the real estate broker business hasn't been disrupted is because it's a service business, it's not a product and its difficult to reduce it to something that scales and can be automated.
There is a large graveyard of startups that have tried.
I gave up my real estate license because of the sentiment you are expressing here, and the amount of competition that was willing to put up with it. It's a crap job in my opinion, it benefits the customers much more than the agents in all but the really hot markets.
"Spending 10 hours on four deals is not physically possible, or you have the luckiest, worst realtor in the history of the broker business."
In lots of cases there simple isn't anything to do. I saw the house, did my own research on it, went to see it, did more research, did more research and then called my Realtor and said "Make an offer." Is that worth $43,000? Why would she make $43,000 for few hours of work? I bought two houses this way, my first and last call to my Realtor (personal friend of mine) was "Here's our offer on the house, send it in." One offer was immediately accepted, the other one went back and forth 3 times over two days.
The simple fact that charging based on percentage of the sale is theft, there is no other way to put it. There is one other "entity" that does this and this is Government with Sales Taxes (also theft :-) ). Can you tell me one good reason why Real Estate Agents do not simply charge by the hour?
For anyone wondering, this is not a technical problem. It is a political/business problem.
It's a solved problem to make a system that stores information about home listings and transactions. Other countries have done it. Yeah, houses are annoying because of all the weird combinations. But the thing is just a CRUD app.
This system still exists because of organizations that are based on information asymmetry. A small group of experienced people could make a really, really nice platform for sharing real estate data in a month and it would never get anywhere.
The biggest one is that realtors exist entirely because of regulation a the state level.
After that is the extremely entrenched business interest in the status quo - a lot of people are making money off the current system, and any real disruption is going to cost 95-99% of the real estate industry their jobs.
As noted elsewhere, the level of regulation is such that even offering better data to buyers/sellers has a lot of regulatory overhead, including physical offices with real people.
Some friends of mine started a wordpress company a few years ago, and after some thrashing trying to find product/market fit, they ended up pursuing real estate SEO, MLS indexing, and easy wordpress + mobile app generation. Check out https://wovax.com/
Good luck. I used to work at a Startup that tried wrangling MLS data. There is the not-so-trivial problem of gathering all of the data together into one schema. But to offer services on MLS data (like a forward facing website) you need to have a physical office in the MLS region your providing. This problem seems to be more legal than anything.
Oh, this a million times. In fact I have a huge pile of cash waiting for someone to fix this.
Right now I'm only pulling from a couple of different RETS services but already it's hellish. Regular short term schema changes, terrible lag in pulling data (10+ minutes to pull 30 listings!), low API limits (cut off for pulling image URL's too fast but they won't give a limit - just saying "if we feel it's too much", so pulling images lags for hours as I have to be extra careful. Also no option to pay extra to speed up). That's before you get to matching up fields across multiple MLS's or dealing with weird crossover effects (MLS X cannot be shown in Y from RETS #1 - buts that fine in RETS #2).
In fact, I will give you all my money right now anyway, I'm going to go live in the hills far away from it all.
Not my industry, but a friend of mine in law was discussing how incredible her in-house software is for managing billable hours relative to all her past companies. I poked around, and most law firms, even very deep pocketed ones, use somewhere between a bad tech system and no tech system to manage and track their work.
A small team could easily collaborate with some law firm (maybe take an investment from a few law firms), and create some very valuable software.
I'm a lawyer and a programmer and run a legal-tech startup. The largest issue I've found with lawyers and technology is that they don't understand it, and there is a lot of fear around it.
Most of the time I spent selling my software was assuaging the risk-averse mindset that exists in the profession, rather than advertising the benefits. I actually gave a TEDx talk on this point as I think it's an enormous issue in the industry.
So the ease of creating the software isn't the issue, it's the culture. That said, legal tech now has a pretty healthy ecosystem which is a delight to see!
I think it is also partly because any inefficiency in work leads to an increase in the Billable Hour. When it makes you more money it isn't really broken.
Spot on! The problem goes past efficiency and is related to the broken pricing model. The financial crisis actually caused this to change a bit as more and more clients were after fixed costs. I hope that trend continues.
You're being facetious but maybe you should actually consider the history behind that sort of thing.
Yesterday my friend in LA invited a few of us in Toronto to come to his BBQ this weekend. The group chat became us discussing the logistics and price of how we might do that. 150 years ago, that would have been an insane concept.
300 years ago, when Benjamin Franklin traveled from Philadelphia to Boston, the trip took two weeks. The fastest mode of travel was by boat, and he nearly died in a boat wreck.
Not sure how much startup potential there is, but in my industry we struggle with adhering to procedures, and the root cause of that is because our procedures are written in the dark by management that doesn't always have a complete picture or much insight into what they're enforcing. The problem that creates is that employees only look for work-around's for the procedures (because they are misguided or misinformed most of the time) and it creates a bigger mess than if we had no procedures at all.
It's almost as though if we had a Procedural Consulting firm that could come in, look at the big picture, and help companies to create EFFICIENT and WORTHWHILE procedures that ACTUALLY do justice to the customer requirements without breaking the bank. Then they can sit everyone down and explain the procedures and enforce the assimilation that most companies usually have growing pains with.
I noticed that when moving to a new company that is trying to grow and achieve higher levels of accreditation. Management had intended a two-way assimilation to take place between my procedural knowledge and the procedures they already had in-place. What happened instead is old procedures are etched in stone and new ones are seen as an obstacle... The assimilation was one-way and the other middle-managers like myself are mainly concerned with keeping everything the same, despite there being improvements that could easily be made.
There are probably a fair number of consulting firms you can find for that.
But another (part of the) solution is using workflow software. Something that allows you to define and share procedures, indicate status of various taskings along the defined workflows, move data and such along. This clarifies to the employees what's needed, and gives an opportunity for them to provide feedback for improvements (too much granularity, not enough, too inflexible, etc.). Management, then, also gets better insight into the actual status of projects and taskings. It's part of continuous improvement to constantly be evaluating these sorts of procedures and clarifying, culling, adding to them. Management that's unfamiliar with these concepts is just bad management.
In the manufacturing world, the Procedural Consulting firm would conduct "lean events". (NB: Many, if not most, places do this incorrectly.)
The correct way is to collect metrics (meaningful) over time, encourage and reward employee feedback about ways to improve things. Then improve them (sometimes as a move across the board, but often by conducting experiments to see how well the new concepts work, then roll out to everyone). The risk is that improved efficiency will obsolete employees, some fear this (like may have actual panic attacks about this level of fear). Instead, you need a culture that sees this improved efficiency as an opportunity for growth (the same number of employees can now do more once people get retasked to new things).
I'm not sure what industry you work in or what size company, but my old department had a similar situation.
It sounds like your processes are either immature (no one really knows what the best way to do things is) or tribal (you have to know the right guy to talk to to find the information or knowledge you need).
Premature optimization is god awful, because it locks you into a world of pain - your business is inefficient and the people actually trying to follow the procedure find it incredibly painful and limiting.
If your process is immature, you need to invest some time and money into finding out what the best way to do whatever it is you're doing. If your process is tribal, you need to find the guy or guys that understand what's happening and pick their brains about everything they know. Their implicit knowledge needs to become explicit. In my situation, I found that just agreeing on common definitions was a huge help.
If you're grossly inefficient, don't write a procedure for a bad process...you're just ingraining inefficiency. Once you've found a good process, then write the procedure so you can train your people to repeat good, efficient practice.
And maybe most importantly, train your people to the procedure but make sure they understand it on a level so they know where it doesn't apply and give them authority to deviate from the procedure when necessary.
Sorry if none of that applies to you. I'm just speaking from my few limited experiences.
These questions pop up every now and then, and while I get the intent, you're not gonna get anything useful out of it. The idea that someone who knows how to code can disrupt an industry that they are not a part of is disingenuous, and the examples that you can find are exceptions, not the rule. It's also extremely naive and presumptuous...what makes you think people in the industry haven't already tried? People who fall ill to this delusion end up in one of two categories: those that attack easy problems with tiny markets, and those that attack hard problems and spend decades learning about and becoming a part of the industry before they solve them.
As someone in the supply chain and logistics industry, I can list for you hundreds of people that know the traveling salesman problem and precisely why its not applicable to their situation. I know hundreds of people that already know how to better manage their safety stock than someone who suggests using Gaussian demand models. I know hundreds of people who can optimize last mile delivery costs orders of magnitude better than a drone engineer. I know hundreds of people who can manage inventory distribution and ordering automation better than someone who knows databases.
And sure, there are companies out there that are doing everything ass backwards and could use some help, even if it is primitive and simplistic. And when they decide to look for it, who are they gonna choose: the guy who saved Amazon $500M/year with their truck load optimization expertise, or you, with your shiny website and a trick you learned from a textbook?
So as a piece of advice, if you aren't part of the industry already, don't try to do B2B in that industry. B2C is fine, because as a consumer you are ostensibly a part of the industry...but B2B is a death march.
I'm sorry, but this is rude, discouraging, and flat out wrong. This sort of stuff happens all the time. I personally work for a company whose founders implanted them in an incredibly calcified industry despite no industry experience, and I can rattle off dozens of successful examples of this.
In fact, this very attitude is the exact reason why there have been industries that are primed for "disruption." If only those within the industry, burdened with preconceived notions and patterns of thought, try to improve an industry, you are very rarely going to see revolutionary change.
You don't need to "disrupt" anything to be a very good business, but I really resent this "leave it to the pros" attitude.
>I'm sorry, but this is rude, discouraging, and flat out wrong.
I'd rather people feel slighted than to have a bunch of kids fresh out of college trying to disrupt, say, the health insurance industry.
This isn't about "fresh eyes". The problem with having no domain knowledge is that you can't identify many of the issues that needs solving within an industry. Every job I've ever had, I see issues that I can solve with my experience.
I'd argue the opposite. Too much domain knowledge leads to creation of the very same "solutions" that have existed with minor tweaks. See Henry Ford faster horse for reference.
This just as wrong as anything can be and you have nothing to back it up. Furthermore you seem to have missed why they are asked. The problems in many industries er hidden problems which can only be understood by those in the industry, but the solutions are often made by people outside the industries in partnership with insighters.
So they are imensly useful and many time more relevant than most other discussions about building companies.
I probably shit on the idea more than I needed to, simply because it's entirely possible that you could be looking to form a team and rally around a serious problem, and that team could be composed of people who deeply understand the problem in addition to fresh faces with a different approach. And I would fully support that approach.
Lately I've been approached by a lot of "startups" who have made it inadvertently apparent that they know about a serious problem in the industry but have no experience in the industry, are taking absolutely idiotic and long-discredited approaches to solving the problem, and don't even speak enough industry lingo to be understood. These are the people I tried to address with my comment.
If you'll notice, I was actually the top commenter on your original discussion [0]. I still stand by everything I've said in that comment. It is a potential multi-billion dollar idea, and the market size can be verified by pretty much anybody with basic market research experience. And while I have plenty of valuable and marketable experience in the Supply Chain and Logistics industry, even I am well short of qualified to lead a company trying to tackle that problem. I would definitely consider being a part of a team that was formed to solve the problem, but here's the thing: If you learned about the idea from an Ask HN thread, I almost certainly wouldn't join your team. Maybe if Elon Musk vouched for you and Marc Andreesen was throwing money at you. Maybe if you and 9 other committed engineers had a collective 100 man years of proven experience solving hard problems across a variety of industries. But trying to be the leader of a team trying to solve a problem that you didn't know about and don't fully understand is a huge obstacle to overcome. And I think it is perfectly reasonable of me to try to dissuade the average HN reader from even trying.
I think this is were you are missing the bigger picture though.
The solutions to the problem in your industry might not be based on the skills or insights of your industry.
It might be that someone recognizes the similarity from another industry where the problem was solved in a specific way or that the insight just so happens to be solvable based on the knowledge of some specific technology.
And it's NOT about ideas it's about recognizing the real problem that hides underneath and which requires experience to understand.
I think both of you are talking past each other here.
The crux of your argument is that an outsider may be able to pattern-match better than industry insiders/domain experts. In other words, an outsider can recognize a potential solution by translating his/her experience with an efficient solution from a different industry.
He is saying that you need to properly understand things from the perspective of an industry insider before you declare that your solution is disruptive. In other words, you need to understand why things are the way they are before talking of reformation. This is the central lesson behind Chesterton's famous "Taking a Fence Down" [0] quote.
The idea that someone who knows how to code can disrupt an industry that they are not a part of is disingenuous
Uh, no. The idea is that you identify the problem, build the team to solve it, build something quick that can prove that you can solve it in a better way than others, iterate to the point that CAC < LTV, scale.
How can you solve a problem that you can't adequately analyze or characterize? How will you know if you aren't trying a failed solution if you don't know what has been tried? How will you know if your simplifications and assumptions are realistic if you haven't seen how they've played out in the past?
Often they don't have the tech skills or desire/will to start something themselves, but are happy to join you if they recognize that it's a big enough problem.
If your expertise is organizational and can put together the right team with those that do have the domain knowledge, I'll concede the point. But domain knowledge is irreplaceable if you actually want to solve hard problems with scalable markets.
I sort of agree. But it is also true that there are many industries and groups that are way behind the state of the art in technology. That may be for good reason, or it may be because it hasn't worked for them yet, or may be because it can't work, or because they avoid tech, or...umpteen reasons.
So yes, a CS grad probably wont come along and revolutionise the Supply Chain industry, but mainly because SC has been at the forefront of areas of tech for decades. A CS grad may, however, come along and revolutionise...well just look at all of the recent famous tech companies...communications, travel, social, arts and crafts...whatever. Many areas are ripe for fresh eyes and new approaches.
I do agree that domain knowledge is often massively under-emphasised though.
before reading the comments i would have agreed but man.. there really are a lot of juicy suggestions.. not just trivial ideas like another travel app.
Rapid generation of high quality 3D models of existing objects. Process should be independent of object size eg. a coke can should use the same process as a car and process time should scale with object size.
Think somewhere on the order of 10,000 models per day throughput.
There's $BNs waiting for you. It's ridiculously hard.
Anyone who does work with 3d content: visual effects, video games, vr, ar, etc. Being able to quickly build your scenes from a huge library or accurate models would be amazing and would save businesses lots of money.
There are multiple but mine specifically is AR. It's valuable also for VR, 3D space planning for Designers/Architects/Engineers, Assets for Game Dev, Objects for modeling and simulation, Training Deep Vision nets and on and on...
Product development engineer here. In the early stages of a project it can be useful to have CAD models of a competitor's product when analyzing how to improve upon them. Recently we had an intern reverse engineer a competitor's product, and we've used some of these CAD models as the basis for our new designs.
Yes there have been a million attempts since the early days of 3D. Most of them are photogrammetry or structured light setups of some kind, that aren't fast enough and don't scale for sizing. Part of it is logistics of getting objects through a scanner, with the accuracy being poor or muddy at best.
IMO it should be done with a mixture image segmentation and procedural generation.
See other responses for the markets it would serve. Use 3D modeling by hand pricing as a comp (Low end $10/model, average in the $50-100/model range, sky's the limit for super HQ stuff).
Not sure what kind of datasets you're looking for. You'll see actual products to test with.
That makes sense. I think there's opportunity for generative ML to eventually help here. An open dataset of (images, description) -> 3d model would go a long way. Check out this paper on using GANs to generate voxel-based models: http://3dgan.csail.mit.edu/
I've been studying and working with GANs for about a year now. They are still very exciting, and I'd love to try to expand my codebase to new types of data.
Additionally, there are some recent techniques that haven't been tried with voxel-based renderings.
Perhaps there is another algorithm that can help go from voxel -> polygons as well.
I think with the right tech, time, and execution this could be a matter of:
Can you clarify what you mean by Procedural Generator? Isn't a generative model already a procedural generator? Its just that a model generated in case of the referred paper is voxel based. Did you mean, generate parameters of a pre-specified model e.g https://graphics.ethz.ch/~edibra/Publications/HS-Nets%20-%20... , although this paper is just learning a regression to the human body model (not using GANs).
Curious to know more about your train of thought. I am working as a researcher in the domain and thinking of experimenting with GANs for 3D model estimation using similar inputs as the one in the paper I referred to.
What kind of structural integrity do you have in mind? Something with the density of industrial packing foam? I've seen set pieces constructed / carved from such material and it can be painted quite well. Putting aside the environmental / toxicity concerns for a moment regarding the type of material to be used, I'm genuinely curious how "rigid" such pieces might need to be.
Oh, so you mean like a big box that could fit XYZ items inside it and capture something like 10,000 per day? I mean, to me it's kind of hard to believe nobody's tried making a "conveyor belt" like process inside a closed system (a shipping container?) with the right optics and resolution to pull it off. Fidelity plus speed plus software consistency. Considering what I saw the gaming industry doing with static models about 10 years ago I kind of thought it'd be a lot further along now, but I guess not. Sounds like a good project for a few Rensselaer Polytechnic Institute grads that otherwise would've been destined for Kodak.
Oh, so you mean like a big box that could fit XYZ items inside it and capture something like 10,000 per day?
Maybe but I actually think that's the wrong approach.
I mean, to me it's kind of hard to believe nobody's tried making a "conveyor belt" like process inside a closed system
Yea they have - kinda. None of it works well or fast enough though. We put up a patent for one a year ago before I thought there was a better way to do it. The manpower required to move items onto/off of a line is a big part of the problem.
Well yeah the human element is exactly what I'd want to eliminate as much as possible; I'm thinking of it more along the lines of what I've seen on How It's Made: Dream Cars in the sense that to get the various layers you're going to want - basic dimensions, surface features, coloration, reflective properties - aren't going to happen in one quick grab I don't think, and I get the feeling the process would work best in "absolute darkness" and isolated as much as possible for vibration.
Taking that 10k number - assuming disparate types of items that might be part of a series like "Bathroom" (toothbrush, hair brush, toilet brush, plunger) - in 24 hours that means cycling each item through in about 8 seconds. The only way I remotely see that possible is essentially having a robot hand pick up the item at the entry point, hold it for the capture sequence (perhaps have a custom-designed 'mount' that can allow for true 360 via a couple positions), and then drop it out the other side.
It's the scale part I'm wondering about, re: one size machine fits all doesn't seem to make sense. One machine for items under a certain dimension (e.g. "hand held") then another for items where the machine has to essentially have super-powers to pick up and rotate objects to complete the imaging process (e.g. a couch, a dresser, a motorcycle, etc). I think trying too hard to accommodate outliers ends up tainting the balance of operations a little? Just thinking out loud, really cool puzzle.
Not really. I know some folks on the RS team and they don't really develop around applications, they are more focused on miniaturizing and making RS more available and lower power.
That said, some people have tried to use RS for this problem, but from what I've seen end up just using Kinects.
Beyond just the 3D: Do you think that capturing the full appearance properties (BRDF etc.) of the object would be useful? This would allow users to very accurately render objects.
This might already exist depending on your exact requirements and it's a fairly common technology in the world of metrology. I regularly work with manufacturers to reverse engineer and/or measure molds, jigs, and fixtures for which there are no drawings available. A ROMER or FARO arm with a laser scanning head outputting its point cloud to a software like the PolyWorks suite can generate an incredibly accurate CAD model of incredibly large parts in a very short amount of time (an hour or two at worst if the mesh needs a lot of cleanup).
I assume that process would be easy to speed up if the requirement for absolute accuracy was removed. The 8' ROMER arm we use is accurate to ~!2 microns over its entire volume which is absolutely overkill for something intended to produce models for visual arts applications. A quick and dirty approach to generating the mesh might increase the inaccuracy by several orders of magnitude but when coke can has dimensional tolerances to the tune of tenths of a millimeter, the quick and dirty mesh will still be representative of the end product.
Unfortunately not. FARO and other structured light systems don't export texture and are generally too precise (micron) in current form. So they take too much post processing by default.
Who would be the primary customers? The entire 3D capturing market is currently several $B per year, including services. Where would be the customers that aren't getting served today that would double this market?
Well it would siphon everything away from the existing 3D capture industry and open it up to smaller groups and those that aren't savvy on it yet. The consumer space generally isn't doing this so anyone that sells anything would get on board at a low enough price point and simplicity.
My colleagues at Creaform have something that works pretty well. Their latest handheld scanner can generate a wire mesh on the fly with 0.030mm resolution.
I'm sure I speak for many of my fellow fans of physics when I say that technology capable 8.6 seconds per scanning a complex smaller object would have applications beyond just making 3D models.
There isn't a combination of laser scanning and/or structured light projection scanning that can accomplish this? Or is it a speed/quality control issue of the output?
Triangulation laser scanning is about the closest you get in terms of accuracy. It can work on virtually any surface, including specularly reflective ones (I've worked on bespoke systems for the steel industry). It's accurate down to microns, but the usual problem is the field of view sucks - either you go further away and sacrifice resolution or you go really close and accept that you need to move the scanner (or object) around a lot. For small things, it's fine. You put your doodad on a turntable. For cars, forget it.
Stereo structured light is great, but doesn't work on specularly reflective objects. You've seen those amazing depth maps from the guys at Middlebury? Wonder how they get perfect ground truth on motorbike cowls that are essentially mirrors? Well they have to spray paint them grey so that you can see the light. The next problem is that you're limited by the resolution of the projector (so I guess if you own a cinema, yay!) and the cameras. Then you have to do all the inter-image code matching which sounds trivial in the papers, but in practice a lot harder (and since you don't get codes at all pixels you need to interpolate, etc, etc).
There are handheld scanners like the Creaform which work pretty well on small things, but I don't know what the accuracy is like.
The ultimate system would probably be a high-resolution, high-accuracy, scanned LIDAR system. Then you lose the problems with scanning ranges/depth of field, but you accept massively higher cost and possibly a much longer scan time for accurate systems.
I got so exited about the thread and it made me realise something very interesting which I turned into an essay. I call it looking for hidden problems underneat obvious solutions.
Well, I'm sure you would still get different people commenting due to one set being home on a Friday night one month and then a different set the next. I would prefer monthly, personally.
Exactly. The whole point is to get access to people with insights about an industry and who can point to the problems and why they haven't been solved and connect them with people who might have solutions to those problems.
Just imagine how much valuable knowledge and insights get lost every time someone retires.
Not suggesting this is an end all solution, since CHF can also stem from non diet related issues, but eating a whole food plant based diet can start to reverse heart disease in as little as a few weeks. It can also prevent and reverse many other diet caused/related diseases.
I think part of the issue has mainly been a lack of education on what a poor diet consists of.
I'm actually close to launching this kind of product. A bit more complex than Word + Git, but I hope it will be worth it.
One question: is there a use case for live group-editing? Same way live code editors work. I have to admit that I'm not super familiar of the workflows in that industry, so I am mostly just designing features the way I would like them to work.
I'm expecting to get the MVP done before the end of this year.
The MVP is mostly targeted at businesses who have to sign/generate contracts at scale. There's a lot of features for those use cases, and then there's features for users that the parent is talking about (pure contract-drafting).
I have not talked to any lawyers yet. I think the worst part is that I don't even know how good the computer-skills of lawyers are, so... designing features by my competence feels wrong. Since I can design the UI to match the general structure of a contract, there's a lot that I can add to make the user's life easier, but it also adds more for the user to learn.
Do you mind sending me a message when you launch? Currently work with a bunch of engineers and technical editors and this is something we are looking to address.
Be very careful trying to solve this "problem". While MS Word change tracking may seem abhorrent to you, most lawyers are well trained in its use and are very comfortable using it. Also, switching costs are huge. A lawyer would not only need to see the benefit of an alternative but also convince all other lawyers they work with to put aside the solution they are already comfortable using and try something new.
I should note - yes, my wife sees no problem at all with this workflow.
My understanding of the workflow is that they will send a clean copy as well as the redline to each other. Sometimes someone will in response further modify the wrong copy, and send back, which causes problems as it becomes unclear what was accepted and what was not and someone has to manually go through and check things. I hear that complaint come up - so I know there is at least one pain point involved.
The in house counsels are probably the weak underbelly of the law industry. Any startup thinking of disrupting this area should probably target the in house lawyers first.
A monthly subscription service for a fee of $5 - $10 where every month I get a new kind of quirky instrument (shaker, wind, wood block and stick, whistle, triangle, or other noise maker) as a surprise by mail.
The point of the service in the music industry is to inspire new sounds and the device can either be kept or given away to somebody without a lot of second thought. Getting stuck is a big problem. Also in music it's important to collaborate and giving gifts is a good way to make connections and impressions.
Great tie-in with various Manufacturers or even retailers to get rid of excess stock / failed impulse buy items / etc.
Eh I was thinking more USPS and less expectation of timeliness. More about the "set it and forget it until it shows up" kind of thing. Point understood though, as $10 starts to get into a competitive space (e.g. Spotify).
Now that you mention it, I think the idea might be inspired by what Dolly Parton does with her reading program. A child can sign up and the program will send them books for free. Considering music and arts funding don't seem to be high priorities in modern US systems (my perspective, could be wrong) having a low-cost or charitable music version sounds like it fits nicely too.
Honestly I'm pulling up memories now of watching other people's kids over the years playing with various music stuff I've handed over (Korg Monotron synths were the biggest hit) and those are really pleasant to recall!
Very cool and thank you for providing input! I've already got the 'spiderweb' model in mind so crunching some numbers might be fun, err, well, informative. I'm planning to spend 2017 learning JavaScript and building small things so to have a list of personal projects seems neat. Appreciate your time!
The ones poorly served by iphones and androids. The ones that see gadgets as a tool and not an identity. The ones that want to augment their abilities without enslaving themselves.
A ticketing system that doesn't sux (I like RequestTracker, but it shows its age). Top players are ridiculously overpriced.
My management style is like this: every task/request is numbered, placed in a queue and assigned to a professional.
What I expect from my ticketing system:
- every manager should be able to assign tasks to someone and set the order they must be executed. He needs know what his team is doing and when they finish each task.
- every professional should know what to do and what are the priorities.
- everything is numbered and linked, all communication recorded.
Everything should be well integrated with email (please, don't send me a notification email about an answer and an url, send me the f* answer). If I answer the email, everything goes into the system, I should be able to send commands to the system by email (for example, add a keyword in order to make it a comment instead of answering).
In a nutshell, I argue that the problem with most ticket systems is that they do not constrain the domain enough, so they wind up having similar problems to email (sifting through a chronologically-ordered pile of text rather than structured, semantically-ordered information).
Your comments make me think the crux of the problem is that people want tickets to be like email and use email to manage them. I'm not sure you can ever overcome the "chronological pile-up" problem if you allow email as a user interface to ticketing.
The simplest solution to the 'chronological pile-up problem' (nice name BTW) is a Wiki model, where replies are appended by default but the entire content can be edited if necessary. (C2 demonstrates this quite well.) For simple problems, conversations behave exactly the way they used to, but when it starts getting complex someone can go in and rearrange the conversation into a more logical form. This actually maps quite well to email: by default replies are appended to the bottom, but they can also be inserted inline (some mailing list etiquettes even demand this) or indeed the entire conversation can be rewritten. You'd probably want some sort of merge algorithm in case someone replies to an older email.
In fact, my usual approach to dealing with tickets/issues/emails which start to develop this problem is to make my own private copy of the thread and edit it in precisely this manner, though I'm the only one this benefits since it doesn't get sent back upstream.
I also have an idiosyncratic way of organizing this stuff, which is basically to use Emacs + mu4e to search my mail, and if I need to create order, write a new document from scratch. I have a coworker who does what you do, now that you mention it—he will take a series of email, dump it into Word and edit it until it is a useful document of some sort.
I still think there is something here though. Stack Overflow replaced message boards, which were basically HTML versions of mailing lists, and part of that was identifying the semantics of question, answer and comment and defining new operators and new expectations for them.
A wiki is a good approach but because it's totally free-form, the user gets stuck doing the work of keeping things hygienic.
JIRA allows you to edit all the properties of a ticket whenever, but it generates such a huge cloud of email notifications in the process, it kind of disincentivises you from using it. And nobody is in the habit of rereading the page to see what is different since last time.
> Your comments make me think the crux of the problem is that people want tickets to be like email and use email to manage them. I'm not sure you can ever overcome the "chronological pile-up" problem if you allow email as a user interface to ticketing.
I agree that's partly it, but that seems ok when you're in the thick of discussing a problem/fix. If you're doing a code review or something after a fix has been pushed, you actually want certain messages to stand out to describe resolutions and whatnot.
So like gmail where you can star/mark certain replies as important and those messages would show up at top-level in the ticket, where all other messages are collapsed.
We're in our first month of releasing Zammad [1] an open source Zendesk alternative with pretty neat features. You can check out some screenshots or a free 30 day trial oft our hosted solution on our commercial site [2]. I really like your feature ideas and will later create issues for them. Would be great if you add some too if you have more of them.
Full disclosure: I'm part of the maintainer staff.
Additional features: custom fields w/ user-chosen types: free text field, drop down list, etc.); time tracking (I spent n hours/minutes on this ticket); these should be searchable.
Major feature that allows me to work around any shortcomings in your office: API access to everything and/or database access (preferably direct read/write access, but even if it's just a downloadable .sql.gz it's a huge benefit).
The problem here seems to be that users/customers insist on customizing any such app to death.
Personally, I think the optimal ticket system would have this data for each ticket:
* A unique, prefixed ticket # (JIRA gets this right)
* An assignee (like an email To:)
* A reporter (like an email From:)
* A one-line summary (like an email Subject:)
* A multi-line body (like an email body, but ideally with markdown)
* Attachments (like email attachments)
* History for edits of all of these (not like email!)
That's it! It really is basically email, but with a unique ID, and editable with history instead of immutable with replies, and a decent UI, perhaps RSS + notifications.
Unfortunately, everybody else seems to think that their ticketing system should embody their vaguely defined and ever-changing workflow, prioritization, approval, and release management system, so they want to be able to add any number of possible statuses, approvals, workflows and and all the rest. Once you add that, you end up with another JIRA or ClearQuest or BugZilla, and the cycle repeats itself.
This sounds very like Redmine [0]. It's ostensibly for project "issues" however it's extremely customizable and all of the above are included in the default config and not much more. It sounds like if you removed about 2 default fields it'd be perfect for the ticketing system you describe above. Plus RSS + Notifications + a solid API.
I'm not associated with them, but I have used them successfully for months at a time (better than most productivity software). The reason is it is well integrated and similar to email.
Asana's start up time is just ridiculously slow. Probably the slowest webapp I've ever used. Also, you can't assign more than one person to a ticket, which is a pretty big limitation
JIRA also doesn't let you assign more than one person to a ticket. Could you expand on why this is a problem? I'm not familiarized with this problem space so I'm just curious.
The recent GitHub updates let you assign multiple people to reviews and such, but I find it's usually better to tag everyone you want to look at something. I don't think assigning something will send a notification.
Asana is pretty bad. It takes forever to load, which sucks when people send you URLs to tasks/projects and you have to open them individually and wait almost 10 seconds for each to open.
I've been building support and dev/ops ticketing systems for years and I still haven't found a platform that suits all needs.
For my latest startup I went looking for a service desk tool. The key criteria was "feels like email". The moment any alternative required a user signup just to lodge a support request, I ruled it out.
I ended up choosing Groove. I don't recommended it. All ticketing systems suck, this one just sucked the least for my support desk. Groove doesn't extend to other ticket types, and it's nowhere near as flexible or extensible as JIRA, and the mobile experience is horrible. But it does "feels like email" for my customers better than every alternative you care to mention.
> every manager should be able to assign tasks to someone and set the order they must be executed. He needs know what his team is doing and when they finish each task.
That sounds like unnecessary micromanaging. You couldn't possibly have enough detailed knowledge to know the proper order of tasks in all cases. Possibly even most cases.
I agree that communicating the priorities are important, but the boots on the ground have a much better understanding of what they're working with than you do.
we use Github issues + Zenhub for that (though Github has recently implemented a lot of Zenhub's features). Managers mainly use the 'boards' and 'milestones' views; devs use 'boards' and whatever else. Messages are included in emails and you can reply by email.
Many doctor offices still have horrifying paper record systems, don't use e-invites, etc. Companies are fixing this but the market is still huge.
I don't know why the few companies that are in the IoT space for oil and gas aren't scooping up literally billions in missed opportunities for sticking cell-enabled sensors on oil platforms, fields, etc. Then there's the next billion dollars waiting for whoever starts sticking controllers next to the sensors on valves, etc.
Drone inspection and repair in oil and gas. People are doing this but it's taking way too long to take off for how much money is there just waiting to be picked up.
Sticking a laser scanner on a drone. Again, people are doing it, but what the fuck, there's so much money just sitting there.
If you're looking for contract work, just start browsing random EPC websites and calling up the shitty ones. Probably a good 10,000 at least that are still rocking 1995 crapsites, and not in the "good" "low functionality low load time" way, the "using tables for layout" way.
Trains should be automated. They already are in Taiwan for some lines, it's been feasible for years.
Motorcycle safety is still subpar for where we are in material science. There are a lot of riders out there that will pay buckets for greater safety. We've figured out how to not get our skin ripped off but I believe there's still a market for preventing broken bones, spinal snaps, decapitations, and the like.
Somehow we still don't have GUI HUDs in our moto helmets. Like BMW is working on something but honestly it would be a relatively simple and profitable thing for a very small startup (2 man team working part time). Literally even just casting your smartphone screen to the visor would be enough to get people buying so they can have a HUD map and shit.
Big market for motorcycle storage solutions. I giggle whenever I see someone with an ammo box strapped onto their sports bike. We already dropped hundreds on gear and thousands on the bike, we spend many more hundreds or thousands on modding the things, there's ample opportunity for more elegant and functional storage solutions.
VR allows for limitless desktop screen space in a portable package. I'd like to be able to bring an HTC vive and a tiny screenless box to plug it into that would allow me to have a "multi-monitor" setup while I travel. Some say the resolution isn't there yet, I say make the text bigger. I have no problem reading the stuff in steamVR.
I get made fun of every time I bring it up but I'm convinced people are stupidly ignoring lighter than air travel, transportation, and data distribution, especially in an automated sense. Google has their wifi balloon thing but they dropped their blimp transport truck project. I think it could've been a thing.
There may be some margins available in teaching low-income people how to cook and eat basic foods instead of frozen meals. It took me to getting scholarshipped into college to realize that we were losing buckets of money eating frozen meals and fast food because we thought it was the "cheapest option," not to mention how unhealthy we were for it. Think like somehow getting low-income folks to buy potatoes, onions, peppers, dried beans, cheap cuts of meat, etc and demonstrating how it's faster, cheaper, healthier, etc. Potentially a gov funding opportunity, would save on EBT and healthcare costs.
Kids learn by doing. Good luck changing anything about education in the USA though.
Someone would be able to take over any industry in Taiwan that they please if they start up the company and instantly pay 2x local salary, give PTO, and have other basic benefits that we take for granted here. It wouldn't be much, you'd be paying ~44k/year USD for an Engineer, for example. You'd be able to poach the best talent, you'd draw shitloads of negative press from pissed of old Taiwanese bus...
EPC means "engineering, procurement, and construction" company. So like, lyondell-basell or jacobs engineering or foster wheeler (now AMEC) or Mustang (not the car) or KBR.
PTO = Paid Time Off. Catchall for vacation time, sick time, holidays.
GP offices: Plenty of solutions already - generally regionally based or by payer. Also an issue with vets, hairdressers etc - pick an appointment based niche. (Have invested in 2 niches, advised others)
Wireless sensors for industry: Thousands of players here, including network providers and different tech stacks. The game is to find a niche that is easy to enter and pats well. (Have invested in wireless water metering, advised others)
Drone inspection: Plenty of players here, but it's a find-a-niche that you can sell to play, and you need operators for each application. (Have advised drone and drone application companies)
Anything on a drone: Drone's don't last long in the sky. (as above)
Trains: Trains get solved at the political level. Good luck in the USA. (am train lobbyist)
Motorbike safety: Check out where Dainese and BMW are going - air bags et al. I spend a fortune on BMW gear. (Motorcyclist)
GUI HUDs: The main player failed (Skully - internal issues) . I know of at least one early stage company attempting a generic HUD solution. (have met, not invested the early stage one and decided, luckily, that I didn't like the Skully people and did not pre-order.)
Motorbike storage: Has been a huge market since the 90s when you could only get adventure boxes from one supplier - in Munich. Now the problem is solved by countless suppliers and manufacturers - unless you buy a sports bike, which have a very different use-case. My first adventure trip was in 98 and I needed to self-assemble, now you can ride out of several branded dealerships with a capable round the world bike. (have motorbiked round what world - in pieces)
VR: The use-case is really only games so far. See Playstation. (have 2 VR kits)
Lighter than Air: Capacity and cost versus shipping are the issue, while air-cargo is well solved. Meanwhile the Helium market is stuffed. But I love the industry too. (read Bill Gate's recommended book on international shipping etc.)
Low income cooking: Those low income people are time-poor, poorly educated (the US system is dreadful) and marketed too by monster franken-food companies that price under the cost of decent food. Yes - its a hard problem to solve. (don't get me started)
Education: Indeed. Start with politics and how schools are funded.
Taiwan: (sadly) Why would you pay more than $1 over market?
Not my industry, but in my area. I'm still looking for a good modular house that can be setup reasonably quickly, low cost, and can survive North Dakota winter and summer. Something suitable for a single person or a couple.
They would seriously have to be cleaned given how toxic the paint on those things is. Plus, I would rather have something not prone to giving the local inspector a fit.
I have no experience with them myself, but I know there's a good bit of interest in them for non-traditional uses. A Google search for shipping container homes will turn up a lot of interesting results. It still might give your inspector a fit, but there are resources out there. You wouldn't be blazing completely new ground.
I don't see a lot on insulating and we are a bit far from the coasts to make availability great. There was a thread a while back where these were discussed.
These guys sell full size or tiny model geodesic domes, which go up pretty quickly on a poured platform. They are supposed to be able to withstand hurricane force winds ...
Regulations will unbundle research from liquidity provision starting sometime in 2018 (moving regulatory target, but theme seems clear). Sellside will have lower incentive to pay large research groups; buyside will have to pay explicit fees for research advisory. There is a big opportunity in providing platforms, with macro and market data live, where researchers can interact with capital managers, given that said researchers will likely find themselves bereft of their current distribution networks (bank sales forces) sometime in the next few years.
Equities already have a hybrid form of this where buyside earns "credits" to be allocated at the end of each year to research providers. But fixed income is at least twice as big as equities, is much more opaque because is essentially unlisted (mainly "OTC" = "over the counter" ie only those in the know), and therefore much more susceptible to disruption.
apologies the only "CSA" I know is the Credit Support Annex between counterparties, which seeks to minimize risk on mark to market of derivative contracts. Is this what you are referring to?
Global CDN with a fixed monthly cost and a capped bandwidth. For example, pay 100$/month and get up to 1Gb/s, traffic above this limit is dropped.
Providers today offer uncapped bandwidth, some, like Amazon, without any cost cap, some, like KeyCDN, allow to set a cost cap, but take the resources off-line when the limit is exceeded.
733 comments
[ 5.0 ms ] story [ 347 ms ] threadI don't know about you, but I despise filling out the same forms over and over again when seeing new healthcare providers. I'd love to start a service modeled after granular smartphone permissions where
(a) I check in at a new office (scan a code, they scan my code, beacon, something like that)
(b) the office then requests x, y, and z information
(c) a push is sent to my phone where I can review the information and approve or disapprove some or all permissions
(d) a final step of either entering my pin at the office, using my thumbprint on my device, or something else.
The key components would be storing the data encrypted at rest, following HIPAA and then some, having a solid auth protocol (keys, jwts, etc).
I think adoption would be helped because the public are already used to permissions like these when installing apps.
The benefits are a lack of paper trail, no one is going to not shred my SSN, my most up to date data is now available, and instead of hosting N apps/databases, I'm storing 1 and can reduce my maintenance, customer support issues because one for all, all for one.
Edit: edited for readability.
I'm sure it's possible to hack together an AHK script[1], combined with Pulover's macro creator[2] to automate virtually anything repetitive on a Windows PC, or use Selenium to automate browser actions[3]. Of course then you run the risk of having to fall into the classic XKCD automation time sink[4].
[1] http://ahkscript.org/
[2] http://www.macrocreator.com/
[3] http://www.seleniumhq.org/
[4] https://xkcd.com/1319/
Agreed. Especially considering they have an ENTIRE department devoted to personnel along with an office at every single unit level above platoon.
I'd suggest something much more low-tech - a website where you can punch in all your details - insurance, allergies, medical history, etc, etc... and then you can print it out (or a subset of it, for different kinds of providers) or generate a PDF that they can copy & paste into their horrible legacy system (an improvement on retyping), or, for those truly at the cutting edge - the kind of electronic transmission you speak of.
I am on board with what you're saying; an escape hatch for non- or semi-adopters. Obviously, printing is a way to go, so maybe on the mobile app, the ability to check each piece of information required then export/email to your preferred destination.
It'd also be interesting to look to make money on conversion i/e replacing, or integrating with, the outdated monsters you're talking about.
Maybe we're not even talking about healthcare anymore, maybe just the ability to piece together PII (personally identifiable information) and deliver it to X.
>>>> on another note
This goes into a topic I've seen posts on recently, and something of interest to me, personal indexing; a better way to throw blobs against the wall and have it indexed for me, leading to a personal Google. I mean, that's already coming, really, between Facebook and Google (especially Google Photos) but currently I see nothing about piecing together information I'd like to share on a professional level.
It's actually a pretty good solution for ad-hoc "working together" with someone (a lawyer / architect / whatever) on a project, where you have lots of files you need to share and refer to during the project.
I wonder if you could stitch together a workflow as a reseller for Google Apps (no clue what their current name is)?
Either way, good suggestion.
Sell the service to the patients for some smallish fee ($5 per month) and then provide the integrations into the various provider systems for free.
Later on you could scale it up to be an add-on to employee benefits or the health plans.
https://www.linkedin.com/in/rancar2
I would love to hear from anyone else with big ideas relating to or are working on driving outcomes towards holistic wellness with patient-center healthcare, patient data collection/quantified self, and patient-powered research networks. In the bigger picture, I am passionate about making the world a better place through innovation and working on what really matters for humanity.
They also offer identity document verification with facial recognition crosscheck. They want to use this to detect visa overstayers for immediate deportation.[3] That now looks like a market with potential.
[1] http://www.card-reader.com/medical_cards.htm [2] http://www.acuantcorp.com/autofill-software/ [3] http://sandhill.com/article/its-high-time-we-build-border-te...
https://crosschx.com/patients
Edit to add: https://en.wikipedia.org/wiki/Carte_Vitale
When you go to the doctor for a sinus infection, the cost of this is not fixed, even across insurance companies. The other factor is the "level" of service. (ref: http://medicaleconomics.modernmedicine.com/medical-economics...)
The more in-depth the examination and the more time you spend with the patient, the more they can charge. All those forms are "taking family history", etc. and it is free money since you have to do the work. Those are then scanned so they can be used later in an audit.
(Source: I also worked at a start-up that was trying to disrupt out patient medical systems. It's very hard and has lots of roadblocks. btw, of the top 50 EMRs in the US, only 3 have APIs and these are mostly to pull data, not push it back in).
Problem is these who would benefit (residents) are often disconnected from those against it (the employees). You would need boards that really are sold on the benefit.
It's a decent industry my friend (not after this posting) makes ~300k a year working on it. However the race may be ending as these companies lock apartment managers into long complicated contracts. Not sure how many apartment managers at scale still need.
I'd tend to agree with the other comments that the hired out management companies would be a much better sales target since they have a real profit motive to efficiency gains. At best a neighborhood HOA is looking at periodically saving some time and being a bit better organized.
/s
Being a SaaS provider, create an extensions that you have to install (something consumers want...). This extension should have the permissions to read history / or urls visiting.
Have a blacklist transmitted of urls (so you don't need to transfer the user's data back to the server) and match with it. If it matches...you have to work on that consumer a lot more.
1) how about you find ways to make retention a continuous objective 2) get used to the fact that not all customers will be retained. I try stuff all the time that I have no idea if it will solve my specific problem. If you do #1 right, I would know that you want to hear about my specific problems and offer ideas on how your saas can solve them. That's the most you can hope for if your service doesn't immediately do what I initially hoped it would (or is more burdensome to implement that I thought, etc)
It's quite mind-boggling; nobody is really doing it on an industry-scale level. Every video game developer has their own way, all of which have their own problems.
It is a very hard problem. Blizzard actually came up with a very good system, but it's not in a state where it can be commercialized or open sourced.
I actually think whoever comes up with a system which solves these problems in a clean and consistent way will be sitting on a little revolution for content distribution.
Which is not at all interesting for self-published games (be it indies who want to avoid steam, big publishers with their own systems etc).
http://store.steampowered.com/search/?publisher=Electronic%2...
Some companies are even selling more or less standard solutions for that, but in reality from any given 1000 games 900 will have very different data formats and all have fairly good reasons to do so - using universal "patch systems" really creates more problems.
Gamedev is riddled with really smart people that reinvent the wheel all the time because they found a way to micro-optimize this or that. They get to do this because until recently, there was no "good enough" solution for a wide range of games (or the "solution" was priced with enough zeroes to make bill gates cringe).
But you saw how popular Unity got, and how fast. That's the games industry in a nutshell: ripe for solutions that work for more than just one studio.
And compare, say, to simple incremental zips of Quake with alphabetic file loading order.. Total no-brainer to implement and use. (I have even seen zips with custom LZMA compression!)
So, if any, someone will have to solve a problem of artificially created obstacles, not a problem per se.
But I'm a huge fan of irrlicht 3d.no idea why no one uses it?
You could then devote so much more time on ideas...
Speaking from experience as I'm currently making the jump to Unity for my projects, the time savings of choosing one of the all-in-one engines instead of gluing together engines is really substantial.
This is done in bits and pieces across existing engines and third-party tools, but there's a lot of room to make it cheaper and easier.
Quoting Wharf's spec intro:
Butler is the commandline tool for generating patches (it can negotiate small diffs from the server without requiring a full local copy of the thing you're diffing against), uploading them and applying them back on the client.It is used to power itch.io's Steam-like application, itch: http://itch.io/app, delivering multi-gigabyte game installs & updates.
- File formats are streams of protobuf messages - efficient serialization, easy to parse from a bunch of programming languages. Most files (patches, signatures) are composed of an uncompressed header, and a brotli-compressed stream (in the reference implementation, compression format are pluggable) of other messages.
- The main diff method is based on rsync. It's slightly tuned, in that: it operates over the hashes of all files (which means rename tracking is seamless - the reference implementation detects that and handles it efficiently), and it takes into account partial blocks (at the end of files, smaller than the block size)
- The reference implementation is quite modular Go, which is nice for portability, and, like elisee mentioned, used in production at itch.io. We assume most things are streaming (so that, for example, you can apply a patch while downloading it, no temporary writes to disk needed), we actually use a virtual file system for all downloads and updates.
- The reference implementation contains support for block-based (4MB default) file delivery, which is useful for a verify/heal process (figure out which parts are missing/have been corrupted and correct them)
- The wharf repo contains the basis of a second diff method, based on rsync - for a secondary patch optimization step. The bsdiff algorithm is well-commented with references to the original paper, and there's an opt-in parallel bsdiff codepath (as in multi-core suffix sorting, not just bsdiff operating on chunks)
- A few other companies (including well-known gaming actors) have started reaching out / using parts of wharf for their own usage, I'll happily name names as soon as it's all become more public :)
I'd be happy to answer any questions!
http://android-developers.blogspot.com.au/2016/12/saving-dat...
https://www.gog.com/galaxy
It's exactly what you want, but this service isn't poplar. It doesn't work because everyone is on Steam. Network effects. It's like when app.net tried to replace twitter. You can get mad at users, but users are using Steam & Origin and Battle.net and they don't care.
Recently I returned to Australia to spend some Christmas time with my extended family.
A few mornings ago, I put some real bread I cut from a sourdough loaf in a toaster. Due to its irregular size, when it popped it didn't pop out completely, resulting in a sort of "toaster is too hot to insert fingers, toast is too hot to hold, toast is ready, find metallic implement to insert in to mains-powered device to extract toast" problem.
I mentally facepalmed.
Someone should really fix toasters.
For toast it may be energy inefficient. But if I'm just making a small pizza, etc, it's way better than turning on the large oven.
Mine are nothing so fancy as this image I could find online, but you get the idea.
A pair of popsicle sticks secured together at one end would be fine.
http://ecx.images-amazon.com/images/I/81HzpTQ93XL.jpg
Perhaps it is playing too much Shenzhen I/O, but I feel like a basic IC and sensors could solve this by detecting the width/height of toast pieces and continuing to release until the toast was substantially ejected from the toaster.
Some toasters have wire meshes that close in on the bread to hold it vertical, but they tend to leave marks.
This happened pretty regularly as it was a European toaster designed for perfectly uniform extruded wonder-"toast" and I kept sticking hand-cut slices from round loaves into it.
They have. They're called dualit.
https://www.amazon.com/Dualit-20293-2-Slice-Toaster-Chrome/d...
I bought my bottom of the barrel Wal-Mart toaster for $6 about 8 years ago. It might occasionally have issues popping out a piece of bread but it cost $6 and it toasts bread consistently.
I'm mostly holding on to this thing to prove a point to the "they don't make them like they used to" and the "you have to spend at least $100+ on X or else it will break in a week" crowd. I know it isn't the best toaster out there but I'd take it over a $189 toaster even if both were offered for free just because I'd feel like an idiot for having something so ostentatious on my countertop.
Maybe I should have linked to this: https://www.amazon.co.uk/Russell-Hobbs-22400-Slice-Toaster/d...
Haha ofc
1) Cleaning the data as it comes in rather than in batches so we can use it sooner, invalid data is discarded, outlier detection, normalizing inputs etc....
2) Warehousing of the data with proper indexes so you can perform some advanced queries on unstructured data
3) Some data is sent in bulk at the end of day, some of the data is streamed in fire hose style. How can we preprocess the fire hose data so that we don't have to wait until the end of the day to parse it all.
4) Oh and all of this data is unstructured and comes from 75 different sources.
Soon the average hedge fund will have more people just cleaning and managing data than they do in quantitative research, dev ops, software development and trading.
Oh and lots of the data is considered proprietary so while AWS/Azure, etc is fine, sending it to a third party to process is not.
TL/DR Help me, I'm drowning in data. How do I get the time from when I acquire data to when I trade based on it down to a reasonable time frame, where reasonable is closer to hours rather than days/weeks.
The same line of inquiry has been evaluated for most 3rd party software that companies rely on. For this specific instance of data collection and cleaning, I'm imagining it's not going to be a much different calculus, although perhaps you'll see a higher percentage of firms choosing to roll their own if they have the chops and pockets (e.g. Two Sigma, Bridgewater, Goldman Sachs, etc.).
I will note that there are commercial mechanisms firms could try to implement to try to limit the downsides in case something like this happens: warranty & damages provisions, and insurance are two come that spring to mind. I'm sure there are numerous other considerations in the age-old "build or buy" cost-benefit analysis.
Cleaning:
Open Refine seems to be the best product in this category. I haven't used anything but my own tools to do this before, so I can't really offer any advice.
Warehousing:
My understanding is that this is just a fancy way to talk about a database with a schema designed for analytics. There are many open source databases which do this very well, the one I use being Cassandra (and/or KairosDB), though it is also likely the one that is hardest to use. For a beginner, you might want to refer to this SO answer: http://stackoverflow.com/questions/8816429/is-there-a-powerf...
Data processing/collection:
This is something that is incredibly dependent on the data sources, so I likely can't tell you anything that will help. Most of my data sources I've worked with have been internally sourced log files, messages from ZMQ, or CSV data - you might be working with something far different though, since there are lots of public data sets and such which are common. Ideally, this would be integrated into the tools that you are using to clean the data, but I don't know if that exists.
Handling input from many different sources at different rates is not a very hard problem to solve if your system is build correctly - you could for example run a daemon for each data source which will populate the database when there is new data available, then send a message off to the processing engine, which will integrate the data into whatever reports you are running.
Specifically for a use case of a hedge fund, the reports could be triggered by a message which is sent when the new data is available, and processing could be done in parallel in Lambda or similar dependent on need to get a nearly instant return, enabling nearly real-time reporting.
Check out my company, SnapLogic: https://www.snaplogic.com/
I have seen local companies working for months/years to finally use his BI package but the trouble at the step 1 is big (and also, to put the data in a "nice" schema).
The problem is that enter in this space is hard. Years ago I was at a company that have a niche product (in foxpro) for this kind of task, and I have dreamed about build something like this based in my experience, but get the funding for this kind of "boring" task is hard (more in my country, Colombia).
P.D: If wanna help, we can talk. I can't give a magical solution but at least I find this kind of "boring" jobs compelling ;)
We threw every rule out the window in the name of performance _when fetching raw data from external sources_. So we had weather station networks, NOAA forecast runs and NASA satellite data in a workable schema in our shop way faster than average. Mix of C, PowerShell, Perl, and the nonstandard parts of T-SQL, highly parallelized, tricky but fast.
After the "workable schema" was established, the rules came back and we acted more responsibly. Smart instead of clever.
Ran this stuff all day long, getting every piece of data asap. Things that can only be calculated with a full day of data we poked and prodded the meteorologists to express in "partial aggregates", which to me were just like the map steps before an EOD reduce.
Took a lot of mutual understanding and iterating but worth it in the end. When the ultimate data source (satellite or radar site for us) posted its last hour of data, we were 95% done with the day's computation work. We do our last step, publish our numbers, and bam, Our ag clients have this stuff a day earlier than they are used to.
http://www.striim.com/
It's a native streaming platform so your data will be cleansed, processed, scanned for outliers event-by-event rather than in batches. We have dozens of streaming connectors IT/Enterprise/Web data sources. We also support initial load for your firehose data. For unstructured data, we have support for RegEx based parsers.
Shoot me a message if you have any more questions. We have many big name users in Aerospace, Banking, Device manufacturing, and Logistics industries.
www.watchful.io
It's open source, and match time is linear in the length of the input string.
(Disclosure: I work at Google on a different open source project)
More like "mitigating" or "occasionally getting regex slightly more right than some other solutions". There are many different approaches to regex and all seem to focus on different parts of functionality (RE2 focuses on quick compiles and simplicity, libpcre has 'all the functionality', we're about streaming + large scale + high performance if you can tolerate long compiles and lots of complexity). A number of new projects are trying very interesting approaches, like icgrep and the Rust regex guys.
I would be curious to hear about your approach.
www.watchful.io
I don't know much about Paxata but I think Trifacta are well-regarded in industry and academia. Trifacta founders worked on research / open-source-? project Data Wrangler http://vis.stanford.edu/wrangler/ and turned it into Trifacta.
I don't know much about either product in truth.
1. The UX is subpar. It insists on running in only a single tab at a time, and attempts to open multiple tabs will instead override whatever it considers to be the master tab. This is a huge pain, because I often need to have a mapping workflow open in one window and some other relevant part of the application open in another. Instead I have to save, go find the thing I want, and go back. Another problem is that when working with data sources containing tons of fields, there's no easy way to search.
2. It offers an expression language to perform some computational tasks, similar to what you'd find in Excel, but it's hamstrung by a poor UI and a limited amount of functions. The built-in editor for expressions is really poor (see Tableau for an example of a great editor for a simple Excel-like language; it even has type linting) and, unless I've misunderstood something, you can't declare any variables so you end up with huge nested expressions. There aren't many functions available, so something as simple as removing whitespace ends up as lstrip(rstrip(foo)). In combination with no support for statements (or at least a let expression like in lisp) this makes any nontrivial data munging completely indecipherable.
I've looked around in this space and it seems like there are a variety of products, but the supplier of our main CRM will only support Informatica Cloud. I think that a company that can offer a product that does what you've said but makes a serious effort at UX could cause users to revolt and demand to use it! I know the joke is that Slack is just a pretty IRC with better UX... but that's exactly why it has become so successful.
In terms of data munging, take a look at Microsoft's Power BI. It's visualization software but it has a nice data munging mode that, crucially, keeps track of all the changes you make and displays them in a linear format. This is great for getting a quick idea as to what was done with the data and is essential for doing reproducible data analyses. Unfortunately, Power BI also suffers from poor UX in insisting on tiny fonts and gray-on-gray palettes that are totally unreadable to anyone over 30.
https://www.holistics.io/data-preparation/
And on the part about firehose data, you might already know this, but Kafka and their line of work should be aligned with what you're after.
Omg, so many sales pitches. You should figure out which of those were automatically generated by someone who's bot is crawling HN and using NLP to find posts like this, and then hire them. There's basically 0 chance that isn't happening...
Sounds like a job for Apache Camel?
+scaleable
Problem solved on all points.
100% fully functional, fully featured, 100% free download right from the website.
Disclaimer: I work for splunk.
Unfortunately, there are very few people that understand both computer science and structural engineering.
In the US there are about 281,400 civil engineers [1]. I couldn't find more detailed information on structural engineers.
-Assume about 10% are practicing structural engineers who need to design concrete structures = 28140.
-Assume a company wants 1 license for every 2 engineers = 14070. (I base this off the fact that my company has 6 licenses for 12 engineers, but we may be higher than average)
-Assume we could get 10% market share = 1407 subscribers.
-Assume $1000/subscriber/year = $1,407,000 from the US market
Obviously this isn't a very rigourous analysis.
[1]: http://www.bls.gov/ooh/architecture-and-engineering/civil-en...
Single seat licences are not the only revenue model. Once a product gains traction consulting, training and providing VIP helpdesk and bugfixing services factor in as well.
You need computational geometry, computer graphics, and structural engineering expert level domain knowledge to implement anything. You need to create traditional 2D machine/construction design drawings from the 3D models. Then you need to sell it to corporations, whose work, most of all, must be dependable and free of guess work.
You need to know what sort of geometries you can use to model the reinforcements. Then you need to know how to design the system so it can handle very large amounts of geometry.
The worst of all is you need to deal with god awful industry standard formats- DWG, DGN, IFC, Step/Iges and so on. Maybe DWG import and export first.
To have any real chance you need a guy or two who are good with numerical code, someone who is familiar with e.g. Game engines, soemone who knows computer graphics, a structural engineer to tell how he does his job and what the thousand inconsistencies in the field are (this is not a trivial domain like housing or transport), a sales/marketing guy to connect and push the product.
And, like someone else estimated, the potential market is not gigantic - which is kinda funny because we all depend on reinforced concrete but don't need so many engineers for the design work...
These are much more standalone, and don't have many of the issues you listed.
No one I know is using the automated concrete design built into analysis programs like ETABS, Tekla, etc.
The utility of your software tools will be very limited if you are restricting yourself to only member design instead of total structure solutions like ETABS. Why should engineer pay you at all if they can use spreadsheet for free to do what you do with your SaaS?
> No one I know is using the automated concrete design built into analysis programs like ETABS, Tekla, etc.
Not too sure about this because I know quite a lot of people who are using these tools. Any reason why the people you know don't use ETABS or Tekla?
Why do businesses invest in new tech? Why pay for excel when I can use a pen and calculator? The answer is because it makes them more efficient. We have excel sheets to do the same thing, matlab code to do the same thing, and yet here we are paying for these member design tools because they are the most efficient for us. If you save an engineer even a couple of minutes for each element they are designing, you essentially pay for the software.
>Any reason why the people you know don't use ETABS or Tekla?
We do use ETABS extensively for analysis. We don't use it for design. It is foolhardy to trust the automated RC design in these software. That seemed to be the standard of practice around here, but perhaps it is different in other areas of the world.
Do you mind if I ask why? I'm working on a sort of general approach toward designing trustworthy engineering software, and I'm trying to collect as many reasons as possible for "can't trust the software".
Its not a distrust so much as a fundamental flaw. For simple gravity design it works fine, but even then we are using spColumn because its just quicker for us.
The software for that kind of modeling is apparently pretty basic, pretty expensive, buggy, etc.
I thought it was shut down largely because much better software simulations were made available; are they still kinda crummy?
Fluid simulation is a very difficult problem to simulate, structures are a lot simpler.
http://99percentinvisible.org/episode/americas-last-top-mode...
It does not mean that you just can't take a couple of web developers and make it usable, but as the market is small it might pass the price point with a supersonic bang...
They have a sheet with around 30 rows and 150 columns, and they have 100 of these sheets (in a single Excel file). Some parts use formulas, but usually when somebody needs to change something they need to go through every single sheet. The issue is now when they try to add new data Excel won't let them.
I don't even want to know how they share the file or do backups.
Goo thing you're in healthcare...
And that pushes the data back into the same spreadsheet and a relational database with change history.
Is there a queryable relational database?
https://www.smartsheet.com/product-tour
(For what it's worth, I'm doing something similar in the transport planning space. And yes, bridging the gap between that and modern CS is a substantial piece of work.)
Where I would invest (if I were Autodesk or their competitor) is in releasing CAD tools for free in exchange for a consent to use the designs/details internally for ML purposes. Would love to contribute if anyone is working on such a product.
The plan is to link CAD to the mathematics and then link the finite element to this as well. The system would also function as a sort of github for engineering where users can find and use functions to do most standard analysis. Email is in my profile if anyone is interested in talking.
I really like the equations and how you only allow to make formally correct equations (including units). Anxious to see how this develops.
(full disclosure: I am co-founder of a Software which tries to achieve the same aims using different concepts: www.valispace.com)
There are places for users to upload and store data as well - datasets.
As of now, the code solves equations in javascript within the browser. This is why documents have to be opened when a requirement changes - because I have no server side of the code to solve them without the browser. It isn't a long term solution, merely a step in the building on the platform. My next step is to add a server side code that is capable of solver more complex and larger equations on the server. When that is done, changes to requirements will update documents without the need to open them manually. I plan on using python and there are several large libraries available.
This will allow me to link documents to CAD. When the math changes, the CAD will change as well. Once that is done, I will add a finite element meshing and solution system to create an engineering platform that essentially does everything.
I like your site. It's nice to see other people addressing these problems. I am also an aerospace guy. I worked on the shuttle for a while and then designed some components for the Orion. Shoot me an email if you want to talk more.
We are working since 1.5 years with some engineers on a software to solve this: www.valispace.com
I would be curious to hear from you whether what we are building with a focus on the space-industry also applies to structural engineering.
- There are over 900 regional MLS providers each with different schemas.
- You use RETS, a complicated, non web standards interface for downloading data.
- MLS data is plagued with errors and denormalized data making queries difficult.
This problem is hard to solve at the root because each MLS is independently owned and heavily influenced by the National Association of Realtors and local governments. As an active RE investor myself, I'd love to work on a problem in this space.
As an aside, I'm working on a RE investment product that solves challenges similar to this one and I'm looking for RE investors to join me. If you're interested, shoot me an email at hello[at]myname.com.
disclaimer: I am affiliated with rew.ca (a competitor on the west coast)
10+ years ago we had other "agents" that made a lot of money, e.g. "Travel Agent" in Travel&Leisure industry. Then internet came along and now we have 1,000's of websites that have replaced what Travel Agents used to do...
For Real-Estate I think MLS is just a start a small piece of the puzzle, someone very smart is going to disrupt the whole industry and make many billions of dollars.
Also, agents for basic residential purchases where the property is about as mundane as it gets provide very little value. However good agents will do proper diligence by inspecting the public records for the property. That part may be disruptable with software and an ambitious enough records maintenance effort. But good agents who do their jobs properly are well worth 6% (and, by the way, that 6% is frequently split two ways, so each agent gets 3%). Of course my bar for "good agent" is much higher than Joe Coldwell or Jane Keller-Williams. "Good" agents know local and state regulations very well and do diligence to inspect the public record and work closely with well qualified inspectors to get the best (lowest for buyer, highest for seller) price they can. Most agents, especially for "big box" brokers do basic humdrum contract paperwork and have a very basic approach to disclosure and negotiation and don't add much value. They're ripe for disruption.
Kind of an aside; what's the motivation for buyer's agents to get good inspectors or the lowest price possible? They're getting a percentage and they're motivated to make sure the deal falls apart. Although, as a buyer it's not like I knew good inspectors, anyway. The best I could do is read their sample reports and go off of my own (sparse) knowledge of home construction.
Australian here: on my last house sale, I did a deal with the agent where if he got a price above a quite-high threshold that he would get 3% instead of his usual rate. This was seen as a remarkably generous offer, unheard of in the industry.
I've only once seen someone use a buying agent: husband and wife were both hospital doctors with three young active kids and they figured they valued their time enough that they would pay for a real estate agent to short-list properties for them.
I wonder what is so much more difficult about real estate in the USA that it requires so much more in the way of services.
When people buy a new home and move they typically need to get new utilities, health and car insurance, doctors, dentists, furniture, appliances, etc... They need to decide where they're going to shop for groceries, which restaurants they want to explore, what activities they're going to get into.
Agents mainly do two things:
1. If selling they can put your house into MLS.
2. For buyers they can let you into a house.
Being able to do these things requires NAR membership, a state license, and other fees and cost real estate agents thousands per year. Also driving meetings clients at houses takes gas and time. Also your 6% commisiion is paying for all the other people that the agent spent gas and time on who did not buy a house.
Would you be willing to advertise in the news paper that your house is for sale and leave the doors unlocked so that random people could check it out?
About the only thing a buyer gets out of the whole deal is a title search to make sure there are no outstanding liens on the property - and they have to pay for that themselves!
But what if someone steals the appliances? That is actually a problem for new construction.
There's a study which draws similar conclusions:
> Our central finding is that, when listings are not tied to brokerage services, a seller's use of a broker reduces the selling price of the typical home by 5.9 to 7.7 percent, which indicates that agency costs exceed the advantages of brokers' knowledge and expertise by a wide margin.
https://www.nber.org/papers/w13796
(We bought our house with them.)
But if your house is sold for $100,000 you'd pay $6,000 or $3,000 to Redfin. The whole thing is just the biggest currently running scam that I can think of and someone will come along and totally disrupt this entire industry of scammers :-)
I'm sure there are little things that go on behind the magic curtain, but definitely something that could be automated in a b2b fashion.
The worst part of it all - all that money is spent basically to protect the seller so they can wash their hands of the whole deal once the house is sold. There is very little to no buyer protection in the real estate world.
"We’re full-service, local agents who get to know you over coffee and on home tours, and we use online tools to make you smarter and faster. More than 10,000 customers buy or sell a home with us each year."
That's pretty much what any brokerage is. The only difference is they pay their people salaries. Any agent that can actually substantially sell and earn would strike out on their own and get their brokers license.
In less that 30 seconds, you can begin the process of listing your home with no contract or fees and includes MLS syndication.
The app auto-dispatches photography, drop-ships signs and lock-boxes, and starts the documentation management process.
At anytime you can reach out to your agent via phone, chat, or email.
When offers come in, they are displayed in "easy to understand" language and you can review then with you agent/listing account manager anytime.
When closing time comes, we dispatch a notary to the home and you exchange everything at the home. Super easy. Super convenient.
How we make money:
We retain a small fee at closing. When you go to buy your next home, we rebate that fee back to you when you use one of our vetted affiliate agents.
It is a win win. We keep the small fee and the agent "rebate" is essentially the acquisition fee for the guaranteed buyer. Agents don't waste their time working leads that will most likely not close (which is part of the reason why big commissions exist) and home sellers save thousands of dollars.
We operate in Colorado right now using brute force but V1 of our app will be released in January of 2017. We will be servicing Washington, Colorado, and Minnesota as of Februaury.
www.bluematchrealty.com
I basically agree, but I will say that when we bought our house and the (FSBO) seller turned out to be a flake and/or an outright nutcase, the realtor we were working with earned every penny of his 3%. I won't go into what he did for us because some of it may not necessarily have been legal for someone who's not an attorney to perform. Let's just say it took a lot of hustle to hold the seller to his commitments and make the deal happen according to the contract... especially when some of the flakiness turned out to be our own fault, due to the (in)actions of our mortgage lender.
Most of the other realtors I've dealt with fit your description perfectly. Nowadays, my thinking is that they're like airline pilots -- most of the time you can take them for granted and even think about how to replace them with automated processes, but when things start to go wrong, you may find yourself appreciating their work more than you thought you would.
https://stratechery.com/2016/opendoor-a-startup-worth-emulat...
https://www.opendoor.com
Here you can sell a house privately, and the only fees you need to pay are the solicitor for conceyancing - the fees are around the same when buying or selling, between £500 - £2000. They will just do the paperwork, check there aren't any issues with the deeds, and do minor contract negotiations, and register the sale.
The purpose of an agent here is really just to have a shop where they can advertise your sale, take clients and guide them around a house (but TBH I'd rather just look myself without an agent as they don't add any value in my experience), and do a bit of management between the buyer and seller. For this they'll usually charge 0.5% - 3% (to the seller), but if you want to do it yourself you can just advertise on sites such as Zoopla.
You are wrong about the amount of time she spent, and wrong about the industry being ripe for disruption. Spending 10 hours on four deals is not physically possible, or you have the luckiest, worst realtor in the history of the broker business.
Many have said the same thing about the industry, where are the results? Realtors are not still in business because of information asymmetry. Redfin has tried, and they have not succeeded. In fact they ended up moving towards the historical model. That tells you something.
I'll let you in on a secret, you can sell and buy any house you own or want yourself! It's called For Sale By Owner and many people do it. You can also negotiate on commission.
But be careful, because there are a lot of laws in place in RE, to protect buyers. It is heavily regulated. You could make a mistake and fail to disclose something that could end up costing you HUUUUGEEE.
What you get with a good realtor is a ton of service. Just the other day I drove 25 minutes each way to go make sure a homes thermostat was on and the house was warm when the temperature dropped. I was doing a favor for a realtor I know, that is selling a house for a family in another part of the country.
A good realtor will do things like that for you. And it's often a thankless job. The house I did that for might not even end up selling, the realtor may not even get a commission and it ends up being a lot of free work.
A good realtor knows attorneys, handy men, plumbers, electricians, landscapers and is a tough negotiator. A good realtor will market your property to a wider audience than would be easy for you to do yourself.
There are a lot of things that can go wrong and disrupt a sale, often times at the last minute. Those connections can save a deal. I've seen it countless times.
The reason the real estate broker business hasn't been disrupted is because it's a service business, it's not a product and its difficult to reduce it to something that scales and can be automated.
There is a large graveyard of startups that have tried.
I gave up my real estate license because of the sentiment you are expressing here, and the amount of competition that was willing to put up with it. It's a crap job in my opinion, it benefits the customers much more than the agents in all but the really hot markets.
"Spending 10 hours on four deals is not physically possible, or you have the luckiest, worst realtor in the history of the broker business."
In lots of cases there simple isn't anything to do. I saw the house, did my own research on it, went to see it, did more research, did more research and then called my Realtor and said "Make an offer." Is that worth $43,000? Why would she make $43,000 for few hours of work? I bought two houses this way, my first and last call to my Realtor (personal friend of mine) was "Here's our offer on the house, send it in." One offer was immediately accepted, the other one went back and forth 3 times over two days.
The simple fact that charging based on percentage of the sale is theft, there is no other way to put it. There is one other "entity" that does this and this is Government with Sales Taxes (also theft :-) ). Can you tell me one good reason why Real Estate Agents do not simply charge by the hour?
It's a solved problem to make a system that stores information about home listings and transactions. Other countries have done it. Yeah, houses are annoying because of all the weird combinations. But the thing is just a CRUD app.
This system still exists because of organizations that are based on information asymmetry. A small group of experienced people could make a really, really nice platform for sharing real estate data in a month and it would never get anywhere.
After that is the extremely entrenched business interest in the status quo - a lot of people are making money off the current system, and any real disruption is going to cost 95-99% of the real estate industry their jobs.
As noted elsewhere, the level of regulation is such that even offering better data to buyers/sellers has a lot of regulatory overhead, including physical offices with real people.
Right now I'm only pulling from a couple of different RETS services but already it's hellish. Regular short term schema changes, terrible lag in pulling data (10+ minutes to pull 30 listings!), low API limits (cut off for pulling image URL's too fast but they won't give a limit - just saying "if we feel it's too much", so pulling images lags for hours as I have to be extra careful. Also no option to pay extra to speed up). That's before you get to matching up fields across multiple MLS's or dealing with weird crossover effects (MLS X cannot be shown in Y from RETS #1 - buts that fine in RETS #2).
In fact, I will give you all my money right now anyway, I'm going to go live in the hills far away from it all.
A small team could easily collaborate with some law firm (maybe take an investment from a few law firms), and create some very valuable software.
This is an exceedingly hard problem unless 100% of your work is input through a revision control system.
Most of the time I spent selling my software was assuaging the risk-averse mindset that exists in the profession, rather than advertising the benefits. I actually gave a TEDx talk on this point as I think it's an enormous issue in the industry.
So the ease of creating the software isn't the issue, it's the culture. That said, legal tech now has a pretty healthy ecosystem which is a delight to see!
This is still a big problem in tech industry.
Yesterday my friend in LA invited a few of us in Toronto to come to his BBQ this weekend. The group chat became us discussing the logistics and price of how we might do that. 150 years ago, that would have been an insane concept.
http://www.imdb.com/title/tt0100802/
It's almost as though if we had a Procedural Consulting firm that could come in, look at the big picture, and help companies to create EFFICIENT and WORTHWHILE procedures that ACTUALLY do justice to the customer requirements without breaking the bank. Then they can sit everyone down and explain the procedures and enforce the assimilation that most companies usually have growing pains with.
I noticed that when moving to a new company that is trying to grow and achieve higher levels of accreditation. Management had intended a two-way assimilation to take place between my procedural knowledge and the procedures they already had in-place. What happened instead is old procedures are etched in stone and new ones are seen as an obstacle... The assimilation was one-way and the other middle-managers like myself are mainly concerned with keeping everything the same, despite there being improvements that could easily be made.
But another (part of the) solution is using workflow software. Something that allows you to define and share procedures, indicate status of various taskings along the defined workflows, move data and such along. This clarifies to the employees what's needed, and gives an opportunity for them to provide feedback for improvements (too much granularity, not enough, too inflexible, etc.). Management, then, also gets better insight into the actual status of projects and taskings. It's part of continuous improvement to constantly be evaluating these sorts of procedures and clarifying, culling, adding to them. Management that's unfamiliar with these concepts is just bad management.
In the manufacturing world, the Procedural Consulting firm would conduct "lean events". (NB: Many, if not most, places do this incorrectly.)
The correct way is to collect metrics (meaningful) over time, encourage and reward employee feedback about ways to improve things. Then improve them (sometimes as a move across the board, but often by conducting experiments to see how well the new concepts work, then roll out to everyone). The risk is that improved efficiency will obsolete employees, some fear this (like may have actual panic attacks about this level of fear). Instead, you need a culture that sees this improved efficiency as an opportunity for growth (the same number of employees can now do more once people get retasked to new things).
It sounds like your processes are either immature (no one really knows what the best way to do things is) or tribal (you have to know the right guy to talk to to find the information or knowledge you need).
Premature optimization is god awful, because it locks you into a world of pain - your business is inefficient and the people actually trying to follow the procedure find it incredibly painful and limiting.
If your process is immature, you need to invest some time and money into finding out what the best way to do whatever it is you're doing. If your process is tribal, you need to find the guy or guys that understand what's happening and pick their brains about everything they know. Their implicit knowledge needs to become explicit. In my situation, I found that just agreeing on common definitions was a huge help.
If you're grossly inefficient, don't write a procedure for a bad process...you're just ingraining inefficiency. Once you've found a good process, then write the procedure so you can train your people to repeat good, efficient practice.
And maybe most importantly, train your people to the procedure but make sure they understand it on a level so they know where it doesn't apply and give them authority to deviate from the procedure when necessary.
Sorry if none of that applies to you. I'm just speaking from my few limited experiences.
As someone in the supply chain and logistics industry, I can list for you hundreds of people that know the traveling salesman problem and precisely why its not applicable to their situation. I know hundreds of people that already know how to better manage their safety stock than someone who suggests using Gaussian demand models. I know hundreds of people who can optimize last mile delivery costs orders of magnitude better than a drone engineer. I know hundreds of people who can manage inventory distribution and ordering automation better than someone who knows databases.
And sure, there are companies out there that are doing everything ass backwards and could use some help, even if it is primitive and simplistic. And when they decide to look for it, who are they gonna choose: the guy who saved Amazon $500M/year with their truck load optimization expertise, or you, with your shiny website and a trick you learned from a textbook?
So as a piece of advice, if you aren't part of the industry already, don't try to do B2B in that industry. B2C is fine, because as a consumer you are ostensibly a part of the industry...but B2B is a death march.
In fact, this very attitude is the exact reason why there have been industries that are primed for "disruption." If only those within the industry, burdened with preconceived notions and patterns of thought, try to improve an industry, you are very rarely going to see revolutionary change.
You don't need to "disrupt" anything to be a very good business, but I really resent this "leave it to the pros" attitude.
I'd rather people feel slighted than to have a bunch of kids fresh out of college trying to disrupt, say, the health insurance industry.
This isn't about "fresh eyes". The problem with having no domain knowledge is that you can't identify many of the issues that needs solving within an industry. Every job I've ever had, I see issues that I can solve with my experience.
So they are imensly useful and many time more relevant than most other discussions about building companies.
Lately I've been approached by a lot of "startups" who have made it inadvertently apparent that they know about a serious problem in the industry but have no experience in the industry, are taking absolutely idiotic and long-discredited approaches to solving the problem, and don't even speak enough industry lingo to be understood. These are the people I tried to address with my comment.
If you'll notice, I was actually the top commenter on your original discussion [0]. I still stand by everything I've said in that comment. It is a potential multi-billion dollar idea, and the market size can be verified by pretty much anybody with basic market research experience. And while I have plenty of valuable and marketable experience in the Supply Chain and Logistics industry, even I am well short of qualified to lead a company trying to tackle that problem. I would definitely consider being a part of a team that was formed to solve the problem, but here's the thing: If you learned about the idea from an Ask HN thread, I almost certainly wouldn't join your team. Maybe if Elon Musk vouched for you and Marc Andreesen was throwing money at you. Maybe if you and 9 other committed engineers had a collective 100 man years of proven experience solving hard problems across a variety of industries. But trying to be the leader of a team trying to solve a problem that you didn't know about and don't fully understand is a huge obstacle to overcome. And I think it is perfectly reasonable of me to try to dissuade the average HN reader from even trying.
[0] https://news.ycombinator.com/item?id=9799851
The solutions to the problem in your industry might not be based on the skills or insights of your industry.
It might be that someone recognizes the similarity from another industry where the problem was solved in a specific way or that the insight just so happens to be solvable based on the knowledge of some specific technology.
And it's NOT about ideas it's about recognizing the real problem that hides underneath and which requires experience to understand.
Here is the essay I ended writing https://medium.com/black-n-white/the-problem-with-problems-4... this might give you a better perspective on how this is useful no matter what constellation.
The crux of your argument is that an outsider may be able to pattern-match better than industry insiders/domain experts. In other words, an outsider can recognize a potential solution by translating his/her experience with an efficient solution from a different industry.
He is saying that you need to properly understand things from the perspective of an industry insider before you declare that your solution is disruptive. In other words, you need to understand why things are the way they are before talking of reformation. This is the central lesson behind Chesterton's famous "Taking a Fence Down" [0] quote.
[0] https://www.chesterton.org/taking-a-fence-down/
Uh, no. The idea is that you identify the problem, build the team to solve it, build something quick that can prove that you can solve it in a better way than others, iterate to the point that CAC < LTV, scale.
Often they don't have the tech skills or desire/will to start something themselves, but are happy to join you if they recognize that it's a big enough problem.
So yes, a CS grad probably wont come along and revolutionise the Supply Chain industry, but mainly because SC has been at the forefront of areas of tech for decades. A CS grad may, however, come along and revolutionise...well just look at all of the recent famous tech companies...communications, travel, social, arts and crafts...whatever. Many areas are ripe for fresh eyes and new approaches.
I do agree that domain knowledge is often massively under-emphasised though.
Think somewhere on the order of 10,000 models per day throughput.
There's $BNs waiting for you. It's ridiculously hard.
And inside the object as well?
I'm not sure what you're asking here. Are you asking if it should be better than what can be done with photogrammetry?
And inside the object as well?
Doing just the outside is a big enough market/problem.
IMO it should be done with a mixture image segmentation and procedural generation.
Not sure what kind of datasets you're looking for. You'll see actual products to test with.
I've been studying and working with GANs for about a year now. They are still very exciting, and I'd love to try to expand my codebase to new types of data.
Additionally, there are some recent techniques that haven't been tried with voxel-based renderings.
Perhaps there is another algorithm that can help go from voxel -> polygons as well.
I think with the right tech, time, and execution this could be a matter of:
1. Take a picture
2. Generate until you get the 3d model you want
Well, not exact cause I don't like their voxel building generation method.
I think a GAN + Procedural Generator is the winner.
edit: Let me know if you want to work on this cause it's an active area of research for us. See my HN profile for contact.
Curious to know more about your train of thought. I am working as a researcher in the domain and thinking of experimenting with GANs for 3D model estimation using similar inputs as the one in the paper I referred to.
http://www1.cs.columbia.edu/CAVE/software/softlib/coil-100.p...
I was musing another kind of 'real world capture' with videogames, because I want to race around my neighborhood in Forza. https://hackernoon.com/dashcam-google-maps-dev-kit-custom-ne...
Maybe but I actually think that's the wrong approach.
I mean, to me it's kind of hard to believe nobody's tried making a "conveyor belt" like process inside a closed system
Yea they have - kinda. None of it works well or fast enough though. We put up a patent for one a year ago before I thought there was a better way to do it. The manpower required to move items onto/off of a line is a big part of the problem.
Taking that 10k number - assuming disparate types of items that might be part of a series like "Bathroom" (toothbrush, hair brush, toilet brush, plunger) - in 24 hours that means cycling each item through in about 8 seconds. The only way I remotely see that possible is essentially having a robot hand pick up the item at the entry point, hold it for the capture sequence (perhaps have a custom-designed 'mount' that can allow for true 360 via a couple positions), and then drop it out the other side.
It's the scale part I'm wondering about, re: one size machine fits all doesn't seem to make sense. One machine for items under a certain dimension (e.g. "hand held") then another for items where the machine has to essentially have super-powers to pick up and rotate objects to complete the imaging process (e.g. a couch, a dresser, a motorcycle, etc). I think trying too hard to accommodate outliers ends up tainting the balance of operations a little? Just thinking out loud, really cool puzzle.
But yea, there are a lot of us working on that.
http://www.intel.in/content/www/in/en/architecture-and-techn...
https://github.com/mapillary/OpenSfM
That said, some people have tried to use RS for this problem, but from what I've seen end up just using Kinects.
I assume that process would be easy to speed up if the requirement for absolute accuracy was removed. The 8' ROMER arm we use is accurate to ~!2 microns over its entire volume which is absolutely overkill for something intended to produce models for visual arts applications. A quick and dirty approach to generating the mesh might increase the inaccuracy by several orders of magnitude but when coke can has dimensional tolerances to the tune of tenths of a millimeter, the quick and dirty mesh will still be representative of the end product.
Who would be the primary customers? The entire 3D capturing market is currently several $B per year, including services. Where would be the customers that aren't getting served today that would double this market?
http://www.creaform3d.com/en
10K scans/day is way beyond their limits, and I'm not sure that's a very common use case. But I bet they could get there if they wanted.
That's been turned into an industry with very high throughput.
Stereo structured light is great, but doesn't work on specularly reflective objects. You've seen those amazing depth maps from the guys at Middlebury? Wonder how they get perfect ground truth on motorbike cowls that are essentially mirrors? Well they have to spray paint them grey so that you can see the light. The next problem is that you're limited by the resolution of the projector (so I guess if you own a cinema, yay!) and the cameras. Then you have to do all the inter-image code matching which sounds trivial in the papers, but in practice a lot harder (and since you don't get codes at all pixels you need to interpolate, etc, etc).
There are handheld scanners like the Creaform which work pretty well on small things, but I don't know what the accuracy is like.
The ultimate system would probably be a high-resolution, high-accuracy, scanned LIDAR system. Then you lose the problems with scanning ranges/depth of field, but you accept massively higher cost and possibly a much longer scan time for accurate systems.
I would suggest we make a monthly of these as they provide important insight into industries.
I got so exited about the thread and it made me realise something very interesting which I turned into an essay. I call it looking for hidden problems underneat obvious solutions.
https://medium.com/black-n-white/the-problem-with-problems-4...
Perhaps Quarterly?
Just imagine how much valuable knowledge and insights get lost every time someone retires.
I think part of the issue has mainly been a lack of education on what a poor diet consists of.
If anyone is interested, Dr. Gregor is a great resource of knowledge on this subject. He did a talk at google https://m.youtube.com/watch?v=7rNY7xKyGCQ which is based on his book "How Not to Die: Discover the Foods Scientifically Proven to Prevent and Reverse Disease" https://www.amazon.com/dp/1250066115/ref=cm_sw_r_cp_api_bEXs...
His website is http://nutritionfacts.org
All not for profit.
Some sort of attorney targeted simple revision tracking front end using git would be a mighty step forward.
One question: is there a use case for live group-editing? Same way live code editors work. I have to admit that I'm not super familiar of the workflows in that industry, so I am mostly just designing features the way I would like them to work.
How far along is the product? Have you talked to any lawyers about it?
The MVP is mostly targeted at businesses who have to sign/generate contracts at scale. There's a lot of features for those use cases, and then there's features for users that the parent is talking about (pure contract-drafting).
I have not talked to any lawyers yet. I think the worst part is that I don't even know how good the computer-skills of lawyers are, so... designing features by my competence feels wrong. Since I can design the UI to match the general structure of a contract, there's a lot that I can add to make the user's life easier, but it also adds more for the user to learn.
Most folks don't get started, so congrats on that, but can I say please do talk to some lawyers asap.
I know it is super painful but, believe me, it can save you months if not years of going down the wrong direction.
Here is a blog post we put together to show how to get in touch with people to interview:
https://blog.nugget.one/2016/09/21/case-study-how-i-got-25-c...
Apologies for the unsolicited advice :)
My understanding of the workflow is that they will send a clean copy as well as the redline to each other. Sometimes someone will in response further modify the wrong copy, and send back, which causes problems as it becomes unclear what was accepted and what was not and someone has to manually go through and check things. I hear that complaint come up - so I know there is at least one pain point involved.
http://thomsonreuters.com/en/products-services/legal/large-l...
that said it doesn't look all shiny and new.
The point of the service in the music industry is to inspire new sounds and the device can either be kept or given away to somebody without a lot of second thought. Getting stuck is a big problem. Also in music it's important to collaborate and giving gifts is a good way to make connections and impressions.
Great tie-in with various Manufacturers or even retailers to get rid of excess stock / failed impulse buy items / etc.
Honestly I'm pulling up memories now of watching other people's kids over the years playing with various music stuff I've handed over (Korg Monotron synths were the biggest hit) and those are really pleasant to recall!
Subscription boxes are a thing :-)
Edit: probably without android.
My management style is like this: every task/request is numbered, placed in a queue and assigned to a professional.
What I expect from my ticketing system:
- every manager should be able to assign tasks to someone and set the order they must be executed. He needs know what his team is doing and when they finish each task. - every professional should know what to do and what are the priorities. - everything is numbered and linked, all communication recorded.
Everything should be well integrated with email (please, don't send me a notification email about an answer and an url, send me the f* answer). If I answer the email, everything goes into the system, I should be able to send commands to the system by email (for example, add a keyword in order to make it a comment instead of answering).
http://www.storytotell.org/constraints-and-semantics/
In a nutshell, I argue that the problem with most ticket systems is that they do not constrain the domain enough, so they wind up having similar problems to email (sifting through a chronologically-ordered pile of text rather than structured, semantically-ordered information).
Your comments make me think the crux of the problem is that people want tickets to be like email and use email to manage them. I'm not sure you can ever overcome the "chronological pile-up" problem if you allow email as a user interface to ticketing.
In fact, my usual approach to dealing with tickets/issues/emails which start to develop this problem is to make my own private copy of the thread and edit it in precisely this manner, though I'm the only one this benefits since it doesn't get sent back upstream.
I still think there is something here though. Stack Overflow replaced message boards, which were basically HTML versions of mailing lists, and part of that was identifying the semantics of question, answer and comment and defining new operators and new expectations for them.
A wiki is a good approach but because it's totally free-form, the user gets stuck doing the work of keeping things hygienic.
JIRA allows you to edit all the properties of a ticket whenever, but it generates such a huge cloud of email notifications in the process, it kind of disincentivises you from using it. And nobody is in the habit of rereading the page to see what is different since last time.
I agree that's partly it, but that seems ok when you're in the thick of discussing a problem/fix. If you're doing a code review or something after a fix has been pushed, you actually want certain messages to stand out to describe resolutions and whatnot.
So like gmail where you can star/mark certain replies as important and those messages would show up at top-level in the ticket, where all other messages are collapsed.
Full disclosure: I'm part of the maintainer staff.
[1] https://github.com/zammad/zammad/ [2] https://zammad.com
Major feature that allows me to work around any shortcomings in your office: API access to everything and/or database access (preferably direct read/write access, but even if it's just a downloadable .sql.gz it's a huge benefit).
I'm probably not a typical user, though, FWIW.
Personally, I think the optimal ticket system would have this data for each ticket:
* A unique, prefixed ticket # (JIRA gets this right)
* An assignee (like an email To:)
* A reporter (like an email From:)
* A one-line summary (like an email Subject:)
* A multi-line body (like an email body, but ideally with markdown)
* Attachments (like email attachments)
* History for edits of all of these (not like email!)
That's it! It really is basically email, but with a unique ID, and editable with history instead of immutable with replies, and a decent UI, perhaps RSS + notifications.
Unfortunately, everybody else seems to think that their ticketing system should embody their vaguely defined and ever-changing workflow, prioritization, approval, and release management system, so they want to be able to add any number of possible statuses, approvals, workflows and and all the rest. Once you add that, you end up with another JIRA or ClearQuest or BugZilla, and the cycle repeats itself.
As is (consequently) friction it creates in changing the workflow as needs change.
Thats just the fundamental and immutable nature of the problem domain.
[0] http://www.redmine.org/
www.asana.com
I'm not associated with them, but I have used them successfully for months at a time (better than most productivity software). The reason is it is well integrated and similar to email.
The recent GitHub updates let you assign multiple people to reviews and such, but I find it's usually better to tag everyone you want to look at something. I don't think assigning something will send a notification.
For my latest startup I went looking for a service desk tool. The key criteria was "feels like email". The moment any alternative required a user signup just to lodge a support request, I ruled it out.
I ended up choosing Groove. I don't recommended it. All ticketing systems suck, this one just sucked the least for my support desk. Groove doesn't extend to other ticket types, and it's nowhere near as flexible or extensible as JIRA, and the mobile experience is horrible. But it does "feels like email" for my customers better than every alternative you care to mention.
That sounds like unnecessary micromanaging. You couldn't possibly have enough detailed knowledge to know the proper order of tasks in all cases. Possibly even most cases.
I agree that communicating the priorities are important, but the boots on the ground have a much better understanding of what they're working with than you do.
I don't know why the few companies that are in the IoT space for oil and gas aren't scooping up literally billions in missed opportunities for sticking cell-enabled sensors on oil platforms, fields, etc. Then there's the next billion dollars waiting for whoever starts sticking controllers next to the sensors on valves, etc.
Drone inspection and repair in oil and gas. People are doing this but it's taking way too long to take off for how much money is there just waiting to be picked up.
Sticking a laser scanner on a drone. Again, people are doing it, but what the fuck, there's so much money just sitting there.
If you're looking for contract work, just start browsing random EPC websites and calling up the shitty ones. Probably a good 10,000 at least that are still rocking 1995 crapsites, and not in the "good" "low functionality low load time" way, the "using tables for layout" way.
Trains should be automated. They already are in Taiwan for some lines, it's been feasible for years.
Motorcycle safety is still subpar for where we are in material science. There are a lot of riders out there that will pay buckets for greater safety. We've figured out how to not get our skin ripped off but I believe there's still a market for preventing broken bones, spinal snaps, decapitations, and the like.
Somehow we still don't have GUI HUDs in our moto helmets. Like BMW is working on something but honestly it would be a relatively simple and profitable thing for a very small startup (2 man team working part time). Literally even just casting your smartphone screen to the visor would be enough to get people buying so they can have a HUD map and shit.
Big market for motorcycle storage solutions. I giggle whenever I see someone with an ammo box strapped onto their sports bike. We already dropped hundreds on gear and thousands on the bike, we spend many more hundreds or thousands on modding the things, there's ample opportunity for more elegant and functional storage solutions.
VR allows for limitless desktop screen space in a portable package. I'd like to be able to bring an HTC vive and a tiny screenless box to plug it into that would allow me to have a "multi-monitor" setup while I travel. Some say the resolution isn't there yet, I say make the text bigger. I have no problem reading the stuff in steamVR.
I get made fun of every time I bring it up but I'm convinced people are stupidly ignoring lighter than air travel, transportation, and data distribution, especially in an automated sense. Google has their wifi balloon thing but they dropped their blimp transport truck project. I think it could've been a thing.
There may be some margins available in teaching low-income people how to cook and eat basic foods instead of frozen meals. It took me to getting scholarshipped into college to realize that we were losing buckets of money eating frozen meals and fast food because we thought it was the "cheapest option," not to mention how unhealthy we were for it. Think like somehow getting low-income folks to buy potatoes, onions, peppers, dried beans, cheap cuts of meat, etc and demonstrating how it's faster, cheaper, healthier, etc. Potentially a gov funding opportunity, would save on EBT and healthcare costs.
Kids learn by doing. Good luck changing anything about education in the USA though.
Someone would be able to take over any industry in Taiwan that they please if they start up the company and instantly pay 2x local salary, give PTO, and have other basic benefits that we take for granted here. It wouldn't be much, you'd be paying ~44k/year USD for an Engineer, for example. You'd be able to poach the best talent, you'd draw shitloads of negative press from pissed of old Taiwanese bus...
EPC means "engineering, procurement, and construction" company. So like, lyondell-basell or jacobs engineering or foster wheeler (now AMEC) or Mustang (not the car) or KBR.
PTO = Paid Time Off. Catchall for vacation time, sick time, holidays.
GP offices: Plenty of solutions already - generally regionally based or by payer. Also an issue with vets, hairdressers etc - pick an appointment based niche. (Have invested in 2 niches, advised others)
Wireless sensors for industry: Thousands of players here, including network providers and different tech stacks. The game is to find a niche that is easy to enter and pats well. (Have invested in wireless water metering, advised others)
Drone inspection: Plenty of players here, but it's a find-a-niche that you can sell to play, and you need operators for each application. (Have advised drone and drone application companies)
Anything on a drone: Drone's don't last long in the sky. (as above)
Trains: Trains get solved at the political level. Good luck in the USA. (am train lobbyist)
Motorbike safety: Check out where Dainese and BMW are going - air bags et al. I spend a fortune on BMW gear. (Motorcyclist)
GUI HUDs: The main player failed (Skully - internal issues) . I know of at least one early stage company attempting a generic HUD solution. (have met, not invested the early stage one and decided, luckily, that I didn't like the Skully people and did not pre-order.)
Motorbike storage: Has been a huge market since the 90s when you could only get adventure boxes from one supplier - in Munich. Now the problem is solved by countless suppliers and manufacturers - unless you buy a sports bike, which have a very different use-case. My first adventure trip was in 98 and I needed to self-assemble, now you can ride out of several branded dealerships with a capable round the world bike. (have motorbiked round what world - in pieces)
VR: The use-case is really only games so far. See Playstation. (have 2 VR kits)
Lighter than Air: Capacity and cost versus shipping are the issue, while air-cargo is well solved. Meanwhile the Helium market is stuffed. But I love the industry too. (read Bill Gate's recommended book on international shipping etc.)
Low income cooking: Those low income people are time-poor, poorly educated (the US system is dreadful) and marketed too by monster franken-food companies that price under the cost of decent food. Yes - its a hard problem to solve. (don't get me started)
Education: Indeed. Start with politics and how schools are funded.
Taiwan: (sadly) Why would you pay more than $1 over market?
http://aidomes.com/tiny-homes/
Regulations will unbundle research from liquidity provision starting sometime in 2018 (moving regulatory target, but theme seems clear). Sellside will have lower incentive to pay large research groups; buyside will have to pay explicit fees for research advisory. There is a big opportunity in providing platforms, with macro and market data live, where researchers can interact with capital managers, given that said researchers will likely find themselves bereft of their current distribution networks (bank sales forces) sometime in the next few years.
Equities already have a hybrid form of this where buyside earns "credits" to be allocated at the end of each year to research providers. But fixed income is at least twice as big as equities, is much more opaque because is essentially unlisted (mainly "OTC" = "over the counter" ie only those in the know), and therefore much more susceptible to disruption.
Providers today offer uncapped bandwidth, some, like Amazon, without any cost cap, some, like KeyCDN, allow to set a cost cap, but take the resources off-line when the limit is exceeded.