Ask HN: Which industry sector should I target?
My interests within the field is not what one would call narrow. Primarily though, I have a fondness for programming language theory, compilers, parsing and writing my own DSLs so that I can make powerful compile-time assertions regarding my code as well as generate tests automatically from the DSL itself.
My StackOverflow cv has an additional set of tasks that pertain to my interests, http://stackoverflow.com/cv/filipallberg.
Regarding my primary interests, the only relevant "thing" I have come up with is web-scraping, but I find it is a dull task as it is a trivial matter to me.
Machine vision is also something that I think a lot about, however most of my ideas are consumer-facing utilities and not something to build a business around. If I get a good idea I'd be willing to go into business myself.
62 comments
[ 3.4 ms ] story [ 118 ms ] threadedit spelling
http://research.microsoft.com/en-us/um/people/simonpj/Papers...
We're even hiring: https://donsbot.wordpress.com/2016/01/18/haskell-developer-r...
I personally enjoy the knowledge of having solved a real (business) problem for real people, and being appreciated both financially and personally for this, even when the actual tech going into this isn't very complicated, and especially, doesn't tick off a lot of "hip" boxes.
I have also met good people who do not thrive in these circumstances, so it's hard to give generic advice as to what might make you happy.
How do you do it, as a consultant or as a 9-5 job? Any advice on getting started as someone who graduated 2014?
I don't have any particular advice, beyond the generic stuff: You'll need a network of people who might (know someone who might) want to pay you to help them achieve business outcomes. Remember that you're selling business results, make sure every line on your invoice is part of a narrative that makes (or reduces risk of losing) money for the business.
Of course, in both cases, your work is ultimately directed by revenue, the shield I'm talking about is more against the tight, client/user responsive iterative process of most modern software development which can make it difficult to carve out the time and space to work on something deeper.
Also, I'm not convinced academia is much of a panacea here: I hear there is a nontrivial number of years of pretty severe drudgery before you arrive in the sort of position with a lot of freedom to work on whatever you like.
* There is considerable structural sharing inside finance regulations, but it is not hierarchical
* Regulations may be changed at any time, and these changes may be future-dated or back-dated
* Users insist on arbitrarily complex exceptions and overrides
* Nontechnical users (particularly legal) will insist on being able to vet and introspect
In my experience, anyone can do this badly, but to make it scalable and robust requires subtlety and a good understanding of formal logic, ontology and DSL design. It's a good fit for FP. Oh, and there's plenty of room for machine learning: Automated discovery, natural language processing, noncompliance detection, etc.
The problem of turning regulations into a expert system that is robust against future changes has not been solved, and there's demand for such systems from even the most sophisticated financial companies, even though everyone expects they'd be able to build such systems in-house.
Droit Financial Technologies http://www.droitfintech.com
I know what kind of tools are available, I know how dead easy debugging is. There is nothing nearly comparable in complexity with a hardware design.
Writing the architecture for a new Intel CPU is much more difficult than writing a calculator on an FPGA or designing a circuit board.
Similarly, designing and maintaining the UX of Amazon.com is a lot more difficult than a personal website.
Is Physics 'harder' than Psychology just because it's 'more hardcore' or 'more pure'?
Apples to oranges. You cannot really make comparisons like that.
I did not say a word about design, I was talking about development. UX design is probably among the most complex problems out there, since it is not solved yet and there is even no glimpse of a distant hope.
> Amazon.com
Well, this one was obviously designed by a drunk cobbler, and he could not have spent more than half an hour overall, judging by how miserable the user experience is.
> Is Physics 'harder' than Psychology just because it's 'more hardcore' or 'more pure'?
Physics is solvable, Psychology is not, the latter is in an early phenomenological state of development. So, on this scale, it is harder to apply it (because one have to operate with a much bigger amount of information), but it is much easier in terms of models.
> You cannot really make comparisons like that.
Of course you can. Complexity is a very well-defined and objective thing.
I've seen people who are great at diving deeply into a narrow field struggle greatly with the broad, changing knowledge needed for web dev, especially at scale. It's not clear to me that one is intellectually more challenging than the other. They often seem like different types of intelligence.
I do not consider any area where this is possible to do (i.e., any area where complexity is artificial, not inherent) to be really complex and challenging, unlike the domains where complexity cannot be eliminated by simply removing the stupidity.
And stupidity got this funny property - it attracts even more stupidity, they stick together, they grow like a snowball, consuming everything around. Ill designed web standards, exceptionally, incomprehensibly shitty language (javascript) - of course it should have created tiers upon tiers of ill designed frameworks, ladders of leaky abstractions and a broken culture of tolerance towards all this shit.
The only sane way to deal with this kind of crap is to shield from it entirely, introduce a simple and clean design which should interact with a broken system, without allowing any of the underlying abstractions to leak to your level. It always works, with all kinds of broken APIs, and I cannot see why web should be any different.
Btw., it's amusing that web devs got so offended by quite an obvious fact that their work is nowhere near in complexity with the hardcore realtime stuff.
I find it not amusing that people can assume things they don't participate in are simple, stupid, and trivial. It's the same type of intellectual closemindedness that leads to major problems in human existence. It's a mindset that deliberately ignores complexity in order to justify preexisting belief and prop up belief as truth while protecting the ego.
Personally, I enjoy working on hardware and working on distrubted systems. I also like building things that look beautiful. But to each their own, I guess.
"The ability to reduce everything to simple fundamental laws does not imply the ability to start from those laws and reconstruct the universe...The behavior of large and complex aggregates of elementary particles, it turns out, is not to be understood in terms of a simple extrapolation of the properties of a few particles."
https://web2.ph.utexas.edu/~wktse/Welcome_files/More_Is_Diff...
This is actually what science is all about - finding an information in data, or, in other words, measuring the complexity. If you can reduce a complex set of data to a simple algorithm, then this algorithm is all the knowledge that is there, and the rest is just a noise. By definition.
What Phil Anderson is talking about is a different kind of complexity, which is sort of irrelevant in this context. Yes, knowing the fundamental laws won't help you to explain, say, a protein folding, not without hundreds of years of a CPU time. But it does not matter. It does not make this science more "complex".
I do not "assume" that web technologies are stupid. I know they're stupid and I can prove they're stupid. Where shall I start? Javascript? All of the W3C issued mess? All the damage left from the browser wars?
This guy said it all much better than I can: http://beauty-of-imagination.blogspot.co.uk/2016/01/tcltk-vs...
When you have something to compare with, something as simple and beautiful as Tk, all the complexity of the modern web looks like a steaming pile of stupidity.
Do you find anything useful in the compile to js (or even html/css) world like Elm, PureScript, etc.? Do these provide clean wrappers to broken systems?
I certainly find the entire front end world (html/css/js and various compilation of those) painful, and I would definitely enjoy a better paradigm.
It's all about overengineering. Tk demonstrated that all this stuff is fundamentally easy. But the modern web technologies are overengineered to such an extend that anyone touching them draw back in fear.
So, yes, now the only way to put all that complexity under control is exactly this - things that compile to js (+ css + html), that can hide the overengineered parts of the underlying ecosystem and only expose simple, obvious interfaces. PureScript is a right thing.
One thing that can be harder to overcome is being trapped by processes that you're not allowed to change (and the kind of institutions that make CPUs are particularly prone to that kind of failure, IME). But even then, it just becomes a case of adding as much value as you can within those restrictions, and that can be fulfilling in its own way. But yeah, some organizations (in any industry) are really badly managed and if you can't live with that then your only recourse is to leave.
PureScript is the right thing indeed, I like the approach, but what exactly makes PureScript so special? I quickly skimmed through, expecting something cool and innovative like, say, gradual typing, but it looks like something fairly standard, I could not find any particularly new ideas there. What did I miss?
What I like about PureScript is the careful typeclass hierarchy - it feels like a rethinking of the Haskell standard library from the ground up. They often end up with a hierarchy of single-method typeclasses, which feels like a first step to something along the lines of http://degoes.net/articles/principled-typeclasses/ . I'd like to combine that with Idris-style full dependent typing but you can't have everything and there's a tension between what's state of the art and what's designed for practical use (I feel like I'd be happier using PureScript in production than Idris). Gradual typing I don't find particularly interesting; YMMV.
Design is not only a complex problem, it is an unsolved problem. Is there any passable design out there? I have not seen it yet. It is either disgusting, incomprehensibly disgusting, or, in the best case, barely tolerable. And the same thing is going on in any kind of an industrial design, it is not limited to web.
As for the type class hierarchy, yes, I did not notice this bit, thanks. I'll take a closer look now.
Like any other kind of programming, if parts are trivial you automate them, until you're at a state where what you have to do to accomplish a given task is precisely the amount of thought/uniqueness that that task involves; if you're a programmer then you never need be doing mindless drudge work (you've got a computer for that). When you've automated most of what used to be your job, reaching up or down the stack is a good way to find more things to automate (or you can simply increase throughput, but often that just means other parts of the business become the bottleneck). IME it's if anything easier to get closer to the "full brain all the time" state with web programming, because there's so little external latency, whereas in hardware there's only so much you can do before you're spending most of your time waiting for slow test cycles or actual manufacturing (even if you try to reduce the importance of those by introducing things like formal methods, others will often want to see a few test cycles to prove the effectiveness of those).
No, it's a different kind of a "design". It's an engineering. You've got a set of constraints and need to optimise a solution until all the constraints are met. Purely mechanical.
Designing something for a human interaction is different. It involves extorting information from a not necessarily cooperative and not very sane human, reducing it to a reasonably sane subset and then finding out the workflows that would make sense for a living human being. Unsolvable. Impossible.
Got nothing in common with something as simple as "here's a high level formal description of an algorithm, turn in into a code".
> whereas in hardware there's only so much you can do before you're spending most of your time waiting for slow test cycles or actual manufacturing
In my book, it's a blessing. Time to slow down and reflect on what a mess you've just created.
For example, Embedded World in Nürnberg, Germany provides a good cross section of industry that has embedded engineering involvement in it. There will be a similar one for telecommunications, Automotive etc.
Good luck on searching for your calling.
I would suggest you leverage your interest in web-scraping and use it to customers in what ever niche you believe you can sell them something.
Collect your "beta" users through web-scraping, figure out a way to reach them at scale. Build/sell a product they want (you can leverage your information advantage to figure out what this is)
This is personally what I do and have created/worked on many bootstrapped companies over my career. Niche for me is fairly irrelevant, as long as I have a pool of interested customers ahead of time. I primarily use my information advantage to figure these out.
The additional skills you will need is:
* learn to think as a "user/customer"
* minor copy-writing skills (or at least understand what shitty copy is and how to improve it)
* UX
* Data mining & Analytics, Analytics, Analytics
* A/B Testing, Iterations, Incremental Improvements (see point above)
* Hypothesis Driven Development (https://www.thoughtworks.com/insights/blog/how-implement-hyp...)
My one suggestion is to avoid any projects which do not have a clear monetization strategy. If you're following my blue-print companies that don't make money from day will only incur costs as you reach your pool of customers at scale.
If you find web-scraping dull don't pursue it. Life is short.
Damn Small Linux, a very small Linux distribution
Definitive software library
Domain-specific language, a computer language designed for a specific problem domain
As a consumer though, augmented reality ( eg Hololens and Magic Leap) seems like it is about to become very big, and probably utilizes machine vision.
Can you recommend some resources for learning about web scraping?
Industries that will have specific uses for machine-vision: autonomous driving cars, drones, robotics, augmented reality, internet of things
Others: crypto-currencies (specifically digital ledgers/blockchains), virtual reality, commercial space exploration, online education, voice recognition, (more as I think of them)
Tech that's already big and probably getting bigger: streaming video, instant stock trading based on algorithms,
Certainly one of those must be interesting to you. They're all pretty interesting to me.
1. Where developers themselves are the likely product users (think Google Analytics, New Relic, Optimizely, Appdynamics)
2. Where there is a direct and immediate connection to $$$ (finance / quant stuff)
3. Generic data science platforms (Cloudera, Hortonworks, Databricks)
I think there is still tons of opportunity for industry-specific turn-key data science tools. Especially for verticals where there is a low coincidence of skilled developers with industry domain expertise.
My personal recommendation would be to find people who are doing interesting things and talking about it in public - blog posts, conference talks, etc etc and reach out to them.
I can make 1 USD per account per month, at 10k accounts it is a pretty good source of passive income.