HN Explorer

Ask HN: What set your hair on fire in 2012?

48 points by diego ↗ HN
I'm curious to know what serious technological / IT problems you encountered in 2012. What things made you work late at night, scrambling for a solution? EC2 outages did it for some people. What else? What failed unexpectedly? For what problems do you still not have a solution, and just hope for the best?

57 comments

[ 2.7 ms ] story [ 111 ms ] thread
pmtarantino
Online payments systems in South America.
sghill
Timezones in javascript. We ended up stitching together 3 libraries into something that sort of looks like 10% of joda-time. Definitely not a solved problem.
OmarIsmail
Which libraries did you end up using?
mrharrison
Yeah, which ones, I also have timezone hair pulling for the front end.
mrharrison
So far moment.js seems to be the best solution for me.
PStamatiou
Getting require.js to play nicely with rails 3 asset pipeline on heroku. It would all work fine locally but compiling js always had some dependency issue. Finally got it working some week or two later.
glyphobet
Jython. It's an unfinished research project/toy/proof of concept that never should have been used in a production environment under any sort of load. Unfortunately my predecessors made the wrong choice and I inherited that technical debt.
seunosewa
Can't you easily rewrite the critical parts in Java? (That should be much easier than writing C extensions for CPython.)
shaunxcode
well for me "hair on fire" means "what were you really inspired about/by" to the point you stay up late working on it. For me:

ML for the working programmer - started down this road as I wanted to make sure I really "get" what is being expounded upon by haskell/shen etc.

shen/klambda - every time I dive into this bottomless pool I come back up with a new insight to how programming could/will someday look.

amber smalltalk - close to my ideal collaborative programming environment - the new helios ide and compiler progress bring it even closer!

edn - seriously I don't think enough hype has been made about what edn can bring to the table, specifically w/ "tagged" data. Stayed up one night and wrote jsedn so I could start to use edn in projects.

datomic - There are no silver bullets in database programming, but datomic is damn close. Trying to wrap my head around the specifics culminated in prototyping datomicism, my smalltalk esque environment for working w/ datomic.

minecraft - which has led to working on a full blown 2d/3d schematic system (stay tuned for http://schematic.io) and a rediscovery of "primitive" computer geometry (namely "geometric principles and procedures for computer graphic applications" by Sylvan H. Chasen). This will hopefully serve as a nice open source example of how to meld coffeescript/backbone/datomic and a edn based DSL for helping kids "code by osmosis" as it follows the bret victor principle of showing the code change as the objects are manipulated (and vice versa).

eflowers
Yes stay tuned for schematic.io - I am doing the UI and UX for this w/ shaunxcode.

What sets my hair on fire is coupling UX with BDD and the rapid development chain of realtime UX and programming. Having UX write the UI cucumber scenarios helps a lot. But this is for pet projects. I'd really like to see the world of business accept and start trying out using the UX process as a precursor to BDD and allowing devs to work more freely on what they are good at with a constant pump of UX feeding them the things they don't want to think about.

Simucal
Can you describe a bit more about schematic.io? I quick google search didn't turn up much. What is this going to be?
shaunxcode
Basically a CAD tool for minecraft in which you can create schematics for structures in a 2d editor w/ a live 3d preview. The "use case" is for sharing a complex build plan with multiple users on a "layer by layer" basis so that many people can collaborate either in real or elapsed time. e.g. I may start building a castle and leave it w/ a sign in the center giving the schematic.io url so that another user can come along and make progress on it according to said schematic.

Right now this works for "block" structures e.g. there is not yet the ability to place redstone wires/circuits - but that is coming soon. The use case there being to increase the level of complexity and radness users can build w/o having to "frame by frame" a youtube video on building complex piston doors!

falcolas
Trying to find the formula to help companies with no MySQL experience (yet running mission critical MySQL instances) get stable, redundant, & backed up fast.

There's nothing like being paged 10 times an hour at 11pm to get you motivated to get things fixed.

biturd
When people in tech say "paged" do they/you literally mean it? Or do you just mean text message but the terminology has stuck as a throwback to the days when pagers were how people were on emergency call.

I suspect there still is a small pager business out there somewhere, perhaps for doctors and adhering to some legacy hippa rules, but they must be sparse.

e1ven
I carry around an honest-to-goodness old-school pager. I love it. It works even when I get next to no cell signal, it works when I screw up the newest ROM of my phone. It works when I'm out of battery. It runs weeks on a AA.

I have messages go to SMS as well, but when the servers are down, damn it, I need to know.

falcolas
It's a combination of growl/SMS/email. The terminology has stuck around (and I know doctors still use actual pagers for their reliability).

And given our issues lately with our phone carrier silently blocking our messages as spam, the thought of going back to an old school pager for reliability is still there.

ovi256
Not knowing how to research 'hair on fire' problems other people have, so that I could create a solution, add value, make money, one-man-startup style. I know it's kinda meta.

Reading Kalzumeus and other entrepreneurs opened the door a little. Lean startup pushed me along the way as well.

edw519
Funny, the "new" technological problems I think OP is referring to are rarely things that are that difficult to deal with.

But these are the things that continue to keep me up at night year after year:

  10. How clever to get with javascript in the browser without going overboard.
  9. How much javascript to depend upon on the client. 
  8. Optimistic or pessimistic database locking?
  7. Do it at compile time or run time?
  6. Do it on the client or the server?
  5. Keep data on the client or make the round trip?
  4. Mouse, keyboard, either one, or something new?
  3. Which browsers and/or platforms to support?
  2. Simplicity vs. capability in the user interface.
  1. How to deal with difficult/stupid people
thetabyte
The prevalence of #1 on your list (among other reasons) are why I always have and always will believe that improvements to education yield by far the most return on investment vs any other allocation of resources. Improving education throughout the world is the single best way to increase economic output.

Now, things I did not say:

* "Put more money into our education system as it is."

We are in need of vast improvement, and we should be investing time and money towards innovation and improvement.

* "Stop funding and/or stop using our existing educational system."

Yes, there are many, many flaws, but our educational systems have improved our society a great deal, and are fantastic avenues and methods of distribution for educational innovation.

* "Edtech is the best industry!"

I'm still unsold on the edtech space as a way of improving education. Particularly, I fear that our existing models for business, when applied to education, will exclude the vital underpriveleged demographic who have some of the most potential for growth. Aditionally, much of the edtech I see used in schools does little to actually aid student learning—but it certainly does look good on a quarterly newsletter where administrators list what fantastic things they're doing for their school. Whoever figures out how to make equitable, effective educational technology that turns a profit will be one of the preeminent innovators of our time.

(Note: If you believe you or your company are doing this, I have Ruby on Rails skills, am willing to learn any technology you work with, and I want to work for you.)

In sum: The act of improving the effectiveness of the systems which themselves improve the effectiveness of human capital yields exponential and undervalued economic gains. I hope in the future it receives more focus.

jt2190
The problems as you've listed here have no context, which will make them extremely difficult to answer, if not impossible.

For example:

  > How clever to get with javascript in the browser 
  > without going overboard.
Depends completely on whether anyone other than you needs to read/maintain the code base, what kind of advantages the "cleverness" brings (performance, maintainability), etc.
skmurphy
edw519 has not listed problems to be solved but dilemmas/trade-offs to be managed.
Achshar
Sending payments via Paypal. Yes, I understand it is very simple, but Paypal has restrictions in my country. And i only have debit card and no credit cards. And debit cards don't work online apparently. I can connect my back account but that requires my pan info (like SSN) and a bank confirmation. My PAN info is still not confirmed. It surprises me how difficult it can be to send money online when you are in a particular country. No only that but I was unable to use my debit card anywhere on the entire internet. I wanted to buy windows 8 and had to use my uncle's credit card who does not live in this country. Had to figure out a way to send him money too. It's ridiculous how you become a second class citizen if you don't have a credit card. This is 2013 (in my timezone it is) and there has to be a better way to transfer money.
AYBABTME
Prepaid Visa/Mastercard cards work and they're pretty much the equivalent of a debit card. You put money on them and use them to buy stuff online.
Achshar
AFAIK we don't have them here. Even if we had they won't work online due to government restrictions. I live in India and they had a few problems with how paypal can be used to launder money.
[deleted]
(comment deleted)
Ixiaus 

    1. Authorize.net's ARB (integrated with it before really good solutions came into view)
    2. A chaotic - but good - life (difficulty in regaining my daily routine due to startup + girlfriend + other commitments)
riffraff
facebook APIs. Random breakages, incomprehensible changes, unanswered PRs etc. And AWS outages. And Heroku outages. And OAuth integrations.

Basically, everything not 100% under my control keeps going wrong. And so does everything under my control, but apparently causing smaller fires.

bradhe
Scaling. Oh god. Scaling.
yesimahuman
My biggest pain in the ass this year was as simple as disk space constantly running out on our VPS servers. That, and automating postgres backups.
themckman
How did you automate your postgres backups? I'm going to be tackling this for a client soon and have a few ideas, but would love to hear how you did it.
yesimahuman
Well, first of all I mistakenly assumed our VPS backups would be sufficient, but the first step is actually dumping the database every interval you'd like. If you don't dump it, you might end up with a corrupt database should you be relying on the VPS backups.

Then, we set up a process to zip the database dump and send it off to tarsnap. We delete the backup done 30 days ago each time we run.

It works fine for us, but I much prefer using Heroku and not having to worry about it :)

heriC
We put all our databases on a mapped /data drive that is larger and allocated in the san for this reason. I'm not the sysadmin and the policy isn't mine, but it seems to work....
drakeandrews
By an large, candles. Candles and long hair are a bad combination.

On a serious note, a piece piece of software called Player/Stage which I am working with is driving me up the wall. Learning how three.js and the collada spec work to try and get the two playing nicely. Cross-browser CSS3/HTML5. Weirdness troubleshooting a flash app running with a PHP backend on a hundred android 1.6 tablets you don't have code access to.

heriC
Chef Scripts. But now that I have Vagrant the fire is out. Seriously. Vagrant is amazing to the point of being nearly a silver bullet like Rails was. Funny thing was, when I first looked at it, I couldn't even figure out why I would want to use it, but now I can't live without it.
ChrisNorstrom
Wordpress E-commerce plugin. Stay. Away. From. It.

There's a reason why it has such terrible ratings and reviews and complaints. It has a lot of features that the others don't, and that tricks you into thinking it's right for your online store. Only to be 1 month in, and start discovering all the terrible programming, glitches, bugs, php errors, and short commings.

Stay away from WP E-Commerce. You've been warned.

mashmac2
Especially if your theme, or anything close to your wordpress install (other plugins, etc) use any javascript libraries that might share variables or have different version numbers. Agreed.
malinens
Refactoring one of my old websites (200,000 lines of code). Code was horrible. I think in 5 years I will tell the same story about my recently refactored code =)
ojr
trying to deploy node.js on a private VPS running on CentOS, to say I was in over my head is an understatement but I did find a new appreciation for sysadmins
gsiener
I was shocked to find out that the node package for CentOS is not actively maintained anymore...
adventured
After building online business for the last 16 years, I took serious venture capital for the first time in 2012. So managing that relationship and meeting someone else's expectations has shaped my 2012 (set my hair on fire). I have a very patient investor, and it's still not easy, I don't know how people deal with the real sharks of VC.
albertoavila
Sencha Touch + Phonegap, as it turns out it's not very pleasant to try to make a good looking responsive multiplatform app, specially with android browser weirdness and bugs.

Lesson learn, if you care about the user experience there is no other way than native.

meaty
What killed me:

Ms11-100 security patch. Utter disaster that broke our entire cache and UI layer. Literally our kit jumped from 10% utilisation to 90% overnight resulting in a last minute rollback. Didn't show up in load testing.

business analysts lack of grasp on reality "its only a few billion rows" (to download our audit data via HTTP in CSV format - not going to happen).

SQL Server 2012 pricing.

Mark shuttleworth being an ass over a amazon integration causing us to move all our Linux stuff to debian.

VMware 2tb lun limit on iscsi. This sucks badly.

Compilation speed of c# - this is knackering us. Java is much better as it doesn't have to create monolithic dependency units (assemblies).

Exploding kit.

kenguish
How to combat frauds with e-commerce transactions:

1. Pro-actively block Tor nodes and all hide-my-ass type of open proxy. To e-commerce merchants, tor node users are not freedom fighters, but fraudsters, every-single-one-of-them.

2. For credit card transaction, always check the return "credit card" country

3. Shopping carts need to track IP address. Fraudsters always can bounce from Brazil to UK to Italy in a single cart session.

4. Have to come up with a score system to determine fraud risk of transactions. Maxmind API seems to be helpful.

5. When in doubt, ask user to take a picture of their credit card, and upload to your site.

jabo
You really cannot be serious about asking a user to take a photo of their credit card and upload it.
lostlogin
As bad as this solution is, I'd choose this form of user identification (id photographed and uploaded) over the last 5-10 captchas I have done. They have been killers for me lately, and one took a day of on and off trying to get past.
raquo
I'm from Russia and I tell you he's serious. I've been asked that a few times.
kenguish
The only other alternative recommended by payment gateway, would be to ask you to fax your passport. The point is, to prove that you really do process the card, not getting it through some shady sites.

Try registering an hosting account on many "managed hosting" company, you are required to fax your passport to them. I will be offended if somebody asks me to do so.

Uploading credit card image is definitely less evil than faxing your passport.

kaffeinecoma
This year I switched to Ruby on Rails after years of doing mostly Java. I'm quite happy with it, but by far the biggest pain point has been Ruby's dependence on native code libraries. Dealing with CLASSPATHs and Maven was never fun, but it sure beats trying to maintain a working set of Ruby gems, many of which have cascading dependencies on C++ compilers, Python interpreters, SSL libraries, Javascript runtimes, etc.
JayNeely
Finding freelancer peers to sub-contract and refer work to. When you're overflowing with work, it's more difficult than ever to find people through events / traditional slow networking, and there's no good online alternative.

I wish there were a site that wasn't transaction-focused, that provided a way to find people with specific skill sets, and that provided search / profile data that was specific to freelancers vs job seekers (e.g. what's your hourly rate? what's your availability? are you looking for sub-contracting / referrals / collaborations?). And most importantly, that connected to the social graph and provided a good reputation system. If the people I've worked with before aren't available, the next best is people they've worked with before, and rated positively.