Why is encryption so hard?

If you take a narrow focus on a particular cryptographic event (such as your encryption of a string with an RSA public key) then you miss the greater story about encryption: it's not just the individual cryptographic primitive that needs to be implemented correctly, it's everything else.

An RSA encryption like that does not stand alone. Keys must be generated, secured and distributed. The RSA library itself must be validated to ensure that it works correctly. The actual primitive must be used correctly (in the case of RSA don't use a stupid exponent as some have done). And the environment within which the encryption is used must be understood and secured (just look at the CRIME and BREACH attacks against TLS to see how something 'secure' can be broken because of something apparently irrelevant, in this case, compression).

The overriding reason that encryption is 'hard' is that secure computer systems have enemies and those enemies (attackers) will do _anything_ to attack the system. They will attack it based on timing, compression problems, flaws in the protocol, freezing the RAM to extract a private key, etc. etc. There's really no end to the variety of things you can try to attack a cryptosystem.

So, building a secure system may have encryption as a necessary condition, but it's not sufficient. So much else can and will go horribly wrong.

If you are interested in this hit the books and understand the history of cryptography. For example, look at how Vigenere was broken by Babbage, or the Venona ciphers, or Lorenz. These 'old' ciphers can tell you a lot about how people actually attack things. Then read about modern ciphers and attacks on them. Wikipedia has much. Read about TEMPEST and imagine other attacks possible in that way.

The results of this is things like the developer who used "1" as the multiplication factor, so to decrypt the data, you need to divide each block by 1...

e.g. easiest to use :: SSH with password <<>> SSH with passphraseless keys <<>> SSH with passphrase-protected keys :: most secure

For example, a common mistake is to assume that by encrypting something, attackers can no longer change it. Or perhaps you'll use your standard equality operation to check whether a decrypted string matches some value, without thinking about timing attacks. Or maybe you'll just use AES in ECB mode.

It's also very hard to figure out if your encryption is bugged or not. I guess that for most us, once your method returns a hash, you expect that everything is secure.

On a side note, I wonder how many people on HN would claim to know the inside out of encryptions. (Not the difference between SHA1/MD5/bcrypt but the actual math behind derivations and how they work)

- Simply encrypting your message as indicated will not protect you from replay attacks. Someone could record your message and re-transmit.

- Simply encrypting your message will not assure that the contents haven't been modified, someone could patiently sit in the middle poking bits to see what happens.

- Most encryption schemes will require you to choose a block cypher, doing so requires some knowledge of the options and the data you're sending. Some handle large amounts of data poorly, others fail when you send identical messages.

- Most encryption schemes will require you to initialize them with truly random data, both an early version of Netscape, and Debian messed something up and provided far less entropy than they appeared. Relying on /dev/urandom on a machine that's just booted, or otherwise faulty entropy providers is fatal.

- Attackers can record your data and play with it forever, so even if a mistake or attack isn't revealed for years, they can still go back and decrypt your data. I believe the NSA broke the Russian's use of a One Time Pad because they re-used pages years later.

- Simply encrypting data doesn't provide assurances that you're communicating with the system you think you are, the initial contact is still tricky.

So there's more to it than a single function call.

  http://nacl.cr.yp.to/features.html
 
  High-level primitives

  A typical cryptographic library requires several
  steps to authenticate and encrypt a message.
  Consider, for example, the following typical
  combination of RSA, AES, etc.:

  * Generate a random AES key.
  * Use the AES key to encrypt the message.
  * Hash the encrypted message using SHA-256.
  * Read the sender's RSA secret key from
    "wire format."
  * Use the sender's RSA secret key to sign the
    hash.
  * Read the recipient's RSA public key from wire format.
  * Use the recipient's public key to encrypt the
    AES key, hash, and signature.
  * Convert the encrypted key, hash, and signature to wire
    format.
  * Concatenate with the encrypted message. 

  Sometimes even more steps are required for storage
  allocation, error handling, etc.

  NaCl provides a simple crypto_box function that
  does everything in one step. The function takes the
  sender's secret key, the recipient's public
  key, and a message, and produces an authenticated
  ciphertext. All objects are represented in wire
  format, as sequences of bytes suitable for
  transmission; the crypto_box function
  automatically handles all necessary conversions,
  initializations, etc.

Somewhat unique to security and cryptography are the number of subtle bugs possible. There are both problems of actual "normal" bugs (like the Debian entropy bug) and system level design errors (like CRIME).

NaCl/Salt tries to reduce the number of errors possible by using the library wrong (as opposed to eg: openssl that has a very (some say too) rich interface). But you could still end up writing the secret key to swap. Or doing something silly with the plain text. Or expose yourself to a buffer overflow in the part of the code that renders those cute avatar-images for your chat application.

edit: formating

There's a massive market for easy-to-use encryption. Easy-to-use does not imply insecure in any way at all.

A good crypto library should keep your data safe for decades. We don't make the same demands (no bugs, due to no updates possible) of other software that often.

Also, proper key management is out of reach for most of us.

To repeat what others have said in answer to your more general question - solutions to "real world" problems include more than a single call to a primitive. So you need to find libraries that provide a higher level API, like parts of NaCL http://nacl.cr.yp.to/, Google's keyczar http://www.keyczar.org/, etc.

Even for simply encrypting a string with a password - https://pypi.python.org/pypi/simple-crypt which is what I talk about in the first link - I needed three things: key strengthening, the encryption itself, and an HMAC. Making those work well together was harder than I expected (at least 5 bugs harder...)

The overarching problem is that you don't really get any feedback about whether what you're doing is right or wrong. For example, no cryptographer would use RSA like that, but that's not obvious just from studying the wiki article. Or from looking at the function output - it does turn ASCII into gibberish, as advertised, and that's where most developers will call it a day.

The moving parts are also treacherous. You're not just going to encrypt a string - someone is meant to decrypt it. Have you authenticated the ciphertext? Are you exposing a padding oracle? Or timing attacks? Are messages susceptible to replay? In crypto systems, these things are equivalent to locking the front door and leaving the window wide open.

In practice, most insecure crypto constructions aren't due to bugs in the implementation of RSA or AES. They're because of developers choosing inappropriate primitives, gluing them together incorrectly, or inadvertently exposing dangerous side channels.

Fortunately, there are libraries that can help. As mentioned elsewhere, NaCl/Sodium and KeyCzar provide higher-level interfaces that can abstract away many of these issues.

From their homepage:

Crypter crypter = new Crypter("/path/to/your/keys");

String ciphertext = crypter.encrypt("Secret message");

I think what we need, for email at least, is a completely new protocol that's end to end secure (as hard as that is). The problem though is that I don't think something like this can be done anymore, without "interested" corporations co-opting or talking it to death. The golden age of the internet is gone.

A given: all software has bugs. Usually, that doesn't matter — a CRUD app will eventually get debugged enough to the point of usability. (Sometimes even maintainability.) We do not understand enough about programming to guarantee perfect execution in all cases, but no one gains any value by causing an obscure input case to cause a null pointer exception.

Whenever we use crypto, however, we inherently have code which protects something valuable: from forum passwords to credit card numbers to state secrets. This means that all the subtleties which break in ordinary code, but no one cares about, suddenly become important. Every interaction of input to memory to processing to storage (to network) must be scrutinized for places where a crucial piece of data may leak an encryption key, or perhaps just enough known plaintext of known cyphertext to mount an attack.

1. For a high value target like Edward Snowden, there is a broad spectrum of attacks, and any operational weakness is fatal. There are many examples of these attacks described on this thread. Unless you know what Snowden knows, odds are you will not get it right.

2. BUT, if everyone had easy encrypted email and real time communication, the mass surveillance machine would be blinded, because the kinds of attacks that are used against high value targets do not scale up well.

disclosure: I'm a co-founder of LaunchKey