About two years ago, OpenSSL introduced a new feature that you’ve never used or even heard about until yesterday, after somebody discovered a bug that could be used to read process memory.
The main heartbleed site has a decent amount of information, but no detailed description of the bug. For that, read Diagnosis of the OpenSSL Heartbleed Bug. Here’s also a short pseudo version, for reference.
packet lives somewhere in the heap and we’re going to copy some user specified amount of data from there into a reply buffer. The user specified length is safe as it’s only 16 bits, and the destination buffer is always large enough. The danger comes from the fact that the input buffer may not be large enough, causing the reply to contain information it should not. If the previous contents of the memory now occupied by packet were interesting, then the reply will also be interesting.
What if the previous contents weren’t interesting? For example, what if malloc overwrote the previous contents of memory before reusing it? As in, exactly what malloc.conf does with the J option. Then the attacker would get a buffer full of 0xd0, which is decidedly uninteresting. But...
There’s always a but. Unless libssl was compiled with the OPENSSL_NO_BUF_FREELISTS option (it wasn’t), libssl will maintain its own freelist, rendering any possible mitigation strategy performed by malloc useless. Yes, OpenSSL includes its own builtin exploit mitigation mitigation. Of course, you could compile your own libssl with that option, but...
When I built libssl without internal freelists, nginx linked against that library would sporadically and erratically refuse connections. Both Firefox (nss) and ftp (libssl) would report errors, such as:
1007048992:error:1409442E:SSL routines:SSL3_READ_BYTES:tlsv1 alert protocol version:libssl/src/ssl/s3_pkt.c:1255:SSL alert number 70
At this point, I gave up. (And... then I figured out what’s happening.)
Guard pages should also prevent copying too much data beyond the input packet buffer. Hit an unmapped page -> boom. I haven’t been able to trigger this; apparently the buffers in libssl are always large enough? Assuming so, this would mean you are restricted to reading previously freed memory, but not any memory in active use. Of course, even casual testing reveals that previously freed memory contains lots of interesting data, like http headers and form data. (Determining exactly how large the input packet buffer is in libssl was deeper into the abyss than I cared to delve. Caveat indagator.)