Veracode has a new blog post, A Tale of Two Compilers, about differing behavior when two compilers are faced with a subtle buffer overflow. It’s somewhat tangential to the main point, but I noticed that even though the compilers Veracode tested had stack overflow protection enabled, neither detected the bug or prevented the exploit. Detection and prevention of precisely this bug was a headline feature of the original ProPolice implementation. The version of gcc used in OpenBSD has changed several times since then, so I tested it to make sure it still works.

code

#include <stdio.h>

int main(int argc, char *argv[]) {
    int x = 0xabad1dea; // the best number.
    char buf[32];

    scanf("%32s", buf);
    printf("%s 0x%x\n", buf, x);

    return 0;
}

The bug is that scanf will overflow buf and write a single nul byte into x, changing the value to 0xabad1d00. As Veracode explains, whether this actually happens depends on padding and alignment. Whether the stack protector can detect the bug also depends on whether the compiler reorders local variables so that overflows are more likely to be detected.

results

> gcc -o scanf scanf.c
> ./scanf
1234567890123456789012345678901
1234567890123456789012345678901 0xabad1dea
> ./scanf
12345678901234567890123456789012
12345678901234567890123456789012 0xabad1dea
Abort trap

As soon as we overflow the buffer by a single byte, the bug is detected. Also notice that the value of x has not been corrupted. On OpenBSD the stack protector does a little more work. Larger variables and arrays like buf are moved to the top of the stack frame, above variables like x. This prevents the typical overflow from hitting them. The stack protector cookie is also placed directly after the end of the last variable, with no padding. Even if you change the size of buf to 33, it will be aligned such that a one byte overflow can be detected.

caveats

The stack protector can only work if the function returns. Had we called exit instead of returning from main, the bug is not detected. The stack protector can only do so much to arrange buffers. If there are two char arrays in the function, only one of them will be next to the overflow detecting cookie.

fallout

Whenever we add a new program correctness security measure (stack protector, malloc assertion, mmap randomization) to OpenBSD, one of the things we check for is that existing programs stop working. We know that existing software has lots of bugs. If nothing breaks, it means the feature isn’t working.

update

I should have mentioned I was working with the beloved i386 architecture. On amd64, the test fails without a little more love.