Disabling the NX bit for specific apps
The NX bit is a feature of recent AMD and Intel CPUs that helps prevent the most common overflow attacks.When enabled and supported by the operating system, buffer overflow attacks are far less likely to succeed.
When it was first introduced, a few applications would not run when the NX bit was present (e.g. Java and .Net runtimes, Skype) but most vendors recompiled their binaries and all was well.
Sadly there are still a few remaining apps that still won’t run… mostly niche commercial apps compiled long ago that, while still for sale, are not actively maintained by the vendor.
When there is no choice but to run these apps, the usual response is todb. disable the NX bit. Most system BIOS have a such setting, and under VMware there is an option to hide it from individual virtual machines. However, disabling NXÂ increases the vulnerability of the entire operating system and every other app running on it.
Fortunately in Linux you can usually disable NX only for the specific applications that can’t handle it, leaving it enabled for the rest of the server.
The execstack command can mark most shared libraries and executables so they ignore the NX bit. It’s a standard part of Red Hat and derived distros (CentOS, Oracle Linux). For Ubuntu / Debian you can find it in the prelink package from the Universe repository.
Finding the executables and shared libraries to mark with execstack can be a chore, depending on the application. The ldd command can help, as can strace and gdb. When in doubt, mark every ELF binary and .so file in the legacy app directory tree… it doesn’t seem to hurt (though you should back up the original files before doing so).
Some files cannot be marked with execstack. From the man page:
execstack doesn’t support yet marking of executables if they do not have PT_GNU_STACK program header entry nor they have room for program segment header table growth.
In my experience, almost all apps can be marked and will run fine, leaving NX bit protection enabled for the rest of the server.
Obviously, without NX enabled the legacy app will be more vulnerable to overflow attacks. Also consider you had to resort to using execstack because the vendor hasn’t yet fixed a bug in their code present since the NX bit was introduced in 2004! What other flaws have they ignored? To be prudent, put additional safeguards around the app, especially if it communicates with untrusted users or networks… for example, using chroot or writing an SELinux or AppArmor policy to contain the app might be a good idea.
Related posts: