||This article needs additional citations for verification. (April 2009)|
A return-to-libc attack is a computer security attack usually starting with a buffer overflow in which the return address on the call stack is replaced by the address of a function that is already loaded in the binary or via shared library. This allows attackers to defeat the Non-Executable Memory stack protection - i.e. a page cannot be marked as write and executable at the same time. In fact, in this way the attacker simply calls preexisting functions without the need to inject malicious code into a program. However, this technique will fail if ASCII-Armoring protection is enabled. Indeed, with ASCII-Armoring, all the system libraries (e.g. libc) addresses contain a NULL byte. Nevertheless, this same concept can be used in a similar and more advanced attack known as return-to-plt, where instead of returning to libc, the attacker uses the PLT functions loaded in the binary (e.g. system@plt, execve@plt, sprintf@plt, strcpy@plt, etc...).
The shared library called "
libc" provides the C runtime on UNIX style systems. Although the attacker could make the code return anywhere,
libc is the most likely target, as it is always linked to the program, and it provides useful calls for an attacker (such as the
system() call to execute an arbitrary program, which needs only one argument). This is why the exploit is called "return-to-libc" even when the return address may point to a completely different location.
Protection from return-to-libc attacks 
A non-executable stack can prevent some buffer overflow exploitation, however it cannot prevent a return-to-libc attack because in the return-to-libc attack only existing executable code is used. On the other hand these attacks can only call preexisting functions. Stack-smashing protection can prevent or obstruct exploitation as it may detect the corruption of the stack and possibly flush out the compromised segment.
ASCII-Armoring prevent this particular kind of attack. In fact, with ASCII-Armoring, the system libraries use the first 16 MB in memory, which includes a NULL byte. However, ASCII-Armoring does not prevent from return-to-plt.
Address space layout randomization (ASLR) makes this type of attack extremely unlikely to succeed on 64-bit machines as the memory locations of functions are random. For 32-bit systems ASLR provides little benefit since there are only 16 bits available for randomization, and they can be defeated by brute force in a matter of minutes.
Related attacks 
return-to-plt is a similar attack where, instead of returning to libc, the attacker uses the PLT functions loaded in the binary (e.g. system@plt, execve@plt, sprintf@plt, strcpy@plt, etc...).
Return-oriented programming is an elaboration of the techniques used in this attack, and can be used to execute more general operations by chaining individual smaller attacks that execute a small number of instructions at a time.
See also 
- Buffer overflow
- Stack buffer overflow
- Stack-smashing protection
- No eXecute (NX) bit
- Address space layout randomization
- Return-oriented programming
- Shacham, Hovav; Page, Matthew; Pfaff, Ben; Goh, Eu-Jin; Modadugu, Nagendra; and Boneh, Dan. "On the Effectiveness of Address-Space Randomization". Proceedings of Computer and Communications Security (CCS'04), October 25–29, 2004, Washington (DC).
- Bypassing non-executable-stack during exploitation using return-to-libc by c0ntex at InfoSecWriters.com