In computer security, a covert channel is a type of computer security attack that creates a capability to transfer information objects between processes that are not supposed to be allowed to communicate by the computer security policy. The term, originated in 1973 by Lampson is defined as "(channels) not intended for information transfer at all, such as the service program's effect on system load." to distinguish it from Legitimate channels that are subjected to access controls by COMPUSEC.
A covert channel is so called because it is hidden from the access control mechanisms of ultra-high-assurance secure operating systems since it does not use the legitimate data transfer mechanisms of the computer system such as read and write, and therefore cannot be detected or controlled by the hardware based security mechanisms that underlie ultra-high-assurance secure operating systems. Covert channels are exceedingly hard to install in real systems, and can often be detected by monitoring system performance; in addition, they suffer from a low signal-to-noise ratio and low data rates (on the order of a few bits per second). They can also be removed manually with a high degree of assurance from secure systems by well established covert channel analysis strategies.
Covert channels are distinct from, and often confused with, legitimate channel exploitations that attack low-assurance pseudo-secure systems using schemes such as steganography or even less sophisticated schemes to disguise prohibited objects inside of legitimate information objects. The legitimate channel misuse by steganography is specifically not a form of covert channel.
Covert channels can tunnel through secure operating systems and require special measures to control. Covert channel analysis is the only proven way to control covert channels. By contrast, secure operating systems can easily prevent misuse of legitimate channels. Distinguishing these is important. Analysis of legitimate channels for hidden objects is often misrepresented as the only successful countermeasure for legitimate channel misuse. Because this amounts to analysis of large amounts of software, it was shown as early as 1972 to be impractical. Without being informed of this, some are misled to believe an analysis will "manage the risk" of these legitimate channels.
Lampson's definition of a covert channel was paraphrased in the TCSEC  specifically to refer to ways of transferring information from a higher classification compartment to a lower classification. In a shared processing environment, it is difficult to completely insulate one process from the effects another process can have on the operating environment. A covert channel is created by a sender process that modulates some condition (such as free space, availability of some service, wait time to execute) that can be detected by a receiving process.
The TCSEC defines two kinds of covert channels:
- Storage channels - Communicate by modifying a "storage location"
- Timing channels - Perform operations that affect the "real response time observed" by the receiver
The TCSEC, also known as the Orange Book, requires analysis of covert storage channels to be classified as a B2 system and analysis of covert timing channels is a requirement for class B3.
Identifying covert channels
Ordinary things, such as existence of a file or time used for a computation, have been the medium through which a covert channel communicates. Covert channels are not easy to find because these media are so numerous and frequently used.
Two relatively old techniques remain the standards for locating potential covert channels. One works by analyzing the resources of a system and other works at the source-code level.
Eliminating covert channels
The possibility of covert channels cannot be completely eliminated, although it can be significantly reduced by careful design and analysis.
The detection of a covert channel can be made more difficult by using characteristics of the communications medium for the legitimate channel that are never controlled or examined by legitimate users. For example, a file can be opened and closed by a program in a specific, timed pattern that can be detected by another program, and the pattern can be interpreted as a string of bits, forming a covert channel. Since it is unlikely that legitimate users will check for patterns of file opening and closing operations, this type of covert channel can remain undetected for long periods.
A similar case is port knocking. In usual communications the timing of requests is irrelevant and unwatched. Port knocking makes it significant.
Data hiding in OSI model
As Handel and Sanford take a broader perspective and focus on covert channels within the general design of network communication protocols. They employ the OSI model as a basis for their development in which they characterize system elements having potential to be used for data hiding. The adopted approach has advantages over these because standards opposed to specific network environments or architectures are considered. Foolproof steganographic schemes are not devised.
Rather, basic principles for data hiding in each of seven OSI layers are established. Besides suggesting the use of the reserved fields of protocols headers (that are easily detectable) at higher network layers, Handel and Sanford also propose the possibility of timing channels involving CSMA/CD manipulation at the physical layer.
The work by them identifies covert channel merit such as:
- Detectability: Covert channel must be measurable by the intended recipient only.
- Indistinguishability: Covert channel must lack identification.
- Bandwidth: number of data hiding bits per channel use.
The covert channel analysis presented here, however does not consider issue such as interoperability of these data hiding techniques with other network nodes, covert channel capacity estimation, effect of data hiding on the network in terms of complexity and compatibility. Moreover, the generality of the techniques cannot be fully justified in practice since the OSI model does not exist per se in functional systems.
Data hiding in LAN environment by covert channels
As Girling first analyzes covert channels in a network environment. His work focuses on local area networks (LANs) in which three obvious covert channels (two storage channel and one timing channel) are identified. This demonstrates the real examples of bandwidth possibilities for simple covert channels in LANs. For a specific LAN environment, the author introduced the notion of a wiretapper who monitors the activities of a specific transmitter on LAN. The covertly communication parties are the transmitter and the wire trapper. The covert information according to Girling can be communicated through any of following obvious ways:
- By observing the addresses as approached by the transmitter. If total number of addresses a sender can approach is 16, then there is a possibility of secret communication having 4 bits for the secret message. The author termed this possibility as covert storage channel as it depends in what is sent (i.e., which address is approached by the sender)
- In the same way, the other obvious storage covert channel would depend on the size of the frame sent by the sender. For the 256 possible sizes, the amount of covert information deciphered from one size of the frame would be of 8 bits. Again this scenario was termed as the covert storage channel.
- The third scenario presented is pertaining to the existence sends can be observed by the wire trappers to decipher for instance “0” for the odd time difference and “1” for the even time difference.
The scenario transmits covert information through “a when-is-sent” strategy therefore termed as timing covert channel. The time to transmit a block of data is calculated as function of software processing time, network speed, network block sizes and protocol overhead. Assuming block of various sizes are transmitted on the LAN, software overhead is computed on average and novel time evaluation is used to estimate the bandwidth (capacity) of covert channels are also presented. The work paves the way for future research.
Data hiding in TCP/IP Protocol suite by covert channels
A more specific approach is adopted by Rowland. Focusing on the IP and TCP headers of TCP/IP Protocol suite, Rowland devises proper encoding and decoding techniques by utilizing the IP identification field, the TCP initial sequence number and acknowledge sequence number fields. These techniques are implemented in a simple utility written for Linux systems running version 2.0 kernels.
Rowland simply provides a proof of concept of existence as well as exploitation of covert channels in TCP/IP protocol suite. This work can, thus, be regarded as a practical breakthrough in this specific area. The adopted encoding and decoding techniques are more pragmatic as compared to previously proposed work. These techniques are analyzed considering security mechanisms like firewall network address translation.
However, the non-detectability of these covert communication techniques is questionable. For instance, a case where sequence number field of TCP header is manipulated, the encoding scheme is adopted such that every time the same alphabet is covertly communicated, it is encoded with the same sequence number.
Moreover, the usages of sequence number field as well as the acknowledgment field cannot be made specific to the ASCII coding of English language alphabet as proposed, since both fields take into account the receipt of data bytes pertaining to specific network packet(s).
The Data Hiding in TCP/IP Protocol suit by covert channels have following important aspects:
- Identify the existence of covert channels in a network environment.
- Point to devising satisfying techniques of embedding and extraction processes at the source and destination, respectively.
- Do not consider the effect of employing covert communications network as a whole.
- Lampson, B.W., A Note on the Confinement Problem. Communications of the ACM, Oct.1973.16(10):p. 613-615. 
- NCSC-TG-030, Covert Channel Analysis of Trusted Systems (Light Pink Book) from the United States Department of Defense (DoD) Rainbow Series publications.
- 5200.28-STD, Trusted Computer System Evaluation Criteria (Orange Book) from the DoD Rainbow Series publications
- Timing Channels an early exploitation of a timing channel in Multics.
- Covert channel tool hides data in IPv6, SecurityFocus, August 11, 2006.
- Covert Channels in the TCP/IP Protocol Suite, 1996 Paper by Craig Rowland on covert channels in the TCP/IP protocol with proof of concept code.
- Raggo, Michael; Hosmer, Chet (2012). Data Hiding: Exposing Concealed Data in Multimedia, Operating Systems, Mobile Devices and Network Protocols. Syngress Publishing. ISBN 978-1597497435.