Gain compression is a reduction in 'differential' or 'slope' gain caused by nonlinearity of the transfer function of the amplifying device. This nonlinearity may be caused by heat due to power dissipation, or by overdriving the active device beyond its linear region. It is a large signal phenomenon of circuits.
Gain compression is relevant in any system with a wide dynamic range, such as audio or RF. It is more common in tube circuits than transistor circuits, due to topology differences, possibly causing the differences in audio performance called "valve sound". The front-end RF amps of radio receivers are particularly susceptible to this phenomenon when overloaded by a strong unwanted signal.
Audio effects 
A tube radio or tube amplifier will increase in volume to a point, and then as the input signal extends beyond the linear range of the device, the effective gain is reduced, altering the shape of the waveform. The effect is also present in transistor circuits. The extent of the effect depends on the topology of the amplifier.
Differences between clipping and compression 
Clipping, as a form of signal compression, differs from the operation of the typical studio audio level compressor, in which gain compression is not instantaneous (delayed in time via attack and release settings).
Clipping destroys any audio information which is over a certain threshold. Compression and limiting change the shape of the entire waveform, not just the shape of the waveform above the threshold. This is why it is possible to limit and compress with very high ratios without causing distortion.
Limiting or clipping 
Gain is a linear operation. "Gain compression" is not linear and, as such, its effect is one of distortion, due to the nonlinearity of the transfer characteristic which also causes a loss of 'slope' or 'differential' gain. So the output is less than expected using the small signal gain of the amplifier.
In clipping, the signal is abruptly limited to a certain amplitude and is thereby distorted in keeping under that level. This creates extra harmonics that are not present in the original signal.
"Soft" clipping or limiting means there isn't a sharp "knee point" in the transfer characteristic. A sine wave that has been softly clipped will become more like a square wave, with more rounded edges, but will still have many extra harmonics.
Compression of gain is caused by non-linear characteristics of the device when run at high levels. With any signal, as the input level is increased beyond the linear range of the amplifier, gain compression will occur.
A transistor's operating point may move with temperature, so higher power output may lead to compression due to collector dissipation. But it's not a change in gain; it's non-linear distortion. The output level stays relatively the same as the input level goes higher. Once the non-linear portion of the transfer characteristic of any amplifier is reached, any increase in input will not be matched by a proportional increase in output. Thus there is compression of gain. Also, at this time because the transfer function is no longer linear, harmonic distortion will result.
Intentional compression 
In intentional compression (sometimes called automatic gain control or audio level compression) as used in devices called 'dynamic range compressors', the overall gain of the circuit is actively changed in response to the level of the input over time, so the transfer function remains linear over a short period of time. A sine wave into such a system will still look like a sine wave at the output, but the overall gain is varied, depending on the level of that sine wave. Above a certain input level, the output sine wave will always be the same amplitude.
Radio frequency compression 
"Gain compression" in RF amps is similar to soft clipping. However, in narrowband systems, the effect "looks" more like gain compression simply because the harmonics are filtered out after amplification. Many data sheets for RF amplifiers list gain compression rather than distortion figures because it's easier to measure and is more important than distortion figures in non linear RF amplifiers.
In wideband and low frequency systems, the non-linear effects are readily visible, e.g. the output is clipped. To see the same thing at one GHz, an oscilloscope with a bandwidth of at least 10 GHz is needed. Observing with a spectrum analyzer, the fundamental compressed and the harmonics picking up.
Examples of RF compression 
A low-noise RF amplifier if fed by a directional antenna to a consumer 900 MHz receiver should improve the transmission range. It works, but the receiver may also pick up a couple of UHF stations around 700 MHz.
For example, if channel 54 is transmitting 6 MW of AM, FM, and PM, the RF front end, expecting −80 dBm, would be grossly overloaded and generate mixing products. This is a typical effect of gain compression.