An autokey cipher (also known as the autoclave cipher) is a cipher which incorporates the message (the plaintext) into the key. There are two forms of autokey cipher: key autokey and text autokey ciphers. A key-autokey cipher uses previous members of the keystream to determine the next element in the keystream. A text-autokey uses the previous message text to determine the next element in the keystream.
In modern cryptography, self-synchronizing stream ciphers are autokey ciphers.
The first autokey cipher was invented by Girolamo Cardano, and contained a fatal defect. Like many autokey ciphers it used the plaintext to encrypt itself; however, since there was no additional key, it is no easier for the intended recipient to read the message than anyone else who knows that the cipher is being used. A number of attempts were made by other cryptographers to produce a system that was neither trivial to break nor too difficult for the intended recipient to decipher. Eventually one was invented in 1564 by Giovan Battista Bellaso using a "reciprocal table" with five alphabets of his invention and another form was described in 1586 by Blaise de Vigenère with a similar reciprocal table of ten alphabets.
One popular form of autokey starts with a tabula recta, a square with 26 copies of the alphabet, the first line starting with 'A', the next line starting with 'B', etc., like the one above. In order to encrypt a plaintext, one locates the row with the first letter to be encrypted, and the column with the first letter of the key. The letter where the line and column cross is the ciphertext letter.
The autokey cipher as used by the members of the American Cryptogram Association is in the way the key is generated. It starts with a relatively short keyword, and appends the message to it. So if the keyword is "QUEENLY", and the message is "ATTACK AT DAWN", the key would be "QUEENLYATTACKATDAWN" 
Plaintext: ATTACK AT DAWN... Key: QUEENL YA TTACK AT DAWN.... Ciphertext: QNXEPV YT WTWP...
The ciphertext message would therefore be "QNXEPVYTWTWP".
Using an example message "meet at the fountain" encrypted with the keyword "KILT":
plaintext: MEETATTHEFOUNTAIN (unknown) key: KILTMEETATTHEFOUN (unknown) ciphertext: WMPMMXXAEYHBRYOCA (known)
ciphertext: WMP MMX XAE YHB RYO CA key: THE THE THE THE THE .. plaintext: DFL TFT ETA FAX YRK .. ciphertext: W MPM MXX AEY HBR YOC A key: . THE THE THE THE THE . plaintext: . TII TQT HXU OUN FHY . ciphertext: WM PMM XXA EYH BRY OCA key: .. THE THE THE THE THE plaintext: .. WFI EQW LRD IKU VVW
We sort the plaintext fragments in order of likelihood:
unlikely <------------------> promising EQW DFL TFT ... ... ... ... ETA OUN FAX
We know that a correct plaintext fragment will also appear in the key, shifted right by the length of the keyword. Similarly our guessed key fragment ("THE") will also appear in the plaintext shifted left. So by guessing keyword lengths (probably between 3 and 12) we can reveal more plaintext and key.
Trying this with "OUN" (possibly after wasting some time with the others):
shift by 4: ciphertext: WMPMMXXAEYHBRYOCA key: ......ETA.THE.OUN plaintext: ......THE.OUN.AIN by 5: ciphertext: WMPMMXXAEYHBRYOCA key: .....EQW..THE..OU plaintext: .....THE..OUN..OG by 6: ciphertext: WMPMMXXAEYHBRYOCA key: ....TQT...THE...O plaintext: ....THE...OUN...M
We see that a shift of 4 looks good (both of the others have unlikely Qs), so we shift the revealed "ETA" back by 4 into the plaintext:
ciphertext: WMPMMXXAEYHBRYOCA key: ..LTM.ETA.THE.OUN plaintext: ..ETA.THE.OUN.AIN
We have a lot to work with now. The keyword is probably 4 characters long ("..LT"), and we have some of the message:
Because our plaintext guesses have an effect on the key 4 characters to the left, we get feedback on correct/incorrect guesses, so we can quickly fill in the gaps:
The ease of cryptanalysis is thanks to the feedback from the relationship between plaintext and key. A 3-character guess reveals 6 more characters, which then reveal further characters, creating a cascade effect, allowing us to rule out incorrect guesses quickly.
Autokey in modern ciphers
Modern autokey ciphers use very different encryption methods, but they follow the same approach of using either key bytes or plaintext bytes to generate more key bytes. Most modern stream ciphers are based on pseudorandom number generators: the key is used to initialize the generator, and either key bytes or plaintext bytes are fed back into the generator to produce more bytes.
Some stream ciphers are said to be "self-synchronizing", because the next key byte usually depends only on the previous N bytes of the message. If a byte in the message is lost or corrupted, therefore, the key-stream will also be corrupted—but only until N bytes have been processed. At that point the keystream goes back to normal, and the rest of the message will decrypt correctly.
- Bellaso, Giovan Battista, Il vero modo di scrivere in cifra con facilità, prestezza, et securezza di Misser Giovan Battista Bellaso, gentil’huomo bresciano, Iacobo Britannico, Bressa 1564.
- Vigenère, Blaise de, Traicté des chiffres ou secrètes manières d’escrire, Abel l’Angelier, Paris 1586. ff. 46r-49v.
- LABRONICUS (Buonafalce, A), Early Forms of the Porta Table, “The Cryptogram”, vol. LX n. 2, Wilbraham 1994.
- Buonafalce, Augusto, Bellaso’s Reciprocal Ciphers, “Cryptologia” 30 (1):39-51, 2006.
- LABRONICUS (Buonafalce, A), Vigenère and Autokey. An Update, “The Cryptogram”, vol. LXXIV n. 3, Plano 2008.
- Secret Code Breaker - AutoKey Cipher Decoder and Encoder