The song was inspired by Harrison's meeting with a man named George Greif. At the 1975 Midem Music Festival, Harrison met the man and remarked to him that he resembled the late comedian Lord Buckley (Harrison had admired Buckley for many years). As chance would have it, Greif was Buckley's former manager, and he invited Harrison to come see Buckley's old Los Angeles home, which he called "Crackerbox Palace". Thinking that the phrase had the makings of a song, Harrison jotted the words "Crackerbox Palace" down on a cigarette pack, and later wrote the song. The song includes references to Greif ("I met a Mr. Greif") and to Lord Buckley ("know that the Lord is well and inside of you").
A whimsical promotional film accompanied the single and was first shown on 20 November 1976, episode of Saturday Night Live. Directed by Monty Python's Eric Idle, the film featured Harrison, Neil Innes (as the carriage-pushing nanny/mother, a bathrobe-clad man with a duck on his head, and as a church authority), future wife Olivia Arias, and various other friends, in an array of wild costumes. The film was shot in and around the grounds of Harrison's home, Friar Park.