Musically, the song marks a shift from the hard rock of 1969's Everybody Knows This Is Nowhere and bears more folk and country influences, which would continue onto 1972's Harvest. The only instruments are two acoustic guitars, played by Young and Nils Lofgren. He is, however, backed by the vocal harmonies of Crazy Horse during the choruses. 'Tell Me Why' has a simple lyrical structure, with 2 verses each followed by a bridge then chorus, and one final bridge and chorus before a short instrumental outro. The chorus line "Tell me why, tell me why/Is it hard to make arrangements with yourself/When you're old enough to repay/but young enough to sell?" is the most famous from the song, typifying the introspective and melancholic nature of not just this song, but the whole album.
Acoustic guitarists who want to play the song should know the song is in standard tuning, but each string is tuned down full step (E down to D, A down to G, D down to C, G down to F, B down to A, and high E down to D).