Morse was born on May 18, 1931 in Newton, Massachusetts, at St. Mary's General Hospital. He was the second of Joseph Xavier and Edna Morse's eight children and mostly raised in Boston, Massachusetts, his family having moved there when he was one. He was ten when his father died in a car accident, and his mother, who suffered a nervous breakdown afterwards, was committed to a mental institution in Florida. Therefore, young Robert and his siblings were forced to live with their maternal grandparents, John and Nancy Porter, in Cambridge. Around this time, unable to escape the stresses in his life, Morse would regularly make trips to the movie theater, seeing the new movies of the day and quickly developing idols like Humphrey Bogart, James Cagney, Bette Davis, Leslie Howard, Orson Welles, Spencer Tracy, and Katharine Hepburn, while Alfred Hitchcock, John Huston, and Billy Wilder were his favorite directors. Beginning at age thirteen, he began appearing in school productions such as Romeo and Juliet, When Christopher Columbus Discovered America, and The Dark Alleys of New York. It was also around age thirteen that he declared his plans to become an actor and hoped that nothing would stand in his way. He graduated from high school in 1949 and spent the next year working as a ditch digger, newspaper deliveryman, waiter, drugstore clerk, and bodybuilder before joining the United States Navy and being sent off to fight in Korea in 1950. After serving in the U.S. Navy during the Korean War, Morse appeared on Broadway as an actor.