The son of a lawyer, Brion was classmates in Thiers with Marcel Pagnol and Albert Cohen. After completing his secondary education in Collège Champittet, Switzerland, he studied law at the University of Aix-en-Provence. Counsel to the bar of Marseille between 1920 and 1924, he abandoned his legal career to turn to literature.
Brion wrote nearly a hundred books in his career, ranging from historical biography to examinations of Italian and German art, and turning later in life to novels. His most famous collection of stories is the 1942 Les Escales de la Haute Nuit (The Shore Leaves Of The Deepest Night). An essay of Brion appears in Our Exagmination Round His Factification for Incamination of Work in Progress, the important 1929 critical appreciation of James Joyce's Finnegans Wake.
He was a friend of the philosopher Xavier Tilliette.
In 1964, Brion was elected to the Académie françaisechair 33, replacing his friend Jean-Louis Vaudoyer. Other distinctions include membership in the Légion d'honneur, the Croix de guerre 1914–1918, a Grand Officer in the French Ordre national du Mérite, and an Officer of the Ordre des Arts et des Lettres.
His son, Patrick Brion, critic and film historian, is the "voice" of Cinema midnight on France 3.