In 1951 he was ordained a priest for the Society of the Divine Word in Bay St. Louis, Mississippi. From 1952 through 1973 he served as Assistant Dean and Dean of Seminarians and rector of the Religious Community. From 1974 through 1982 he was Secretary of Studies for all USA Divine Word Seminaries and Rector of the Religious Community of Divine Word Seminary (in Epworth, Iowa). St. Anthony's parish in Lafayette, Louisiana was his first pastorate. In the last two years of that assignment, he also served as part-time Vicar for Black Catholics in the Diocese of Lafayette in Louisiana.
He became full-time Vicar in 1986. Two years later, on November 7, 1988, he was appointed Auxiliary Bishop of the Roman Catholic Archdiocese of Washington by Pope John Paul II. Consecrated bishop on December 20, 1988, he served as Regional Bishop of Northeast, Southeast and East Northwest Deaneries of the District of Columbia, and Deaneries of North and Middle Prince George's Counties and, since 1995, the Deaneries of Northeast and Southeast of the District of Columbia, of Lower Prince George County, and Counties of Charles, Calvert, and St. Mary's.
Olivier was a Fourth Degree Knight of St. Peter Claver, a Fourth Degree Knight of St. John and Columbus. He served as the Convener of the African American Catholic Bishops Subcommittee on Youth, the Ad Hoc Steering Committee for the National Strategy on Vocations, and the Task Force Group for American Adaptations to the Order of Christian Marriage, and the Liturgy Committee of USCC/NCCB. He was a member of NCCB Committee on Bishops Life and Ministry, board member of Covenant House Washington, member of the Inter-Faith Conference, member of Maryland Catholic Conference, board member of the National Black Catholic Congress, member of National Black Catholic Clergy Caucus, and Episcopal Moderator of the Pan African Roman Catholic Clergy Conference. On November 19, 2014, Bishop Olivier died.