Kalyanji was born to Virji Shah, a Kutchi businessman in Kundrodi, Kutch, Gujarat, who migrated from Kutch to Mumbai to start a Kirana (provision store).
He and his brothers began to learn music from a music teacher, who actually knew no music but taught them in lieu of paying his bills to their father. One of their four grand parents was a folk musician of some eminence. They spent most of their formative years in the hamlet of Girgaum (a district in Mumbai) amidst Marathi and Gujarati environs — some eminent musical talent resided in the vicinity.
Kalyanji's breakthrough was with the theme entitled Been music from the film Nagin (1954).
The brothers were known for their humility and affable nature. They also were known for their love songs and poetry readings. Kalyanji's son Viju Shah is also a famous music director.