He was born Watanabe Sadayasu in Edo (now Tokyo) to a poor samurai family, and his artistic talent was developed from an early age. His family served the lord of the Tahara Domain, located in present day Aichi prefecture. Watanabe himself served the lord of Tahara as a senior councilor. He was heavily influenced by the artistic styles of the West, forming a unique style with elements of Japanese and European art.
He was the first Japanese artist to paint realistic portraits of his subjects using the effects of shading which he learned from European paintings. On the one hand, he was a traditionalist Confucian, who believed in filial piety and loyalty to his daimyo, and on the other he was enthusiastic about Western ideas regarding science and politics. He wrote two private essays which were interpreted as being critical of the Shogunate's defense of Tokyo Bay and promoting Western ideas. Although these papers were discarded by Watanabe, they were found and he was tried and exiled to his home province of Tahara. One of the conditions of his exile was that he wouldn't sell his paintings, however Watanabe continued selling his paintings in secret due to financial hardships. This was eventually discovered leading to the suppression of his works and house arrest.