A precise inverse of the inequality cannot hold: for every , there are distributions with but . An easy example is given by the two-point space with and . 
However, an inverse inequality holds on finite spaces with a constant depending on . More specifically, it can be shown that with the definition we have for any measure which is absolutely continuous to
As a consequence, if has full support (i.e. for all ), then
^Tsybakov, Alexandre (2009). Introduction to Nonparametric Estimation. Springer. p. 132. ISBN9780387790527.
^The divergence becomes infinite whenever one of the two distributions assigns probability zero to an event while the other assigns it a nonzero probability (no matter how small); see e.g. Basu, Mitra; Ho, Tin Kam (2006). Data Complexity in Pattern Recognition. Springer. p. 161. ISBN9781846281723..
^see Lemma 4.1 in Götze, Friedrich; Sambale, Holger; Sinulis, Arthur. "Higher order concentration for functions of weakly dependent random variables". arXiv:1801.06348.