Separable pure states
For simplicity, the following assumes all relevant state spaces are finite-dimensional. First, consider separability for pure states.
Let and be quantum mechanical state spaces, that is, finite-dimensional Hilbert spaces with basis states and , respectively. By a postulate of quantum mechanics, the state space of the composite system is given by the tensor product
with base states , or in more compact notation . From the very definition of the tensor product, any vector of norm 1, i.e. a pure state of the composite system, can be written as
where is a constant. If a pure state can be written in the form where is a pure state of the i-th subsystem, it is said to be separable. Otherwise it is called entangled. When a system is in an entangled pure state, it is not possible to assign states to its subsystems. This will be true, in the appropriate sense, for the mixed state case as well.
Formally, the embedding of a product of states into the product space is given by the Segre embedding. That is, a quantum-mechanical pure state is separable if and only if it is in the image of the Segre embedding.
The above discussion can be extended to the case of when the state space is infinite-dimensional with virtually nothing changed.
Separability for mixed states
Consider the mixed state case. A mixed state of the composite system is described by a density matrix acting on . ρ is separable if there exist , and which are mixed states of the respective subsystems such that
Otherwise is called an entangled state. We can assume without loss of generality in the above expression that and are all rank-1 projections, that is, they represent pure ensembles of the appropriate subsystems. It is clear from the definition that the family of separable states is a convex set.
Notice that, again from the definition of the tensor product, any density matrix, indeed any matrix acting on the composite state space, can be trivially written in the desired form, if we drop the requirement that and are themselves states and If these requirements are satisfied, then we can interpret the total state as a probability distribution over uncorrelated product states.
When the state spaces are infinite-dimensional, density matrices are replaced by positive trace class operators with trace 1, and a state is separable if it can be approximated, in trace norm, by states of the above form.
If there is only a single non-zero , then the state is called simply separable (or it is called a "product state").
Extending to the multipartite case
The above discussion generalizes easily to the case of a quantum system consisting of more than two subsystems. Let a system have n subsystems and have state space . A pure state is separable if it takes the form
Similarly, a mixed state ρ acting on H is separable if it is a convex sum
Or, in the infinite-dimensional case, ρ is separable if it can be approximated in the trace norm by states of the above form.
The problem of deciding whether a state is separable in general is sometimes called the separability problem in quantum information theory. It is considered to be a difficult problem. It has been shown to be NP-hard. Some appreciation for this difficulty can be obtained if one attempts to solve the problem by employing the direct brute force approach, for a fixed dimension. We see that the problem quickly becomes intractable, even for low dimensions. Thus more sophisticated formulations are required. The separability problem is a subject of current research.
A separability criterion is a necessary condition a state must satisfy to be separable. In the low-dimensional (2 X 2 and 2 X 3) cases, the Peres-Horodecki criterion is actually a necessary and sufficient condition for separability. Other separability criteria include the range criterion and reduction criterion. See Ref. for a review of separability criteria in discrete variable systems.
In continuous variable systems, the Peres-Horodecki criterion also applies. Specifically, Simon  formulated a particular version of the Peres-Horodecki criterion in terms of the second-order moments of canonical operators and showed that it is necessary and sufficient for -mode Gaussian states (see Ref. for a seemingly different but essentially equivalent approach). It was later found  that Simon's condition is also necessary and sufficient for -mode Gaussian states, but no longer sufficient for -mode Gaussian states. Simon's condition can be generalized by taking into account the higher order moments of canonical operators  or by using entropic measures.
Characterization via algebraic geometry
Quantum mechanics may be modelled on a projective Hilbert space, and the categorical product of two such spaces is the Segre embedding. In the bipartite case, a quantum state is separable if and only if it lies in the image of the Segre embedding. Jon Magne Leinaas, Jan Myrheim and Eirik Ovrum in their paper "Geometrical aspects of entanglement" describe the problem and study the geometry of the separable states as a subset of the general state matrices. This subset have some intersection with the subset of states holding Peres-Horodecki criterion. In this paper, Leinaas et al. also give a numerical approach to test for separability in the general case.
Testing for separability
Testing for separability in the general case is an NP-hard problem. Leinaas et. al. formulated an iterative, probabilistic algorithm for testing if a given state is separable. When the algorithm is successful, it gives an explicit, random, representation of the given state as a separable state. Otherwise it gives the distance of the given state from the nearest separabel state it can find. A web-app implementation of the algorithm (including a built in Peres-Horodecki criterion testing) is "StateSeparator"
- Gurvits, L., Classical deterministic complexity of Edmonds’ problem and quantum entanglement, in Proceedings of the 35th ACM Symposium on Theory of Computing, ACM Press, New York, 2003.
- Sevag Gharibian, Strong NP-Hardness of the Quantum Separability Problem, Quantum Information and Computation, Vol. 10, No. 3&4, pp. 343-360, 2010. arXiv:0810.4507.
- Gühne, Otfried; Tóth, Géza. "Entanglement detection". Physics Reports. 474 (1–6): 1–75. arXiv:0811.2803. Bibcode:2009PhR...474....1G. doi:10.1016/j.physrep.2009.02.004.
- Simon, R. "Peres-Horodecki Separability Criterion for Continuous Variable Systems". Physical Review Letters. 84 (12): 2726–2729. arXiv:quant-ph/9909044. Bibcode:2000PhRvL..84.2726S. doi:10.1103/PhysRevLett.84.2726. PMID 11017310.
- Duan, Lu-Ming; Giedke, G.; Cirac, J. I.; Zoller, P. "Inseparability Criterion for Continuous Variable Systems". Physical Review Letters. 84 (12): 2722–2725. arXiv:quant-ph/9908056. Bibcode:2000PhRvL..84.2722D. doi:10.1103/PhysRevLett.84.2722.
- Werner, R. F.; Wolf, M. M. "Bound Entangled Gaussian States". Physical Review Letters. 86 (16): 3658–3661. arXiv:quant-ph/0009118. Bibcode:2001PhRvL..86.3658W. doi:10.1103/PhysRevLett.86.3658.
- Shchukin, E.; Vogel, W. "Inseparability Criteria for Continuous Bipartite Quantum States". Physical Review Letters. 95 (23). arXiv:quant-ph/0508132. Bibcode:2005PhRvL..95w0502S. doi:10.1103/PhysRevLett.95.230502. Hillery, Mark; Zubairy, M.Suhail. "Entanglement Conditions for Two-Mode States". Physical Review Letters. 96 (5): 050503. arXiv:quant-ph/0507168. Bibcode:2006PhRvL..96e0503H. doi:10.1103/PhysRevLett.96.050503. PMID 16486912.
- Walborn, S.; Taketani, B.; Salles, A.; Toscano, F.; de Matos Filho, R. "Entropic Entanglement Criteria for Continuous Variables". Physical Review Letters. 103 (16): 160505. arXiv:0909.0147. Bibcode:2009PhRvL.103p0505W. doi:10.1103/PhysRevLett.103.160505. PMID 19905682. Huang, Yichen. "Entanglement Detection: Complexity and Shannon Entropic Criteria". IEEE Transactions on Information Theory. 59 (10): 6774–6778. doi:10.1109/TIT.2013.2257936.
- "Geometrical aspects of entanglement", Physical Review A 74, 012313 (2006)