In physics, a physical system is the portion of the physical universe chosen for analysis. Everything outside the system is known as the environment, within which analysis is ignored excluding its effects on the system.
Complexity in physical systems
If one considers a classical Newtonian ball situation with a number of perfectly moving physical bodies bouncing off the walls of a container, the system-state probability does not change over time. The entropy of the system changes over time, but the probability of the state vector does not change. One can periodically evaluate the complexity of this system, and the complexity of this system does not change.
In a physical system, a lower probability state vector is equivalent to a higher complexity. A self-sustaining low probability state vector allows the physical system to remain in a higher complexity state. The study of such systems as applied to our universe is in its infancy and speculative in nature, but it appears that there are some low probability systems that are able to sustain themselves through time.
In mathematical systems, one can consider the complexity of particular states more easily. For example, if one considers a Turing machine that generates random symbols and then uses them to create a new sequences of symbols, the complexity of the final string of symbols is nearly mathematically equivalent to the minimum size of a string required to produce a larger string on a Turing machine as defined by algorithmic information theory.
The cut between system and the world is a free choice, generally made to simplify the analysis as much as possible. An isolated system is one which is supposed to have negligible interaction with its environment.
Often a system in this sense is chosen to correspond to the more usual meaning of system, such as a particular machine. But physical systems are often more esoteric: an atom, the water in a lake, or indeed the water in the left-hand half of a lake can all be considered as physical systems. In the study of quantum decoherence the "system" may refer to the macroscopic properties of an object (e.g. the position of a pendulum bob), while the relevant "environment" may be the internal degrees of freedom, described classically by the pendulum's thermal vibrations.