Branch and bound
|Graph and tree
Branch and bound (BB or B&B) is an algorithm design paradigm for discrete and combinatorial optimization problems, as well as general real valued problems. A branch-and-bound algorithm consists of a systematic enumeration of candidate solutions by means of state space search: the set of candidate solutions is thought of as forming a rooted tree with the full set at the root. The algorithm explores branches of this tree, which represent subsets of the solution set. Before enumerating the candidate solutions of a branch, the branch is checked against upper and lower estimated bounds on the optimal solution, and is discarded if it cannot produce a better solution than the best one found so far by the algorithm.
The method was first proposed by A. H. Land and A. G. Doig in 1960 for discrete programming, and has become the most commonly used tool for solving NP-hard optimization problems. The name "branch and bound" first occurred in the work of Little et al. on the traveling salesman problem.
In order to facilitate a concrete description, we assume that the goal is to find the minimum value of a function , where ranges over some set of admissible or candidate solutions (the search space or feasible region). Note that one can find the maximum value of by finding the minimum of . (For example, could be the set of all possible trip schedules for a bus fleet, and could be the expected revenue for schedule .)
A branch-and-bound procedure requires two tools. The first one is a splitting procedure that, given a set of candidates, returns two or more smaller sets whose union covers . Note that the minimum of over is , where each is the minimum of within . This step is called branching, since its recursive application defines a search tree whose nodes are the subsets of .
The second tool is a procedure that computes upper and lower bounds for the minimum value of within a given subset of . This step is called bounding.
The key idea of the BB algorithm is: if the lower bound for some tree node (set of candidates) is greater than the upper bound for some other node , then may be safely discarded from the search. This step is called pruning, and is usually implemented by maintaining a global variable (shared among all nodes of the tree) that records the minimum upper bound seen among all subregions examined so far. Any node whose lower bound is greater than can be discarded.
The recursion stops when the current candidate set is reduced to a single element, or when the upper bound for set matches the lower bound. Either way, any element of will be a minimum of the function within .
The following is the skeleton of a generic branch and bound algorithm for minimizing an arbitrary objective function f. To obtain an actual algorithm from this, one requires a bounding function g, that computes lower bounds of f on nodes of the search tree, as well as a problem-specific branching rule.
- Using a heuristic, find a solution xh to the optimization problem. Store its value, B = f(xh). (If no heuristic is available, set B to infinity.) B will denote the best solution found so far, and will be used as an upper bound on candidate solutions.
- Initialize a queue to hold a partial solution with none of the variables of the problem assigned.
- Loop until the queue is empty:
- Take a node N off the queue.
- If N represents a single candidate solution x and f(x) < B, then x is the best solution so far. Record it and set B ← f(x).
- Else, branch on N to produce new nodes Ni. For each of these:
- If g(Ni) > B, do nothing; since the lower bound on this node is greater than the upper bound of the problem, it will never lead to the optimal solution, and can be discarded.
- Else, store Ni on the queue.
Several different queue data structures can be used. A stack (LIFO queue) will yield a depth-first algorithm. A best-first branch and bound algorithm can be obtained by using a priority queue that sorts nodes on their g-value. The depth-first variant is recommended when no good heuristic is available for producing an initial solution, because it quickly produces full solutions, and therefore upper bounds.
This approach is used for a number of NP-hard problems
- Integer programming
- Nonlinear programming
- Travelling salesman problem (TSP)
- Quadratic assignment problem (QAP)
- Maximum satisfiability problem (MAX-SAT)
- Nearest neighbor search (NNS)
- Cutting stock problem
- False noise analysis (FNA)
- Computational phylogenetics
- Set inversion
- Parameter estimation
- 0/1 knapsack problem
- Feature selection in machine learning
- k-nearest neighbor search
- Structured prediction in computer vision:267–276
Branch-and-bound may also be a base of various heuristics. For example, one may wish to stop branching when the gap between the upper and lower bounds becomes smaller than a certain threshold. This is used when the solution is "good enough for practical purposes" and can greatly reduce the computations required. This type of solution is particularly applicable when the cost function used is noisy or is the result of statistical estimates and so is not known precisely but rather only known to lie within a range of values with a specific probability.
- Alpha-beta pruning
- Branch-and-cut, a hybrid between branch-and-bound and the cutting plane methods that is used extensively for solving integer linear programs.
- A. H. Land and A. G. Doig (1960). "An automatic method of solving discrete programming problems". Econometrica 28 (3). pp. 497–520. doi:10.2307/1910129.
- Clausen, Jens (1999). Branch and Bound Algorithms—Principles and Examples (PDF) (Technical report). University of Copenhagen.
- Little, John D. C.; Murty, Katta G.; Sweeney, Dura W.; Karel, Caroline (1963). "An algorithm for the traveling salesman problem" (PDF). Operations Research 11 (6): 972–989. doi:10.1287/opre.11.6.972.
- Balas, Egon; Toth, Paolo (1983). Branch and bound methods for the traveling salesman problem (PDF) (Report) (Management Science Research Report MSRR-488). Carnegie Mellon University Graduate School of Industrial Administration.
- Moore, R. E. (1966). Interval Analysis. Englewood Cliff, New Jersey: Prentice-Hall. ISBN 0-13-476853-1.
- Jaulin, L.; Kieffer, M.; Didrit, O.; Walter, E. (2001). Applied Interval Analysis. Berlin: Springer. ISBN 1-85233-219-0.
- Hansen, E.R. (1992). Global Optimization using Interval Analysis. New York: Marcel Dekker.
- Mehlhorn, Kurt; Sanders, Peter (2008). Algorithms and data structures: The basic toolbox. Springer. p. 249.
- Conway, Richard Walter; Maxwell, William L.; Miller, Louis W. (2003). Theory of Scheduling. Courier Dover Publications. pp. 56–61.
- Narendra, Patrenahalli M.; Fukunaga, K. (1977). "A branch and bound algorithm for feature subset selection". IEEE Transactions on Computers C–26 (9): 917–922. doi:10.1109/TC.1977.1674939.
- Fukunaga, Keinosuke; Narendra, Patrenahalli M. (1975). "A branch and bound algorithm for computing k-nearest neighbors". IEEE Transactions on Computers: 750–753.
- Nowozin, Sebastian; Lampert, Christoph H. (2011). "Structured Learning and Prediction in Computer Vision". Foundations and Trends in Computer Graphics and Vision 6 (3–4): 185–365. doi:10.1561/0600000033. ISBN 978-1-60198-457-9.