Fault-tolerant computer system
Fault-tolerant computer systems are systems designed around the concepts of fault tolerance. In essence, they have to be able to keep working to a level of satisfaction in the presence of faults.
Types of fault tolerance
Most fault-tolerant computer systems are designed to be able to handle several possible failures, including hardware-related faults such as hard disk failures, input or output device failures, or other temporary or permanent failures; software bugs and errors; interface errors between the hardware and software, including driver failures; operator errors, such as erroneous keystrokes, bad command sequences, or installing unexpected software; and physical damage or other flaws introduced to the system from an outside source.
Hardware fault-tolerance is the most common application of these systems, designed to prevent failures due to hardware components. Most basically this is provided by redundancy, particularly dual modular redundancy. Typically, components have multiple backups and are separated into smaller "segments" that act to contain a fault, and extra redundancy is built into all physical connectors, power supplies, fans, etc. There are special software and instrumentation packages designed to detect failures, such as fault masking, which is a way to ignore faults by seamlessly preparing a backup component to execute something as soon as the instruction is sent, using a sort of voting protocol where if the main and backups don't give the same results, the flawed output is ignored.
Software fault-tolerance is based more around nullifying programming errors using real-time redundancy, or static "emergency" subprograms to fill in for programs that crash. There are many ways to conduct such fault-regulation, depending on the application and the available hardware.
The first known fault-tolerant computer was SAPO, built in 1951 in Czechoslovakia by Antonin Svoboda. Its basic design was magnetic drums connected via relays, with a voting method of memory error detection (triple modular redundancy). Several other machines were developed along this line, mostly for military use. Eventually, they separated into three distinct categories: machines that would last a long time without any maintenance, such as the ones used on NASA space probes and satellites; computers that were very dependable but required constant monitoring, such as those used to monitor and control nuclear power plants or supercollider experiments; and finally, computers with a high amount of runtime which would be under heavy use, such as many of the supercomputers used by insurance companies for their probability monitoring.
Most of the development in the so-called LLNM (Long Life, No Maintenance) computing was done by NASA during the 1960s, in preparation for Project Apollo and other research aspects. NASA's first machine went into a space observatory, and their second attempt, the JSTAR computer, was used in Voyager. This computer had a backup of memory arrays to use memory recovery methods and thus it was called the JPL Self-Testing-And-Repairing computer. It could detect its own errors and fix them or bring up redundant modules as needed. The computer is still working today.
Hyper-dependable computers were pioneered mostly by aircraft manufacturers, nuclear power companies, and the railroad industry in the USA. These needed computers with massive amounts of uptime that would fail gracefully enough with a fault to allow continued operation, while relying on the fact that the computer output would be constantly monitored by humans to detect faults. Again, IBM developed the first computer of this kind for NASA for guidance of Saturn V rockets, but later on BNSF, Unisys, and General Electric built their own.
In general, the early efforts at fault-tolerant designs were focused mainly on internal diagnosis, where a fault would indicate something was failing and a worker could replace it. SAPO, for instance, had a method by which faulty memory drums would emit a noise before failure. Later efforts showed that, to be fully effective, the system had to be self-repairing and diagnosing – isolating a fault and then implementing a redundant backup while alerting a need for repair. This is known as N-model redundancy, where faults cause automatic fail safes and a warning to the operator, and it is still the most common form of level one fault-tolerant design in use today.
Voting was another initial method, as discussed above, with multiple redundant backups operating constantly and checking each other's results, with the outcome that if, for example, four components reported an answer of 5 and one component reported an answer of 6, the other four would "vote" that the fifth component was faulty and have it taken out of service. This is called M out of N majority voting.
Historically, motion has always been to move further from N-model and more to M out of N due to the fact that the complexity of systems and the difficulty of ensuring the transitive state from fault-negative to fault-positive did not disrupt operations.
Fault tolerance verification and validation
The most important requirement of design in a fault tolerant computer system is making sure it actually meets its requirements for reliability. This is done by using various failure models to simulate various failures, and analyzing how well the system reacts. These statistical models are very complex, involving probability curves and specific fault rates, latency curves, error rates, and the like. The most commonly used models are HARP, SAVE, and SHARPE in the USA, and SURF or LASS in Europe.
Fault tolerance research
Research into the kinds of tolerances needed for critical systems involves a large amount of interdisciplinary work. The more complex the system, the more carefully all possible interactions have to be considered and prepared for. Considering the importance of high-value systems in transport, public utilities and the military, the field of topics that touch on research is very wide: it can include such obvious subjects as software modeling and reliability, or hardware design, to arcane elements such as stochastic models, graph theory, formal or exclusionary logic, parallel processing, remote data transmission, and more.
Failure-oblivious computing is a technique that enables computer programs to continue executing despite memory errors. The technique handles attempts to read invalid memory by returning a manufactured value to the program, which in turn, makes use of the manufactured value and ignores the former memory value it tried to access. This is a great contrast to typical memory checkers, which inform the program of the error or abort the program. In failure-oblivious computing, no attempt is made to inform the program that an error occurred.
The approach has performance costs: because the technique rewrites code to insert dynamic checks for address validity, execution time will increase by 80% to 500%.
- Fault-tolerant system
- Resilience (network)
- List of system quality attributes
- immunity aware programming
- Fault-tolerant computer system design book contents. Dhiraj K. Pradhan, Pages: 135 – 138 1996 ISBN 0-13-057887-8
- Formal Techniques in Real-Time and Fault-Tolerant Systems: Second International Symposium, Nijmegen, the Netherlands, January 8–10, 1992, Proceedings By Jan Vytopil Contributor Jan Vytopil, Published by Springer, 1991, ISBN 3-540-55092-5, 978-3-540-55092-1
- Fault-tolerant computer system design book contents. Dhiraj K. Pradhan, Pages: 221 – 235 1996 ISBN 0-13-057887-8
- Computer structures: principles and examples, pg 155 By Daniel P. Siewiorek, C. Gordon Bell, Allen Newell Published by McGraw-Hill, 1982 ISBN 0-07-057302-6, 978-0-07-057302-4
- Computer structures: principles and examples, pg 189 By Daniel P. Siewiorek, C. Gordon Bell, Allen Newell Published by McGraw-Hill, 1982 ISBN 0-07-057302-6, 978-0-07-057302-4
- Computer structures: principles and examples, pg 210 By Daniel P. Siewiorek, C. Gordon Bell, Allen Newell Published by McGraw-Hill, 1982 ISBN 0-07-057302-6, 978-0-07-057302-4
- Computer structures: principles and examples, pg 223 By Daniel P. Siewiorek, C. Gordon Bell, Allen Newell Published by McGraw-Hill, 1982 ISBN 0-07-057302-6, 978-0-07-057302-4
- Fault tolerant computing in computer design Neilforoshan, M.R Journal of Computing Sciences in Colleges archive Volume 18 , Issue 4 (April 2003) Pages: 213 – 220 ISSN:1937-4771
- Reliability Evaluation of Some Fault-Tolerant Computer Architectures By Shunji Osaki, Toshihiko Nishio Published by Springer, 1980 ISBN 3-540-10274-4, 978-3-540-10274-8
- Rinard, Martin; Cadar, Cristian; Dumitran, Daniel; Roy, Daniel M.; Leu, Tudor; Beebee, William S. (2004), "Enhancing server availability and security through failure-oblivious computing", Proceedings of the 6th conference on Symposium on Operating Systems Design & Implementation 6, Berkeley, CA: USENIX Association, CiteSeerX: 10.1.1.68.9926
- Keromytis, Angelos D. (2007), "Characterizing Software Self-Healing Systems", in Gorodetski, Vladimir I.; Kotenko, Igor; Skormin, Victor A., Computer network security: Fourth International Conference on Mathematical Methods, Models, and Architectures for Computer Network Security, Springer, ISBN 3-540-73985-8
- Primer on Fault-Tolerant Computer Systems from UCLA
- Bruckert, William F., Bissett, Thomas D., Mazur, Dennis, Munzer, John, Bernaby, Frank, and Bhatia, Jay H. Fault tolerant computer systems with fault isolation and repair. US Patent 5,099,485, filed September 4, 1987 and issued Mar 24, 1992 A fault-tolerant patent with a lot of basic information on specific ways to detect faults
- Enhancing Server Availability and Security Through Failure-Oblivious Computing. CiteSeerX: 10.1.1.68.9926.