Therac-25

The Therac-25 is a computer-controlled radiation therapy machine produced by Atomic Energy of Canada Limited (AECL) in 1982 after the Therac-6 and Therac-20 units (the earlier units had been produced in partnership with Compagnie générale de radiologie (CGR) of France).

The Therac-25 was involved in at least six accidents between 1985 and 1987, in which patients were given massive overdoses of radiation. Because of concurrent programming errors (also known as race conditions), it sometimes gave its patients radiation doses that were hundreds of times greater than normal, resulting in death or serious injury. These accidents highlighted the dangers of software control of safety-critical systems, and they have become a standard case study in health informatics, software engineering, and computer ethics. Additionally, the overconfidence of the engineers and lack of proper due diligence to resolve reported software bugs are highlighted as an extreme case where the engineers' overconfidence in their initial work and failure to believe the end users' claims caused drastic repercussions.

History

Linear accelerator.

Animation of the operation of a medical use linear accelerator.

The French company CGR manufactured the Neptune and Sagittaire linear accelerators.

In the early 1970s, CGR and the Canadian public company Atomic Energy of Canada Limited (AECL) collaborated on the construction of linear accelerators controlled by a DEC PDP-11 minicomputer: the Therac-6, which produced X-rays of up to 6 MeV, and the Therac-20, which could produce X-rays or electrons of up to 20 MeV. The computer added some ease of use because the accelerator could operate without it. CGR developed the software for the Therac-6 and reused some subroutines for the Therac-20.

In 1981, the two companies ended their collaboration agreement. AECL developed a new double pass concept for electron acceleration in a more confined space, changing its energy source from klystron to magnetron. In certain techniques, the electrons produced are used directly, while in others they are made to collide against a tungsten anode to produce X-ray beams. This dual accelerator concept was applied to the Therac-20 and Therac-25, with the latter being much more compact, versatile, and easy to use. It was also more economical for a hospital to have a dual machine that could apply treatments of electrons and X-rays, instead of two machines.

The Therac-25 was designed as a machine controlled by a computer, and some safety mechanisms switched from hardware to software. AECL decided not to duplicate some safety mechanisms. AECL reused modules and code routines from the Therac-20 for the Therac-25.

The first prototype of the Therac-25 was built in 1976. It began to be marketed at the end of 1982.

The software for the Therac-25 was developed by one person over several years using PDP-11 assembly language. It was an evolution of the Therac-6 software. In 1986, the programmer left AECL. In a lawsuit, lawyers could not identify the programmer or learn about his qualification and experience.

Five machines were installed in the United States and six in Canada.

After the accidents, in 1988 AECL dissolved the AECL Medical section and the company Theratronics International Ltd took over the maintenance of the installed Therac-25 machines.

Design

The machine had three modes of operation, with a turntable moving some apparatus into position for each of those modes: either a light, some scan magnets, or a tungsten target and flattener.

• A "field light" mode, which allowed the patient and collimator to be correctly positioned by illuminating the treatment area with visible light.
• Direct electron-beam therapy, in which a narrow, low-current beam of high-energy (5 to 25 MeV (0.80 to 4.01 pJ)) electrons was scanned over the treatment area by magnets;
• Megavolt X-ray (or photon) therapy, which delivered a beam of 25 MeV X-ray photons. The X-ray photons are produced by colliding a high current, narrow beam of electrons with a tungsten target. The X-rays are then passed through a flattening filter, and then measured using an X-ray ion chamber. The flattening filter resembles an inverted ice-cream cone, and it shapes and attenuates the X-rays. The electron beam current required to produce the X-rays is about 100 times greater than that used for electron therapy.

Turntable rotation.

The patient is placed on a fixed stretcher. Above them is a turntable to which the components that modify the electron beam are fixed. The turntable has a position for the X-ray mode (photons), another position for the electron mode and a third position for making adjustments using visible light. In this position an electron beam is not expected, and a light that is reflected in a stainless steel mirror simulates the beam. In this position there is no ion chamber acting as a radiation dosimeter because the radiation beam is not expected to function.

The turntable has some microswitches that indicate the position to the computer. When the plate is in one of the three allowed fixed positions a plunger locks it by interlocking. In this type of machine, electromechanical locks were traditionally used to ensure that the turntable was in the correct position before starting treatment. In the Therac-25, these were replaced by software checks.

Problem description

Simulated Therac-25 user interface

The six documented accidents occurred when the high-current electron beam generated in X-ray mode was delivered directly to patients. Two software faults were to blame. One was when the operator incorrectly selected X-ray mode before quickly changing to electron mode, which allowed the electron beam to be set for X-ray mode without the X-ray target being in place. A second fault allowed the electron beam to activate during field-light mode, during which no beam scanner was active or target was in place.

Previous models had hardware interlocks to prevent such faults, but the Therac-25 had removed them, depending instead on software checks for safety.

The high-current electron beam struck the patients with approximately 100 times the intended dose of radiation, and over a narrower area, delivering a potentially lethal dose of beta radiation. The feeling was described by patient Ray Cox as "an intense electric shock", causing him to scream and run out of the treatment room. Several days later, radiation burns appeared, and the patients showed the symptoms of radiation poisoning; in three cases, the injured patients later died as a result of the overdose.

Radiation overexposure incidents

Kennestone Regional Oncology Center, 1985

A Therac-25 had been in operation for six months in Marietta, Georgia at the Kennestone Regional Oncology Center when, on June 3, 1985, applied radiation therapy treatment following a lumpectomy was being performed on 61-year-old woman Katie Yarbrough. She was set to receive a 10-MeV dose of electron therapy to her clavicle. When therapy began, she stated she experienced a "tremendous force of heat...this red-hot sensation." The technician entered the room, to whom Katie stated, "you burned me." The technician assured her this was not possible. She returned home where, in the following days, she experienced reddening of the treatment area. Shortly after, her shoulder became locked in place and she experienced spasms. Within two weeks, the aforementioned redness spread from her chest to her back, indicating that the source of the burn had passed through her, which is the case with radiation burns. The staff at the treatment center did not believe it was possible for the Therac-25 to cause such an injury, and it was treated as a symptom of her cancer. Later, the hospital physicist consulted the AECL about the incident. He calculated that the applied dose was between 15,000 and 20,000 rad (radiation absorbed dose) when she should have been dosed with 200 rad. A dose of 1000 rad can be fatal. In October 1985, Katie sued the hospital and the manufacturer of the machine. In November 1985, the AECL was notified of the lawsuit. It was not until March 1986, after another incident involving the Therac-25, that the AECL informed the FDA that it had received a complaint from the patient.

Due to the radiation overdose, her breast had to be surgically removed, an arm and shoulder were immobilized, and she was in constant pain. The treatment printout function was not activated at the time of treatment and there was no record of the applied radiation data. An out-of-court settlement was reached to resolve the lawsuit.

Ontario Cancer Foundation, 1985

Terminal DEC VT-100.

The Therac-25 had been in operation in the clinic for six months when, on July 26, 1985 a 40-year-old patient was receiving her 24th treatment for cervical cancer. The operator activated the treatment, but after five seconds the machine stopped with the error message "H-tilt", the treatment pause indication and the dosimeter indicating that no radiation had been applied. The operator pressed the P key (Proceed : continue). The machine stopped again. The operator repeated the process five times until the machine stopped the treatment. A technician was called and found no problem. The machine treated six other patients on the same day.

The patient complained of burning and swelling in the area and was hospitalized on July 30. She was suspected of a radiation overdose and the machine was taken out of service. On November 3, 1985 the patient herself died of cancer, although the autopsy mentioned that if she had not died then, she would have had to have a hip replacement due to damage from the radiation overdose. A technician estimated that she received between 13,000 and 17,000 rad.

The incident was reported to the FDA and the Canadian Radiation Protection Bureau.

The AECL suspected that there might be an error with three microswitches that reported the position of the turntable. The AECL was unable to replicate a failure of the microswitches and microswitch testing was inconclusive. They then changed the method to be tolerant of one failure and modified the software to check if the turntable was moving or in the treatment position.

Afterward, the AECL claimed that the modifications represented a five-order-of-magnitude increase in safety.

Yakima Valley Memorial Hospital, 1985

In December 1985 a woman developed an erythema with a parallel band pattern after receiving treatment from a Therac-25 unit. Hospital staff sent a letter on January 31, 1986 to the AECL about the incident. The AECL responded in two pages detailing the reasons why radiation overdose was impossible on the Therac-25, stating both machine failure and operator error were not possible.

Six months later, the patient developed chronic ulcers under the skin due to tissue necrosis. She had surgery and skin grafts were placed. The patient continued to live with minor sequelae.

East Texas Cancer Center, Tyler, March 1986

Over two years, this hospital treated more than 500 patients with the Therac-25 with no incident. On March 21, 1986, a patient presented for his ninth treatment session for a tumor on his back. The treatment was set to be 22-MeV of electrons with a dose of 180 rad in an area of 10x17 cm, with an accumulated radiation in 6 weeks of 6000 rad.

The experienced operator entered the session data and realized that she had written an X instead of an E as the type of treatment. With the cursor she went up and changed the X to an E and since the rest of the parameters were correct she pressed ↵ Enter until she got down to the command box. All parameters were marked "Verified" and the message "Rays ready" was displayed. She hit the B key ("Beam on"). The machine stopped and displayed the message "Malfunction 54" (error 54). It also showed 'Treatment pause'. The manual said that the "Malfunction 54" message was a "dose input 2" error. A technician later testified that "dose input 2" meant that the radiation delivered was either too high or too low.

The radiation monitor (dosimeter) marked 6 units supplied when it had demanded 202 units. The operator pressed P ( Proceed : continue). The machine stopped again with the message "Malfunction 54" (error 54) and the dosimeter indicated that it had delivered fewer units than required. The surveillance camera in the radiation room was offline and the intercom had been broken that day.

With the first dose the patient felt an electric shock and heard a crackle from the machine. Since it was his ninth session, he realized that it was not normal. He started to get up from the table to ask for help. At that moment the operator pressed P to continue the treatment. The patient felt a shock of electricity through his arm, as if his hand was torn off. He reached the door and began to bang on it until the operator opened it. A physician was immediately called to the scene, where they observed intense erythema in the area, suspecting that it had been a simple electric shock. He sent the patient home. The hospital physicist checked the machine and, because it was calibrated to the correct specification, it continued to treat patients throughout the day. The technicians were unaware that the patient had received a massive dose of radiation between 16,500 to 25,000 rads in less than a second over an area of one cm². The crackling of the machine had been produced by saturation of the ionization chambers, which had the consequence that they indicated that the applied radiation dose had been very low.

Over the following weeks the patient experienced paralysis of the left arm, nausea, vomiting, and ended up being hospitalized for radiation-induced myelitis of the spinal cord. His legs, mid-diaphragm and vocal cords ended up paralyzed. He also had recurrent herpes simplex skin infections. He died five months after the overdose.

From the day after the accident, AECL technicians checked the machine and were unable to replicate error 54. They checked the grounding of the machine to rule out electric shock as the cause. The machine was back in operation on April 7, 1986.

East Texas Cancer Center, Tyler, April 1986

On April 11, 1986, a patient was to receive electron treatment for skin cancer on the face. The prescription was 10 MeV for an area of 7x10 cm. The operator was the same as the one in the March incident, three weeks earlier. After filling in all the treatment data he realized that he had to change the mode from X to E. He did so and pressed ↵ Enter to go down to the command box. As "Beam ready" was displayed, he pressed P (Proceed : continue). The machine produced a loud noise, which was heard through the intercom. Error 54 was displayed. The operator entered the room and the patient described a burning sensation on his face. The patient died on May 1, 1986. The autopsy showed severe radiation damage to the right temporal lobe and brain stem.

The hospital physicist stopped the machine treatments and notified the AECL. After strenuous work, the physicist and operator were able to reproduce the error 54 message. They determined that speed in editing the data entry was a key factor in producing error 54. After much practice, he was able to reproduce the error 54 at will. The AECL stated they could not reproduce the error and they only got it after following the instructions of the physicist so that the data entry was very rapid.

Yakima Valley Memorial Hospital, 1987

On January 17, 1987 a patient was to receive a treatment with two film-verification exposures of 4 and 3 rads, plus a 79-rad photon treatment for a total exposure of 86 rads. Film was placed under the patient and 4 rads were administered through a 22 cm × 18 cm opening. The machine was stopped, the aperture was opened to 35 cm × 35 cm and a dose of 3 rad was administered. The machine stopped. The operator entered the room to remove the film plates and adjust the patient's position. He used the hand control inside the room to adjust the turntable. He left the room, forgetting the film plates. In the control room, after seeing the "Beam ready" message, he pressed the B key to fire the beams. After 5 seconds the machine stopped and displayed a message that quickly disappeared. Since the machine was paused, the operator pressed P (Proceed : continue). The machine stopped, showing "Flatness" as the reason. The operator heard the patient on the intercom, but could not understand him, and entered the room. The patient had felt a severe burning sensation in his chest. The screen showed that he had only been given 7 rad. A few hours later, the patient showed burns on the skin in the area. Four days later the reddening of the area had a banded pattern similar to that produced in the incident the previous year, and for which they had not found the cause. The AECL began an investigation, but was unable to reproduce the event.

The hospital physicist conducted tests with film plates to see if he could recreate the incident. Two X-ray parameters with the turntable in field-light position. The film appeared to match the film that was left by mistake under the patient during the accident. It was found the patient was exposed to between 8,000 and 10,000 rad instead of the prescribed 86 rad. The patient died in April 1987 from complications due to radiation overdose. The relatives filed a lawsuit that ended with an out-of-court settlement.

Root causes

A commission attributed the primary cause to general poor software design and development practices rather than single-out specific coding errors. In particular, the software was designed so that it was realistically impossible to test it in a rigorous, automated way.

Researchers who investigated the accidents found several contributing causes. These included the following institutional causes:

• AECL did not have the software code independently reviewed and chose to rely on in-house code, including the operating system.
• AECL did not consider the design of the software during its assessment of how the machine might produce the desired results and what failure modes existed, focusing purely on hardware and asserting that the software was free of bugs.
• Machine operators were reassured by AECL personnel that overdoses were impossible, leading them to dismiss the Therac-25 as the potential cause of many incidents.
• AECL had never tested the Therac-25 with the combination of software and hardware until it was assembled at the hospital.

The researchers also found several engineering issues:

• Several error messages merely displayed the word "MALFUNCTION" followed by a number from 1 to 64. The user manual did not explain or even address the error codes, nor give any indication that these errors could pose a threat to patient safety.
• The system distinguished between errors that halted the machine, requiring a restart, and errors which merely paused the machine (which allowed operators to continue with the same settings using a keypress). However, some errors which endangered the patient merely paused the machine, and the frequent occurrence of minor errors caused operators to become accustomed to habitually unpausing the machine.
  • One failure occurred when a particular sequence of keystrokes was entered on the VT-100 terminal which controlled the PDP-11 computer: if the operator were to press "X" to (erroneously) select 25 MeV photon mode, then use "cursor up" to edit the input to "E" to (correctly) select 25 MeV Electron mode, then "Enter", all within eight seconds of the first keypress and well within the capability of an experienced user of the machine, the edit would not be processed and an overdose could be administered. These edits were not noticed as it would take 8 seconds for startup, so it would go with the default setup.
• The design did not have any hardware interlocks to prevent the electron-beam from operating in its high-energy mode without the target in place.
• The engineer had reused software from the Therac-6 and Therac-20, which used hardware interlocks that masked their software defects. Those hardware safeties had no way of reporting that they had been triggered, so preexisting errors were overlooked.
• The hardware provided no way for the software to verify that sensors were working correctly. The table-position system was the first implicated in Therac-25's failures; the manufacturer revised it with redundant switches to cross-check their operation.
• The software set a flag variable by incrementing it, rather than by setting it to a fixed non-zero value. Occasionally an arithmetic overflow occurred, causing the flag to return to zero and the software to bypass safety checks.

Leveson notes that a lesson to be drawn from the incident is to not assume that reused software is safe: "A naive assumption is often made that reusing software or using commercial off-the-shelf software will increase safety because the software will have been exercised extensively. Reusing software modules does not guarantee safety in the new system to which they are transferred..." In response to incidents like those associated with Therac-25, the IEC 62304 standard was created, which introduces development life cycle standards for medical device software and specific guidance on using software of unknown pedigree.

See also

• Computer ethics
• High integrity software
• IEC 62304
• Ionizing radiation
• List of civilian radiation accidents
• Radiation protection
• Radioactive scrap metal
• List of orphan source incidents
• 1962 Mexico City radiation accident
• Goiânia accident
• Nuclear and radiation accidents and incidents
• Ciudad Juárez cobalt-60 contamination incident
• Samut Prakan radiation accident
• 1990 Clinic of Zaragoza radiotherapy accident

Further reading

• Gallagher, Troy. THERAC-25: Computerized Radiation Therapy. Archived from the original on 2007-12-12. (short summary of the Therac-25 Accidents)