US6928583B2 - Apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep - Google Patents

Apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep Download PDF

Info

Publication number
US6928583B2
US6928583B2 US09/832,466 US83246601A US6928583B2 US 6928583 B2 US6928583 B2 US 6928583B2 US 83246601 A US83246601 A US 83246601A US 6928583 B2 US6928583 B2 US 6928583B2
Authority
US
United States
Prior art keywords
computing element
local
output
fault
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime, expires
Application number
US09/832,466
Other versions
US20020152418A1 (en
Inventor
Gerry Griffin
Michael McLoughlin
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Stratus Technologies Ireland Ltd
Original Assignee
Stratus Technologies Bermuda Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Stratus Technologies Bermuda Ltd filed Critical Stratus Technologies Bermuda Ltd
Priority to US09/832,466 priority Critical patent/US6928583B2/en
Assigned to STRATUS TECHNOLOGIES INTERNATIONAL, S.A.R.L. reassignment STRATUS TECHNOLOGIES INTERNATIONAL, S.A.R.L. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GRIFFIN, GERRY, MCLOUGHLIN, MICHAEL
Priority to AU2002257131A priority patent/AU2002257131A1/en
Priority to PCT/US2002/011192 priority patent/WO2002084488A2/en
Assigned to STRATUS TECHNOLOGIES BERMUDA LTD. reassignment STRATUS TECHNOLOGIES BERMUDA LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: STRATUS TECHNOLOGIES INTERNATIONAL, S.A.R.L.
Assigned to JPMORGAN CHASE BANK, AS ADMINISTRATIVE AGENT reassignment JPMORGAN CHASE BANK, AS ADMINISTRATIVE AGENT SECURITY AGREEMENT Assignors: STRATUS TECHNOLOGIES BERMUDA LTD.
Publication of US20020152418A1 publication Critical patent/US20020152418A1/en
Assigned to JPMORGAN CHASE BANK, AS ADMINISTRATIVE AGENT reassignment JPMORGAN CHASE BANK, AS ADMINISTRATIVE AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: STRATUS TECHNOLOGIES BERMUDA LTD.
Publication of US6928583B2 publication Critical patent/US6928583B2/en
Application granted granted Critical
Assigned to STRATUS TECHNOLOGIES BERMUDA, LTD. reassignment STRATUS TECHNOLOGIES BERMUDA, LTD. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: JPMORGAN CHASE BANK, N.A., FORMERLY KNOWN AS JPMORGAN CHASE BANK
Assigned to GOLDMAN SACHS CREDIT PARTNERS L.P. reassignment GOLDMAN SACHS CREDIT PARTNERS L.P. PATENT SECURITY AGREEMENT (FIRST LIEN) Assignors: STRATUS TECHNOLOGIES BERMUDA LTD.
Assigned to DEUTSCHE BANK TRUST COMPANY AMERICAS reassignment DEUTSCHE BANK TRUST COMPANY AMERICAS PATENT SECURITY AGREEMENT (SECOND LIEN) Assignors: STRATUS TECHNOLOGIES BERMUDA LTD.
Assigned to STRATUS TECHNOLOGIES BERMUDA LTD. reassignment STRATUS TECHNOLOGIES BERMUDA LTD. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: JPMORGAN CHASE BANK
Assigned to STRATUS TECHNOLOGIES BERMUDA LTD. reassignment STRATUS TECHNOLOGIES BERMUDA LTD. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: GOLDMAN SACHS CREDIT PARTNERS L.P.
Assigned to THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT reassignment THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS COLLATERAL AGENT INDENTURE PATENT SECURITY AGREEMENT Assignors: STRATUS TECHNOLOGIES BERMUDA LTD.
Assigned to JEFFERIES FINANCE LLC, AS ADMINISTRATIVE AGENT reassignment JEFFERIES FINANCE LLC, AS ADMINISTRATIVE AGENT SUPER PRIORITY PATENT SECURITY AGREEMENT Assignors: STRATUS TECHNOLOGIES BERMUDA LTD.
Assigned to STRATUS TECHNOLOGIES BERMUDA LTD. reassignment STRATUS TECHNOLOGIES BERMUDA LTD. RELEASE OF PATENT SECURITY AGREEMENT (SECOND LIEN) Assignors: WILMINGTON TRUST NATIONAL ASSOCIATION; SUCCESSOR-IN-INTEREST TO WILMINGTON TRUST FSB AS SUCCESSOR-IN-INTEREST TO DEUTSCHE BANK TRUST COMPANY AMERICAS
Assigned to STRATUS TECHNOLOGIES BERMUDA LTD. reassignment STRATUS TECHNOLOGIES BERMUDA LTD. RELEASE OF SUPER PRIORITY PATENT SECURITY AGREEMENT Assignors: JEFFERIES FINANCE LLC
Assigned to STRATUS TECHNOLOGIES BERMUDA LTD. reassignment STRATUS TECHNOLOGIES BERMUDA LTD. RELEASE OF INDENTURE PATENT SECURITY AGREEMENT Assignors: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Assigned to SUNTRUST BANK reassignment SUNTRUST BANK SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: STRATUS TECHNOLOGIES BERMUDA LTD.
Assigned to STRATUS TECHNOLOGIES IRELAND LTD. reassignment STRATUS TECHNOLOGIES IRELAND LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: STRATUS TECHNOLOGIES BERMUDA LTD.
Assigned to TRUST BANK (AS SUCCESSOR BY MERGER TO SUNTRUST BANK) reassignment TRUST BANK (AS SUCCESSOR BY MERGER TO SUNTRUST BANK) SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: STRATUS TECHNOLOGIES IRELAND LIMITED
Assigned to CERBERUS BUSINESS FINANCE AGENCY, LLC, AS COLLATERAL AGENT reassignment CERBERUS BUSINESS FINANCE AGENCY, LLC, AS COLLATERAL AGENT GRANT OF SECURITY INTEREST IN PATENT RIGHTS Assignors: STRATUS TECHNOLOGIES BERMUDA LTD., STRATUS TECHNOLOGIES IRELAND LIMITED
Assigned to STRATUS TECHNOLOGIES BERMUDA LTD., STRATUS TECHNOLOGIES IRELAND LIMITED reassignment STRATUS TECHNOLOGIES BERMUDA LTD. RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (057254/0557) Assignors: CERBERUS BUSINESS FINANCE AGENCY, LLC
Adjusted expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/1629Error detection by comparing the output of redundant processing systems
    • G06F11/1641Error detection by comparing the output of redundant processing systems where the comparison is not performed by the redundant processing components
    • G06F11/1645Error detection by comparing the output of redundant processing systems where the comparison is not performed by the redundant processing components and the comparison itself uses redundant hardware
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/1675Temporal synchronisation or re-synchronisation of redundant processing components

Definitions

  • the present invention relates generally to computer systems and more specifically to a method and apparatus for enabling fault-tolerant server to execute in lockstep.
  • computers Over the past decade, the use of computers and related technology has increased tremendously. In particular, computers often support air traffic control systems, banking systems, and mission critical defense systems, such as computer systems controlling the launch and flight of defense missiles. Deployed in such a ubiquitous manner, the computers can cause severe problems in the functioning of society if any were to fail. Because of the potential for far-reaching adverse effects in the event of failure, computers are being required to ensure ever-higher reliability. Fault-tolerant computers are computers that generally provide this reliability aspect in such systems.
  • a fault-tolerant computer typically includes one or more redundant central processor units (CPUs) and one or more redundant input-output (I/O) boards, or subsystems.
  • the redundant CPUs often execute in “lockstep,” that is, each CPU executes substantially identical copies of an operating system and application programs and executes substantially identical instruction streams, substantially simultaneously, or in cycle-by-cycle synchronism. This enables a first CPU to replace a second CPU upon the failure of the second CPU without loss of operation of the fault-tolerant server. Such a replacement of CPUs is unnoticeable to the user of the fault-tolerant computer.
  • the I/O subsystems To verify that the redundant CPUs are executing identical instruction streams, the I/O subsystems typically compare the I/O instructions that each redundant CPU generates. When the redundant CPUs and I/O subsystems are included in a single system, enabling the verification that the CPUs execute in lockstep is readily obtainable because each I/O subsystem can communicate with each CPU to compare the generated instructions.
  • the present invention relates to apparatus and methods for two computing elements in a fault-tolerant server to execute instructions in lockstep.
  • the invention comprises a first computing element and a second computing element and each communicates with a communications link.
  • the first computing element provides a first instruction to the communications link.
  • the second computing element provides a second instruction to the communications link.
  • the first computing element also communicates with a first local I/O subsystem and the second computing element communicates with a second local I/O subsystem.
  • one of the local I/O subsystems compares the first instruction and the second instruction. The local I/O then indicates a fault of one of the computing elements upon the detection of a miscompare of the first instruction and the second instruction.
  • each computing element includes a central processing unit (CPU) that communicates with a respective local mass storage device.
  • the communications link comprises a switching fabric that communicates with the respective CPU and both of the respective I/O subsystems.
  • the switching fabric communicates with the I/O subsystem on the opposite computing element over a backplane link and a backplane.
  • the invention consists of a method for a first computing element and a second computing element to execute in lockstep in a fault-tolerant server.
  • the method includes the step of establishing communication between a first computing element and a communications link.
  • the method further includes the step of establishing communication between a second computing element and the communications link.
  • the first computing element transmits a first instruction to the communications link and the second computing element transmits a second instruction to the communications link.
  • an I/O subsystem compares the first instruction and the second instruction and indicates a fault of one of the computing elements in response to a miscompare.
  • the method further includes the step of transmitting a stop command to each computing element when the first instruction does not equal the second instruction.
  • the method may also include detecting an error introduced by the communications link.
  • the method can additionally include assigning a priority to each respective computing element and determining whether one of the computing elements is faulty based on the assigned priority.
  • FIG. 1 is a block diagram of a prior art computer system
  • FIG. 2 is a block diagram of an embodiment of a fault-tolerant computer system
  • FIG. 3 is a more detailed block diagram of an embodiment of a fault-tolerant computer system
  • FIG. 4 is a flow diagram generally illustrating an embodiment of a process for handling errors in the fault-tolerant computer system shown in FIG. 2 ;
  • FIG. 5 is a flow diagram generally illustrating an embodiment of a process to identify and isolate a faulty CPU
  • FIG. 6 is a block diagram of an embodiment of a fault-tolerant server
  • FIG. 7 is a block diagram of a computing element of the fault-tolerant server of FIG. 6 ;
  • FIG. 8 is a more detailed block diagram of the fault-tolerant server of FIG. 6 ;
  • FIG. 9 is a flow diagram illustrating an embodiment of a process to compare instructions generated by each computing element of the fault-tolerant server of FIG. 6 ;
  • FIG. 10 is a block diagram of the fault-tolerant server of FIG. 6 mounted in a cabinet fashion.
  • a typical computer system 20 such as a server, as known in the prior art includes a central processor unit (CPU) 22 , a main memory unit 24 for storing programs and/or data, an input/output (I/O) subsystem 26 , a display device 28 , and a system communications bus 30 coupling these components to allow communication between these units.
  • the memory 24 may include random access memory (RAM), read only memory (ROM), and one or more memory registers.
  • the computer system 20 typically also has one or more peripheral devices, such as input devices 32 .
  • the computer system 20 may include a keyboard 34 (e.g., an alphanumeric keyboard and/or other types of keyboards such as a reduced-key keyboard, or a musical keyboard) and a computer pointing device 36 for translating user movements into computer gestures (e.g., a mouse, a track ball, a track pad, a digitizing tablet, a joystick, a data glove).
  • a keyboard 34 e.g., an alphanumeric keyboard and/or other types of keyboards such as a reduced-key keyboard, or a musical keyboard
  • a computer pointing device 36 for translating user movements into computer gestures (e.g., a mouse, a track ball, a track pad, a digitizing tablet, a joystick, a data glove).
  • the computer system 20 typically also includes one or more mass storage devices, such as a hard disk drive 38 and a floppy disk drive 40 for receiving floppy disks such as 3.5-inch disks.
  • Other additional peripheral devices 42 also can be part of the computer system 20 including output devices (e.g., printer or plotter) and/or optical disk drives for receiving, reading, and/or writing digital data on a CD-ROM.
  • one or more computer programs shown in phantom define the operational capabilities of the computer system 20 . These programs can be loaded onto the hard disk drive 38 and/or into the main memory 24 of the computer CPU 22 via the floppy disk drive 40 . A user of the computer system 20 may execute these applications by using the computer-pointing device 36 to double-click on an icon related to the applications.
  • one or more of the computer system's mass storage devices such as the hard disk drive 38 or the other additional peripheral devices 42 (e.g., a CD-ROM 42 ), stores the controlling software program(s) and all of the data utilized by the program(s).
  • the system communications bus 30 allows data to be transferred between the various components in the computer system 20 .
  • the CPU 22 may retrieve program data from the main memory 24 over the system communications bus 30 .
  • Various system busses 30 are standard in computer systems 20 , such as the Video Electronics Standards Association (VESA) Local Bus, the industry standard architecture (ISA) bus, the Extended Industry Standard Architecture bus (EISA), the Micro Channel Architecture bus (MCA) and a Peripheral Component Interconnect (PCI) bus.
  • VESA Video Electronics Standards Association
  • ISA industry standard architecture
  • EISA Extended Industry Standard Architecture bus
  • MCA Micro Channel Architecture
  • PCI Peripheral Component Interconnect
  • multiple system communication busses 30 may be used to provide access to different units of the system 20 .
  • a computer system 20 may use a PCI bus to connect a CPU 22 to peripheral devices 28 , 34 , 36 , 38 , 40 , 42 and concurrently to connect the CPU 22 to main memory 24 using an MCA bus.
  • Other embodiments include a system bus 30 comprised of other bus architectures, or combination of bus architectures, such as an Accelerated Graphics Port (AGP) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), a Personal Computer Memory Card Industry Association (PCMCIA) bus, a NuBus, a TURBOchannel bus, a Multibus, a STD bus, or a Versa Module Europa (VME) bus.
  • AGP Accelerated Graphics Port
  • SCSI Small Computer System Interface
  • USB Universal Serial Bus
  • PCMCIA Personal Computer Memory Card Industry Association
  • NuBus NuBus
  • TURBOchannel a Multibus
  • Multibus a STD bus
  • VME Versa Module Europa
  • a redundant, fault-tolerant system achieves an extremely high level of availability by using redundant components and data paths to insure uninterrupted operation.
  • a redundant, fault-tolerant system may be provided with any number of redundant components.
  • Some configurations include dual-mode redundant (DMR) systems, which include duplicates of certain hardware units found in FIG. 1 , for example, duplicate, redundant CPUs 22 and main memory units 24 executing substantially identical instruction streams.
  • DMR dual-mode redundant
  • TMR triple-mode redundant
  • FIG. 2 one embodiment of a TMR, fault-tolerant computer system 20 is shown that includes three CPU boards 22 , 22 ′, 22 ′′ (generally 22 ), at least two I/O subsystems 26 , 26 ′, 26 ′′, 26 ′′′ (generally 26 ), redundant communications busses 30 , 30 ′, 30 ′′, 30 ′′′ (generally 30 ), one or more first peripheral busses 64 a through 64 m (generally 64 ), and one or more peripheral devices 42 a through 42 n (generally 42 ).
  • Each of the CPU boards 22 communicates with the first I/O subsystem 26 through the first communications bus 30 .
  • Each of the CPU boards 22 communicates with the second I/O subsystem 26 ′ through the second, redundant communications bus 30 ′.
  • the CPU board 22 is a module, or chassis, while in yet other embodiments the CPU board 22 is a single chip.
  • the system communication busses 30 are standard system busses such as those described for the computer system 20 illustrated in FIG. 1 (ISA, VESA, EISA, MCA, PCI, and the like).
  • peripheral devices 42 include the peripheral devices previously identified in FIG. 1 , e.g., a display device (e.g., a monitor), a hard disk drive, a CD ROM drive, one or more input device(s) (e.g., a keyboard or a computer pointing device), a printer, a plotter, and a floppy disk drive 40 .
  • the fault-tolerant computer system 20 includes more than two I/O subsystems (e.g., 26 ′′ and 26 ′′′ shown in phantom view) to allow the fault-tolerant computer system 20 to control additional peripheral devices 42 .
  • the additional I/O subsystems 26 ′′, 26 ′′′ are similarly in communication with the CPU boards 22 through additional communication busses 30 ′′ and 30 ′′′, also shown in phantom.
  • the CPU boards 22 are redundant CPU boards 22 executing substantially identical instruction streams. By executing substantially identical instruction streams, the CPU boards 22 are configured in a “failover” mode. That is, at any instant in time, one CPU board 22 , e.g., the second CPU board 22 ′, remains ready to replace the first CPU board 22 upon a failure of the first CPU board 22 . As a consequence of the replacement, the second CPU board 22 ′ experiences no loss of data, as the second CPU board 22 ′ operates in place of the first CPU board 22 .
  • the I/O subsystems 26 are redundant components configured in failover mode. That is, at any instant in time, I/O communications between the CPU boards 22 and the peripheral devices 42 are communicated through one of the I/O subsystems 26 , e.g., the first I/O subsystem 26 , but are not simultaneously communicated through the second I/O subsystem 26 ′. The second, I/O subsystem 26 ′ remains ready to replace the first I/O subsystem 26 in the event of a failure. Failover of the I/O subsystems 26 is controlled by fault-tolerant control elements discussed later in more detail.
  • each I/O subsystems 26 communicates with a mass storage device 45 a , 45 b (generally 45 ).
  • Each mass storage device 45 can be grouped with another mass storage device 45 to form a pair of devices 47 , such as a pair of disks.
  • the fault-tolerant computer system 20 simultaneously writes the data to another mass storage device (e.g., 45 b ).
  • This process is known to those skilled in the art as “disk mirroring”. The mirroring of the data enables the fault-tolerant computer system 20 to access the same data from the second mass storage device 45 b following a failure of the first mass storage device 45 a , and vice-versa.
  • a mass storage device 45 may be, without limitation, a hard disk, CD-ROM, magnetic disk, or magneto-optical drive.
  • the pair of mass storage devices 45 are part of a redundant array of independent disks (RAID arrays) used as failure-tolerant persistent mass storage.
  • the fault-tolerant computer system 20 processes each write transaction to the mass storage devices 45 in parallel, writing it to each device 45 in the array. If the computer system 20 fails, then the mass storage device 45 with the most accurate set of contents available can be used as a master, copying all its contents to the other devices 45 in the array (RAID level 1).
  • Another solution not only stores one copy of the transaction information across multiple mass storage devices 45 , but also stores parity information concerning the transaction data (RAID level 5).
  • the mass storage devices 45 are Fibre Channel disks.
  • the mass storage devices 45 are a group of disks.
  • the I/O subsystem 26 associated with the failed mass storage device 45 can no longer access the data previously stored on the failed mass storage device 45 .
  • the I/O subsystem 26 associated with the failed mass storage device 45 a cannot access the mirrored mass storage device 45 b because the mirrored mass storage device 45 b is associated with an independent I/O subsystem 26 ′.
  • the I/O subsystem 26 associated with the failed mass storage device 45 is replaced by a redundant I/O subsystem 26 ′ to prevent loss of data (i.e., failover).
  • a failure of the first mass storage device 45 a causes the first CPU board 22 to connect to the redundant, second I/O subsystem 26 ′ through bus 30 ′.
  • each CPU board 22 contains at least one processor 44 and the main memory 24 .
  • each CPU board 22 contains multiple processors 44 , 44 ′, 44 ′′, and 44 ′′′ (generally 44 ).
  • each of the multiple processors 44 of a CPU board 22 may process different instruction streams.
  • Respective processors 44 on different CPU boards 22 e.g., processor 44 on CPU board 22 and processor 44 on CPU board 22 ′
  • the processors 44 are selected from the “x86” family of processors manufactured by Intel Corporation of Santa Clara, Calif.
  • the x86 family of processors includes the 80286 processor, the 80386 processor, the 80486 processor, and the Pentium®, Pentium® II, Pentium® III, Pentium® III XeonTM, Pentium IV processors, and the 64-bit ItaniumTM family of processors.
  • the processors 44 are selected from the “680x0” family of processors manufactured by Motorola Corporation of Schaumburg, Ill.
  • the 680x0 family of processors includes the 68000, 68020, 68030, and 68040 processors.
  • processors 44 are processor families include the Power PC line of processors manufactured by Motorola Corporation, the Alpha line of processors manufactured by Compaq Corporation of Houston, Tex., and the Crusoe line of processors manufactured by Transmeta Corporation of Santa Clara, Calif.
  • processor 44 is an Athlon processor, manufactured by Advanced Micro Devices (AMD) of Sunnyvale, Calif.
  • the redundant CPU boards 22 execute in “lockstep,” that is, each CPU board 22 executes substantially identical copies of the operating system and application programs, substantially simultaneously, in cycle-by-cycle synchronism.
  • the replicated CPU boards 22 store identical data in the replicated main memory 24 at all times.
  • a single reference clock source 48 (shown in phantom) is provided in communication with each of the CPU boards 22 , and each of the CPU boards 22 synchronizes to the common clock source 48 .
  • the redundant CPU boards 22 execute identical operating systems and application programs and execute substantially equivalent instruction streams in a loosely synchronized, or “loose-stepped” manner.
  • each of the CPU boards 22 can include its own clock source 49 , 49 ′, 49 ′′ (generally 49 ) shown in phantom in FIG. 2 , running asynchronously with respect to the clock sources 49 of the other CPU boards 22 .
  • Loose-step synchronization does not require the common clock source 48 used for lock-step embodiments.
  • the CPU boards 22 maintain synchronization of the fault-tolerant computer system 20 by counting the instructions processed and initiating a synchronizing procedure after counting some quantum of the instructions. In other embodiments, the CPU boards 22 maintain synchronization of the fault-tolerant computer system 20 by monitoring events, such as memory references. In these embodiments any of the CPU boards 22 performing a monitored event before the remainder of the loose-stepped CPU boards 22 is stalled. Once the remainder of the CPU boards 22 perform the monitored event, the stalled CPU boards 22 are allowed to continue processing.
  • the fault-tolerant computer system 20 includes at least a first redundant CPU 22 and a second redundant CPU 22 ′, a first and second communications channel 30 and 30 ′, and at least one I/O subsystem 26 .
  • the redundant CPUs 22 , 22 ′ are in communication with the I/O subsystem 26 through the respective communications channel 30 , 30 ′.
  • the I/O subsystem 26 communicates with one or more peripheral devices 42 z through the I/O bus 58 , and optionally through the peripheral busses 64 as shown in FIG. 2 .
  • I/O instructions can be generated at each of the CPUs 22 , or the peripheral devices 42 , and can be directed at any of the other CPUs 22 or peripheral devices 42 .
  • the I/O instructions include memory read or writes, configuration read or writes, mass storage device read or writes, or other special instructions.
  • the I/O subsystem 26 includes an I/O bus 58 and I/O bus interface 68 .
  • the I/O bus 58 interconnects one or more of the peripheral devices 42 to the I/O bus interface 68 .
  • the I/O subsystem 26 includes a peripheral controller 72 shown in phantom.
  • the I/O subsystem 26 includes I/O fault-tolerant logic 52 and optionally a voter delay buffer 98 , shown in phantom.
  • the I/O fault tolerant logic 52 communicates with the I/O bus interface 68 and the voter delay buffer 98 , if present.
  • the I/O fault-tolerant logic 52 is implemented within the I/O bus interface 68 , such as on a single application specific integrated circuit (ASIC).
  • ASIC application specific integrated circuit
  • the I/O fault tolerant logic 52 detects faults within the fault-tolerant computer system 20 .
  • the I/O fault-tolerant logic 52 includes a comparator 92 that performs comparisons of the I/O instruction streams.
  • the CPU instructions are substantially identical where the same instructions occur for each of the redundant CPUs 22 on the same clock cycle.
  • voting of the I/O instructions from the CPU 22 is conducted during each clock cycle.
  • the I/O fault tolerant logic 52 compares the equivalent I/O instruction streams from each of the redundant CPUs 22 to identify a fault upon the detection of a miscompare.
  • the I/O fault-tolerant logic 52 also includes a buffer 94 , shown in phantom.
  • the buffer 94 can be used for holding I/O instructions from one or more of the redundant CPUs 22 in a loose-step fault-tolerant system 20 .
  • the time of voting, or comparison, in a loose-step embodiment is determined by some event.
  • the I/O fault-tolerant logic 52 identifies errors when at least one of the inputs to the comparator 92 is different from the other equivalent, redundant inputs.
  • the comparator 92 substantially simultaneously performs a bit-by-bit comparison of the voted I/O instruction. When the comparator 92 determines it received input data from each of the redundant CPUs 22 that are identical at the time of voting, no errors are detected.
  • the redundant I/O instructions are stored in respective registers and voting occurs such that the equivalent bits of each register are compared by a plurality of comparators, one for each bit of the instruction.
  • the I/O fault-tolerant logic 52 provides the fault-tolerant computer system 20 with a fault-tolerant capability.
  • the I/O fault-tolerant logic 52 is implemented as a separate set of logic on the I/O subsystem 26 .
  • the I/O fault-tolerant logic 52 may be provided as an ASIC, a field programmable gate array (FPGA), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a programmable logic device (PLD), or a read-only memory device (ROM).
  • the I/O fault-tolerant logic 52 compares the results of each operation performed by the separate CPU boards 22 to the results of the same operation performed on one of the other CPU boards 22 ′, 22 ′′. If the I/O fault logic 52 determines a discrepancy then a failure has occurred.
  • the DMR fault-tolerant computer system 20 may include a priority module 86 (shown in phantom) and the voter delay buffer 98 (shown in phantom).
  • the I/O fault-tolerant logic 52 includes a priority register 46 , also shown in phantom.
  • each CPU board 22 contains a CPU bus interface 66 and CPU fault-tolerant logic 50 .
  • the CPU fault-tolerant logic 50 communicates with the CPU bus interface 66 .
  • the priority register 46 of the I/O fault-tolerant logic 52 communicates with the priority module 86 through a CPU PRIORITY signal 88 .
  • the CPU fault-tolerant logic 50 on each redundant CPU 22 communicates with the I/O fault-tolerant logic 52 through a respective command line 96 , 96 ′.
  • the priority module 86 assigns a priority to each of the redundant CPUs 22 .
  • the assigned priority is stored in the priority register 46 .
  • the I/O fault-tolerant logic 52 or the priority module 86 may update or change the assigned priority during operation.
  • the priority can be established by a number of factors, such as historical performance of the CPU 22 or prior/current diagnostic results from the maintenance and diagnostics module 84 .
  • the I/O fault-tolerant logic comparator 92 performs a bit-by-bit cycle compare procedure on the data output streams passing into the I/O subsystem 26 on the I/O busses 30 and 30 ′.
  • the I/O subsystem 26 includes buffer 94 , shown in phantom, before the comparator 92 , to hold the transactions of the data output streams that will be compared from each of the redundant CPU boards 22 . When each of the transactions to be compared from each of the respective redundant CPU boards 22 arrives at the buffer 94 , the transactions are provided to the comparator 92 .
  • the comparator 92 may be, for example, an XOR gate or any other known component for comparing two bit values. If the cycle-compare procedure detects a difference between the two data output streams, this may be an indication that one of the CPU boards 22 and 22 ′ is malfunctioning. Accordingly, the I/O subsystem 26 responds by issuing a “STOP” command to both the first CPU 22 and the second CPU 22 ′ over a first command line 96 and a second command line 96 ′ respectively.
  • the I/O fault tolerant logic 52 generates a command to take the suspected faulty CPU 22 off line.
  • the command is communicated to the determined faulty CPU 22 over the respective command line 96 .
  • voting results in a miscompare within a DMR system 20 additional information is required to determine which CPU 22 is faulty and should be taken off line.
  • the I/O fault-tolerant logic 52 determines which of the two redundant CPUs 22 is faulty by requesting that the maintenance and diagnostics module 84 of each redundant CPU 22 perform diagnostics to verify the integrity of the respective CPU 22 .
  • the DMR fault-tolerant system 20 restricts the propagation of faulty data by issuing a stop command to the CPUs 22 until diagnostics are completed, and by directing commands issued by the CPUs from the time of the issued faulty I/O instruction to the time that the stop command is executed by the CPUs 22 .
  • the I/O instructions in the “pipeline” from each of the two redundant CPUs 22 are directed to a respective first and second first-in-first-out buffer (FIFO) buffer 102 , 102 ′ within the voter delay buffer 98 . If one of the two CPUs, for example CPU 22 , diagnoses a failure, that CPU 22 is taken off line for repair and replacement and any commands in the related FIFO 102 are discarded, whereas the commands associated with the “good” CPU 22 ′ are reissued from the respective FIFO 102 ′.
  • FIFO first-in-first-out buffer
  • the I/O fault-tolerant logic 52 can determine which of the two redundant CPUs 22 is suspected faulty by defining the CPU 22 with the lower priority value stored in the priority register 46 as being the faulty CPU. In yet another embodiment, combinations of results from the maintenance and diagnostics modules 84 and the priority values can be used by the fault-tolerant logic 50 , 52 to determine which CPU 22 is suspected faulty.
  • the data output streams on the I/O busses 30 and 30 ′ are bit-by-bit compared by the comparator 92 (step 110 ) to provide a comparative reading from which it can be determined if there are differences between the monitored data output streams. If there are no such differences detected, the comparator 92 continues to monitor the data output streams. If the comparator 92 detects differences, the I/O fault-tolerant logic 52 issues a STOP command (step 112 ). In some embodiments, the issuance of a STOP command prompts the initiation of a CPU diagnostic procedure on each of the redundant CPUs 22 to check the status of each CPU 22 . Subsequently, the data output streams on the I/O busses 30 and 30 ′ are diverted to the voter delay buffer 98 (step 114 ).
  • the first CPU board 22 executes its ongoing diagnostic procedure (step 116 ). If the diagnosis indicates that the first CPU board 22 is malfunctioning, the first CPU board 22 is isolated (step 118 ) and operation of the computer system 20 continues with the second CPU board 22 ′.
  • the data stored in the second FIFO buffer 102 ′ is output over the system I/O bus 30 (step 120 ) and thereafter subsequently processed data from the second CPU board 22 ′ is output over the system I/O bus 30 .
  • the second CPU board 22 Contemporaneously with the ongoing diagnosis procedure in the first CPU board 22 (step 116 ), the second CPU board 22 also executes its diagnostic procedure (step 122 ). If, on the other hand, the resulting diagnosis indicates that the second CPU board 22 ′ is malfunctioning, the second CPU board 22 ′ is isolated (step 124 ) and operation of the computer system 20 continues with the first CPU board 22 .
  • the data stored in the first FIFO buffer 102 is output over the system I/O bus 30 (step 126 ) and subsequent processed data from the first CPU board 22 is output over the system I/O bus 30 .
  • the relative CPU priorities determined by the priority module 86 and maintained within the priority register 46 are used as the determinative factor.
  • the relative CPU priorities are read from each of the priority registers 46 , 46 ′ to establish which of the first CPU board 22 or the second CPU board 22 ′ has the higher priority (step 128 ).
  • the relative priorities of the CPU boards 22 have been determined by one or more criteria, such as their operational histories or the comparative cumulative record of their internal error corrections. If the second CPU board 22 ′ has been assigned the higher priority, for example, the computer system 20 selects the first CPU board 22 as the malfunctioning CPU board 22 and continues to operate with only the second CPU board 22 ′ (step 130 ).
  • the data stored in the second FIFO buffer 102 ′ is output (step 132 ).
  • the computer system 20 selects the second CPU board 22 ′ as the malfunctioning CPU board 22 and the operation of the computer system 20 continues with the first CPU board 22 (step 130 ). In this case, the data stored in the first FIFO buffer 102 is output (step 132 ).
  • TMR fault-tolerant computer system 20 is similar to that shown in FIG. 2 with the internal detail of the CPU 22 and the I/O subsystem 26 shown in FIG. 3 .
  • the I/O fault-tolerant logic 52 votes, or compares each of the input values of each I/O transaction generated by the three redundant CPUs 22 and received at the I/O bus interface 68 to determine if the three input instructions are not the same.
  • the voting constitutes a bit-by-bit comparison of each of the bits of the three redundant I/O instructions, performed within the comparator 92 .
  • the comparison determines if all the inputs are the same or, conversely, if one of the inputs is different from the other and, from that, identifies which one of the three differs from the others.
  • the comparator 92 is implemented in hardware.
  • the comparator 92 includes combinatorial logic.
  • the comparator 92 is implemented in software.
  • the comparator 92 when the comparator 92 determines a miscompare among the three input I/O instructions, the comparator 92 also identifies which of the three CPUs 22 , referred to as the minority CPU 22 , is not in agreement with the other two CPUs 22 , referred to as the majority CPUs 22 .
  • the majority value is determined to be the correct valid instruction and the minority is determined to be faulty.
  • the I/O fault tolerant logic 52 commands that the suspected faulty CPU 22 be taken off line.
  • the priority module and priority register 46 can also be used to make further determinations of identifying a failed CPU 22 in a miscompare.
  • the maintenance and diagnostics module 84 and voter delay buffer 98 can also be used to make further determinations of identifying a failed CPU 22 in a miscompare.
  • the TMR fault-tolerant computer system 20 will function as the previously identified DMR fault-tolerant computer system 20 when one of the three CPUs 22 has been taken off-line, leaving two remaining on-line CPUs 22 .
  • the I/O fault-tolerant logic 52 compares I/O transactions from each of the CPU boards 22 , 22 ′, 22 ′′ (step 140 ).
  • the I/O fault-tolerant logic comparator 92 performs a voting function by comparing each of the instruction streams by a bit-by-bit comparison and identifying a minority CPU board 22 that produces an I/O instruction stream that does not match the I/O instruction stream from the majority (step 142 )—the other two CPU boards 22 ′, 22 ′′ in a TMR system.
  • the minority CPU board 22 is then taken off-line to avoid the propagation of errors into the system and to allow for diagnostics, repair and replacement of the identified minority CPU board 22 if necessary (step 144 ).
  • the I/O subsystem 26 stops transmitting output data on the I/O bus 58 and routes the data output streams on the busses 30 and 30 ′ to a voter delay buffer 98 via a delay buffer line 100 .
  • the data received from the first CPU board 22 is sent to a first FIFO buffer 102
  • the data received from the second CPU board 22 ′ is sent to a second FIFO buffer 102 ′.
  • the fault-tolerant logic 50 , 52 identifies a faulty CPU 22 and notifies the voter delay buffer which CPU 22 is faulty. The voter delay buffer 98 then releases the buffered commands from the respective FIFO 102 , 102 ′ for the valid CPU 22 and normal processing continues.
  • the maintenance and diagnostic subsystems 84 and 84 ′ continually run their respective diagnostic procedures. It should be understood that, even after the STOP command has been issued to the CPU boards 22 and 22 ′, the I/O subsystem 26 continues to forward input data streams sent by the peripheral devices to the CPU boards 22 and 22 ′. The CPU boards 22 and 22 ′ continue to process the data while running the diagnostic procedures, in accordance with normal operational procedures. Thus, from the point of view of the peripheral devices, the fault-tolerant computer system 20 functions normally.
  • the fault-tolerant server 20 includes a first computing element 150 and a second computing element 150 ′ (generally 150 ).
  • each computing element 150 is an independent motherboard that includes the CPU board 22 of FIG. 2 and FIG. 3 and a “local” I/O subsystem 152 , 152 ′ (generally 152 ).
  • a “local” I/O subsystem 152 is an I/O subsystem that is associated with a particular CPU board 22 .
  • Each computing element 150 also includes a “local” mass storage device 154 , 154 ′ (generally 154 ).
  • a “local” mass storage device 154 is a mass storage device that is associated with a local I/O subsystem, and thus associated with a particular CPU board 22 and computing element 150 .
  • each respective local mass storage device 154 is located on the same board as the respective computing element 150 .
  • the local mass storage device 154 is an external component of the computing element 150 .
  • the local mass storage device 154 also has a disk controller (not shown) associated with the mass storage device 154 and with the particular computing element 150 .
  • the disk controller is implemented in the local I/O subsystem 152 .
  • the disk controller is implemented as an independent component in communication with the I/O subsystem 152 and the local mass storage device 154 .
  • a fault-tolerant server includes two independent computing elements that include devices local to the computing element (i.e., a local I/O subsystem and a local mass storage device), each CPU board typically does not have access to each local I/O subsystem and each mass storage device, unlike the fault-tolerant computer system 20 of FIG. 2 .
  • a typical fault-tolerant server having two independent computing elements cannot use a first CPU board located on the first computing element to access a mass storage device located on the second computing element.
  • the fault-tolerant server 20 additionally includes a backplane 158 .
  • a backplane is a circuit board or framework that supports other circuit boards, devices, and the interconnections among devices, and provides power and data signals to supported devices.
  • a computer system may also have expansion cards plugged into sockets of a backplane to increase the capabilities of the computer system.
  • Backplanes are often described by those skilled in the art as being either “active” or “passive”. Active backplanes contain, in addition to the sockets, logical circuitry that performs computing functions. In contrast, passive backplanes contain almost no computing circuitry.
  • the backplane 158 enables one computing element 150 and, more particularly, one CPU board 22 to access either local I/O subsystem 152 , 152 ′ and thus either mass storage device 154 , 154 ′ even though the mass storage devices 154 , 154 ′ are local to one computing element 150 , 150 ′.
  • the computing elements 150 generally plug into a socket in the backplane 158 , thereby facilitating the swapping of one computing element 150 with another computing element 150 ′ (e.g., replacing a computing element 150 upon a failure).
  • each CPU board 22 , 22 ′ communicates with its respective local I/O subsystem 152 , 152 ′ over a communications link.
  • the communications link includes a first communications channel 162 , 162 ′ ( 162 ′ not shown) (generally 162 ), a switching fabric 166 , 166 ′ ( 166 ′ not shown) (generally 166 ), and a second communications channel 164 , 164 ′ ( 164 ′ not shown) (generally 164 ).
  • the communications link also includes a backplane 158 and a respective backplane link 161 , 161 ′ (generally 161 ).
  • the communications link or some or all of the components that make up the communications link are standard system busses such as those described for the computer system 20 illustrated in FIG. 1 (ISA, VESA, EISA, MCA, PCI, and the like).
  • the communications link or some or all of the components that make up the communications link may be a “twisted-pair” cable, coaxial cable, fiber optic cable, or wireless links, such as radio links or free-optics links.
  • the first CPU board 22 communicates with the first switching fabric 166 over a first communications line 163 a .
  • the first CPU board 22 communicates with the second computing element 150 ′ (i.e., the second switching fabric 166 ′) over a second communications line 163 b , the first backplane link 161 , the backplane 158 , and the second backplane link 161 ′.
  • the first switching fabric 166 communicates with the first local I/O subsystem 152 over a third communications line 165 a .
  • the first switching fabric 166 communicates transmission received from the second computing element 150 ′ to the first local I/O subsystem 152 over the fourth communications line 165 b.
  • the second CPU board 22 ′ communicates with the second switching fabric 166 ′ (not shown) over a first communications line 163 a ′ (not shown).
  • the second CPU board 22 ′ communicates with the first computing element 150 (i.e., the first switching fabric 166 ) over a second communications line 163 b ′ (not shown), the second backplane link 161 ′ (not shown), the backplane 158 , and the first backplane link 161 .
  • the second switching fabric 166 ′ communicates with the second local I/O subsystem 152 ′ (not shown) over a third communications line 165 a ′.
  • the second switching fabric 166 communicates transmission received from the first computing element 150 to the second local I/O subsystem 152 ′ over the fourth communications line 165 b ′ (not shown).
  • first communications lines 163 a , 163 a ′ (generally 163 a ), the second communications lines 163 b , 163 b ′ (generally 163 b ), the third communications lines 165 a , 165 a ′ (generally 165 a ), and/or the fourth communications lines 165 b , 165 b ′ (generally 165 b ) may each be, in one embodiment, a standard system bus such as those described for the computer system 20 illustrated in FIG. 1 (ISA, VESA, EISA, MCA, PCI, and the like).
  • the communications lines 163 a , 163 b , 165 a , 165 b may be a “twisted-pair” cable, coaxial cable, fiber optic cable, or wireless links, such as radio links or free-optics links.
  • the backplane 158 includes an active logic module 176 .
  • the logic module 176 provides the computing elements 150 with system information required on power-up, such as the system identification number, or system ID.
  • the logic module 176 provides the first CPU board 22 with a system ID of 0 (i.e., CPU 0 ) and provides the second CPU board 22 ′ with a system ID of 1 (i.e., CPU 1 ).
  • the logic module 176 may provide a common clock source to the CPU boards 22 on each computing element 150 .
  • the backplane 158 may include a connector interface (not shown) that facilitates the physical and electrical connection of the server 20 to external resources.
  • the connector interface is an external interface that provides a connection to an external network via, for example, an RJ-45 connector or coaxial cable connection.
  • the external connector interface may connect to an external modem of the fault-tolerant server 20 and thus provides a network connection to the internet.
  • the connector interface can also connect to one or more peripheral devices, such as a keyboard or mouse via, for example, a DB-9 connector, a DB-25 connector, or a USB port.
  • the external connector interface facilitates “blind mating” between the backplane 158 and the computing elements 150 . That is, the computing elements 150 need only make connection with the backplane 158 in order to have access to all network and external physical resources.
  • the connector interface is internal to the computing element 150 .
  • Each computing element 150 also includes a mass storage device controller 178 (also referred to throughout as a disk controller) to interface with the local mass storage device 154 .
  • the local mass storage device 154 is a disk drive capable of communicating using the Fibre Channel protocol (i.e., “Fibre Channel disks”).
  • each computing element 150 includes multiple local mass storage devices (e.g., a first local mass storage device 154 a and a second local mass storage device 154 b ) that communicate with the mass storage device controller 178 .
  • the disks 154 may be connected in a loop topology with the I/O subsystem 152 .
  • This arrangement is generally referred to as a Fibre Channel Arbitrated Loop (FC-AL).
  • FC-AL Fibre Channel Arbitrated Loop
  • an I/O subsystem may communicate with multiple FC-ALs.
  • the first local I/O subsystem 152 may directly communicate with the disk drives 154 present on the first FC-AL and may also directly communicate with the remote disk drives 154 present on the remote FC-AL.
  • This communication from one I/O subsystem 152 to either or both FC-ALs may be useful when one of the local I/O subsystems 152 fails because the other local I/O subsystem 152 can communicate with the FC-AL 154 associated with the failed local I/O subsystem 152 to continue normal operation.
  • the communication between the I/O subsystem 152 and the FC-AL is through the backplane 158 .
  • the I/O subsystem 152 communicates directly with the remote FC-AL using a very high density metric (VHDM) connector (not shown).
  • VHDM very high density metric
  • a subset of the connector pins in the VHDM connector provides electrical communication between the I/O subsystem 152 and the remote FC-AL.
  • another subset of the connector pins of the VHDM connector routes the switching fabric 166 located on one computing element 150 to the switching fabric 166 located on the other computing element 150 .
  • Each computing element 150 also includes an I/O synchronization bus, or sync bus, 180 .
  • the sync bus 180 communicates with the backplane 158 and enables synchronization of all of the local I/O subsystems 152 .
  • the sync bus 180 synchronizes state information about each CPU board 22 between the local I/O subsystems 152 .
  • the state information of a CPU board 22 described above includes, for example, if the CPU board 22 is an “on-line” CPU board 22 (i.e., operating correctly), a “broken” CPU board 22 (i.e., operating incorrectly), or an “offline” CPU board 22 (i.e., executing diagnostics or ready to be brought into service).
  • the second computing element 150 ′ includes the same components as the first computing element 150 and the description applies to the second computing element 150 as well with minor modifications.
  • each local I/O subsystem 152 detects faults within the fault-tolerant server 20 by comparing each of the instruction streams that each CPU board 22 produces.
  • the switching fabric 166 used in conjunction with the backplane 158 enables synchronization of each local I/O subsystem 152 in the reception of the I/O instruction streams provided by each CPU board 22 .
  • the switching fabric 166 uses the system ID described above to determine which communications line (e.g., first communications line 163 a , second communications line 163 b ) to route to the backplane link 161 (i.e., to the backplane 158 ) and which communications line (e.g., first communications line 163 a , second communications line 163 b ) to route to the local I/O subsystem 152 that is located on the same computing element 150 as the switching fabric 166 .
  • the communications line e.g., first communications line 163 a , second communications line 163 b
  • the first switching fabric 166 provides a communications path between the first CPU board 22 and the first local I/O subsystem 152 using the first communications line 163 a and the third communications line 165 a of the first communications channel 162 .
  • the first switching fabric 166 also provides a communications path connecting the first communications line 163 a to a first delay module 184 .
  • Data e.g., I/O instructions, are delayed by the delay module 184 prior to transmission of the data to the first local I/O subsystem 152 .
  • the first switching fabric 166 also provides a communications path between the first CPU board 22 and the second local I/O subsystem 152 ′ on the second computing element 150 ′ using the second communications line 163 b of the first communications channel 162 and the first backplane link 161 .
  • the first switching fabric 166 routes all I/O instructions transmitted on the second communications line 163 b to the second local I/O subsystem 152 ′ through the backplane 158 .
  • the second switching fabric 166 ′ provides a communications path between the second CPU board 22 ′ and the second local I/O subsystem 152 ′ using the second communications line 163 b ′ of the first communications channel 162 ′.
  • the second switching fabric 166 ′ also provides a communications path from the second communications line 163 b ′ to a second delay module 184 ′ where data is held prior to transmission to the second local I/O subsystem 152 ′.
  • the first and second delay modules 184 , 184 ′ (generally 184 ) are described in greater detail below.
  • the second switching fabric 166 ′ provides a communications path between the second CPU board 22 ′ and the first local I/O subsystem 152 using the first communications line 163 a ′ of the first communications channel 162 ′.
  • the second switching fabric 166 ′ routes all I/O instructions transmitted on the first communications line 163 a ′ to the first local I/O subsystem 152 through the backplane 158 .
  • the local I/O subsystem 152 that receives the I/O instructions from the CPU boards 22 compares each I/O instruction. To enable such a comparison, the fault-tolerant server 20 establishes (step 204 ) communication between the first computing element 150 and the communications link described above. Similarly, the fault-tolerant server 20 establishes (step 208 ) communication between the second computing element 150 ′ and the communications link.
  • the computing element 150 establishes communications between the CPU board 22 and the first and second local I/O subsystems 152 , 152 ′ using the communication paths that the switching fabric 166 and the backplane 158 provides to each CPU board 22 , as described above.
  • the first CPU board 22 establishes communications between the local I/O subsystems 152 by transmitting a test message to the first local I/O subsystem 152 and the second local I/O subsystem 152 ′ over the first switching fabric 166 , the backplane link 161 , the backplane 158 , the backplane link 161 ′, and the second switching fabric 166 ′.
  • the first local I/O subsystem 152 and the second local I/O subsystem 152 ′ each receives the particular test message
  • the first local I/O subsystem 152 and the second local I/O subsystem 152 ′ each transmit a response message to the first CPU board 22 .
  • each response message identifies the local I/O subsystem 152 that sends the response message to the CPU board 22 .
  • the response message includes the system ID for the computing element 150 which transmitted the response message.
  • the first CPU board 22 then generates (step 212 ) a first I/O instruction, such as to store a datum in the first local mass storage device 154 .
  • the first CPU board 22 receives the first I/O instruction from a peripheral device 42 (shown in FIG. 3 ), such as a modem.
  • the first CPU board 22 transmits the I/O instruction (e.g., “store datum”) to the first switching fabric 166 over the first communications line 163 a.
  • the second computing element 150 ′ simultaneously generates (step 216 ) (or receives from a peripheral device 42 ) a second I/O instruction that is identical to the first instruction (e.g., to store a datum in the first local mass storage device 154 ).
  • the second CPU board 22 ′ transmits the I/O instruction to the second switching fabric 166 ′ over the first communications line 163 a ′.
  • the first switching fabric 166 and the second switching fabric 166 ′ then transmit (step 220 ) the first I/O instruction and the second I/O instruction, respectively, to the first local I/O subsystem 152 . More specifically, the second switching fabric 166 ′ transmits in step 220 the second I/O instruction to the first local I/O subsystem 152 via the first communications line 163 a ′ and the backplane 158 .
  • each switching fabric 166 delays the transmission of the I/O instruction to the local I/O subsystem 152 (located on the same computing element 150 as the switching fabric 166 ) with the delay module 184 noted above.
  • the delay module 184 is tuned to the backplane 158 so that the delay module 184 provides a delay that is equivalent to the amount of time an instruction takes to reach the local I/O subsystem 152 on the opposite computing element 150 .
  • the fault-tolerant logic module 52 (not shown) of the first local I/O subsystem 152 uses the comparator 92 (shown in FIG. 3 ) described above to compare (step 224 ) the first and second I/O instructions (e.g., bit by bit comparison). If the I/O fault-tolerant logic module 52 on the first computing element 150 determines (step 228 ) that differences exist between the first instruction and the second instruction, the first local I/O subsystem 152 issues (step 232 ) a “STOP” command to both the first CPU board 22 and the second CPU board 22 ′.
  • the first local I/O subsystem 152 transmits the “STOP” command to the first switching fabric 166 over the third communications line 165 a and then to the first CPU board 22 over the first communications line 163 a .
  • the first local I/O subsystem 152 transmits the “STOP” command to the second switching fabric 166 ′ over the fourth communications line 165 b , the first backplane link 161 , the backplane 158 , and the fourth communications line 165 b ′.
  • the first local I/O subsystem 152 executes (step 236 ) the I/O instruction, such as by storing the datum in the first local mass storage device 154 .
  • the fault-tolerant server 20 detects errors in communications that the communications channel 162 , the backplane 158 , the local mass storage devices 154 , the local I/O subsystems 152 , and/or the second communications channel 164 introduce.
  • each CPU board 22 introduces a parity bit to the I/O instructions.
  • each local I/O subsystem 152 and/or each local mass storage devices 154 also introduces a parity bit to any communication to either computing element 150 .
  • the communications channel 162 , 164 performs parity checking on any incoming instruction stream (e.g., from either CPU board 22 , from either switching fabric 166 , from the backplane 158 ).
  • the first switching fabric 166 performs parity checking on the first I/O instruction when the switching fabric 166 receives the instruction from the first communications line 163 a . Similarly and in further embodiments, the first switching fabric 166 performs parity checking on the second I/O instruction upon reception of the second I/O instruction from the backplane 158 . If the first switching fabric 166 detects a parity error in the second I/O instruction, the switching fabric 166 alerts the CPU board 22 that the backplane 158 may be faulty (assuming no other error discovered in prior parity checks of the communication of the second I/O instruction before reaching the first switching fabric 166 ).
  • the error detection enables the fault-tolerant server 20 to isolate faults of a CPU board 22 , the backplane 158 , or the switching fabric 166 .
  • the fault-tolerant server 20 may support other protocols for ensuring transmission accuracy, such as, without limitation, Microcom Networking Protocol (MNP), V.42, Hamming coding, and the like.
  • MNP Microcom Networking Protocol
  • V.42 V.42
  • Hamming coding and the like.
  • the first CPU board 22 provides a first I/O instruction to read a datum from the first local mass storage device 154 .
  • the second CPU board 22 ′ provides a second I/O instruction to read the datum from the first local mass storage device 154 .
  • the first I/O subsystem 152 accesses the datum from the first local mass storage device 154 and transmits the datum to each CPU board 22 . More specifically and in one embodiment, the first local I/O subsystem 152 transmits the datum to the second CPU board 22 ′ over the backplane 158 .
  • mirroring software updates the contents of one of the local mass storage devices 154 (e.g., the second local mass storage device 154 ′) so that the contents are identical to the contents of the other local mass storage device 154 (e.g., the first local mass storage device 154 ).
  • the second local mass storage device 154 ′ “mirrors” the first local mass storage device 154 .
  • first computing element 150 and the components therein (e.g., the first CPU board 22 , the first local I/O board 152 ), it should be clear that the description also applies to additional computing elements 150 (e.g., the second computing element 150 ′) and the respective components therein (e.g., the second CPU board 22 ′, the second local I/O subsystem 152 ′).
  • additional computing elements 150 e.g., the second computing element 150 ′
  • respective components therein e.g., the second CPU board 22 ′, the second local I/O subsystem 152 ′.
  • the fault-tolerant server 20 may be organized in a cabinet fashion, with each computing element 150 being a 1U rack-mounted motherboard.
  • a rack-mounted motherboard is a motherboard that is built for installation in a metal frame or cabinet of standard width (typically 19 inches or 23 inches) and mounting arrangements.
  • a “U” as used above is a standard unit of measure for designating the height in computer enclosures and rack cabinets. In one embodiment, a U equals 1.75 inches. For example, a 4U chassis is 7 inches high and a 40U rack cabinet is 70 inches high.
  • the fault-tolerant server 20 includes two 1U rack-mounted motherboards 150 ; thus the fault-tolerant server 20 is a 2U server. Although described as a 2U fault-tolerant server 20 , it should be noted that the fault-tolerant server 20 can use any number of 1U rack-mounted motherboards 150 (e.g., the third computing element 150 ′′ shown in shadow). In other embodiments, the rack-mounted motherboards 150 may be any reasonable height (e.g., 1U, 2U, 3U, 4U).

Abstract

An apparatus and method for a first computing element and a second computing element to execute in lockstep in a fault-tolerant server. In one embodiment, the first computing element provides a first instruction to a communications link and the second computing element provides a second instruction to a communications link. In one embodiment, a first local input-output (I/O) subsystem and a second local I/O subsystem are each in communication with the communications link. The first and/or the second local I/O subsystem compare the first instruction and the second instruction. In one embodiment, the first and second local I/O subsystems indicate a fault of the first computing element or the second computing element. Such a fault may be determined by a miscompare of the first instruction and the second instruction.

Description

FIELD OF THE INVENTION
The present invention relates generally to computer systems and more specifically to a method and apparatus for enabling fault-tolerant server to execute in lockstep.
BACKGROUND OF THE INVENTION
Over the past decade, the use of computers and related technology has increased tremendously. In particular, computers often support air traffic control systems, banking systems, and mission critical defense systems, such as computer systems controlling the launch and flight of defense missiles. Deployed in such a ubiquitous manner, the computers can cause severe problems in the functioning of society if any were to fail. Because of the potential for far-reaching adverse effects in the event of failure, computers are being required to ensure ever-higher reliability. Fault-tolerant computers are computers that generally provide this reliability aspect in such systems.
Typically, a fault-tolerant computer includes one or more redundant central processor units (CPUs) and one or more redundant input-output (I/O) boards, or subsystems. In a fault-tolerant server, the redundant CPUs often execute in “lockstep,” that is, each CPU executes substantially identical copies of an operating system and application programs and executes substantially identical instruction streams, substantially simultaneously, or in cycle-by-cycle synchronism. This enables a first CPU to replace a second CPU upon the failure of the second CPU without loss of operation of the fault-tolerant server. Such a replacement of CPUs is unnoticeable to the user of the fault-tolerant computer.
To verify that the redundant CPUs are executing identical instruction streams, the I/O subsystems typically compare the I/O instructions that each redundant CPU generates. When the redundant CPUs and I/O subsystems are included in a single system, enabling the verification that the CPUs execute in lockstep is readily obtainable because each I/O subsystem can communicate with each CPU to compare the generated instructions.
However, when these redundant components (CPUs, I/O subsystems) are located on more than one independent system, enabling the lockstep operation of the CPUs is frequently not readily obtainable. Thus, there remains a need to enable more than one CPU located on more than one independent system to execute in lockstep.
SUMMARY OF THE INVENTION
The present invention relates to apparatus and methods for two computing elements in a fault-tolerant server to execute instructions in lockstep. In a first aspect, the invention comprises a first computing element and a second computing element and each communicates with a communications link. The first computing element provides a first instruction to the communications link. Similarly, the second computing element provides a second instruction to the communications link. The first computing element also communicates with a first local I/O subsystem and the second computing element communicates with a second local I/O subsystem. In one embodiment, one of the local I/O subsystems compares the first instruction and the second instruction. The local I/O then indicates a fault of one of the computing elements upon the detection of a miscompare of the first instruction and the second instruction.
In one embodiment, each computing element includes a central processing unit (CPU) that communicates with a respective local mass storage device. Further, the communications link comprises a switching fabric that communicates with the respective CPU and both of the respective I/O subsystems. The switching fabric communicates with the I/O subsystem on the opposite computing element over a backplane link and a backplane.
In another aspect, the invention consists of a method for a first computing element and a second computing element to execute in lockstep in a fault-tolerant server. The method includes the step of establishing communication between a first computing element and a communications link. The method further includes the step of establishing communication between a second computing element and the communications link. The first computing element transmits a first instruction to the communications link and the second computing element transmits a second instruction to the communications link. In one embodiment, an I/O subsystem compares the first instruction and the second instruction and indicates a fault of one of the computing elements in response to a miscompare.
In one embodiment, the method further includes the step of transmitting a stop command to each computing element when the first instruction does not equal the second instruction. The method may also include detecting an error introduced by the communications link. The method can additionally include assigning a priority to each respective computing element and determining whether one of the computing elements is faulty based on the assigned priority.
BRIEF DESCRIPTION OF THE DRAWINGS
The invention is pointed out with particularity in the appended claims. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. Like reference characters in the respective drawing figures indicate corresponding parts. The advantages of the invention may be better understood by referring to the following description taken in conjunction with the accompanying drawings in which:
FIG. 1 is a block diagram of a prior art computer system;
FIG. 2 is a block diagram of an embodiment of a fault-tolerant computer system;
FIG. 3 is a more detailed block diagram of an embodiment of a fault-tolerant computer system;
FIG. 4 is a flow diagram generally illustrating an embodiment of a process for handling errors in the fault-tolerant computer system shown in FIG. 2;
FIG. 5 is a flow diagram generally illustrating an embodiment of a process to identify and isolate a faulty CPU;
FIG. 6 is a block diagram of an embodiment of a fault-tolerant server;
FIG. 7 is a block diagram of a computing element of the fault-tolerant server of FIG. 6;
FIG. 8 is a more detailed block diagram of the fault-tolerant server of FIG. 6;
FIG. 9 is a flow diagram illustrating an embodiment of a process to compare instructions generated by each computing element of the fault-tolerant server of FIG. 6; and
FIG. 10 is a block diagram of the fault-tolerant server of FIG. 6 mounted in a cabinet fashion.
DETAILED DESCRIPTION OF THE INVENTION
Referring now to FIG. 1, a typical computer system 20, such as a server, as known in the prior art includes a central processor unit (CPU) 22, a main memory unit 24 for storing programs and/or data, an input/output (I/O) subsystem 26, a display device 28, and a system communications bus 30 coupling these components to allow communication between these units. The memory 24 may include random access memory (RAM), read only memory (ROM), and one or more memory registers. The computer system 20 typically also has one or more peripheral devices, such as input devices 32. For example, the computer system 20 may include a keyboard 34 (e.g., an alphanumeric keyboard and/or other types of keyboards such as a reduced-key keyboard, or a musical keyboard) and a computer pointing device 36 for translating user movements into computer gestures (e.g., a mouse, a track ball, a track pad, a digitizing tablet, a joystick, a data glove).
The computer system 20 typically also includes one or more mass storage devices, such as a hard disk drive 38 and a floppy disk drive 40 for receiving floppy disks such as 3.5-inch disks. Other additional peripheral devices 42 also can be part of the computer system 20 including output devices (e.g., printer or plotter) and/or optical disk drives for receiving, reading, and/or writing digital data on a CD-ROM. In the disclosed embodiment, one or more computer programs shown in phantom define the operational capabilities of the computer system 20. These programs can be loaded onto the hard disk drive 38 and/or into the main memory 24 of the computer CPU 22 via the floppy disk drive 40. A user of the computer system 20 may execute these applications by using the computer-pointing device 36 to double-click on an icon related to the applications. In general, one or more of the computer system's mass storage devices, such as the hard disk drive 38 or the other additional peripheral devices 42 (e.g., a CD-ROM 42), stores the controlling software program(s) and all of the data utilized by the program(s).
The system communications bus 30 allows data to be transferred between the various components in the computer system 20. For example, the CPU 22 may retrieve program data from the main memory 24 over the system communications bus 30. Various system busses 30 are standard in computer systems 20, such as the Video Electronics Standards Association (VESA) Local Bus, the industry standard architecture (ISA) bus, the Extended Industry Standard Architecture bus (EISA), the Micro Channel Architecture bus (MCA) and a Peripheral Component Interconnect (PCI) bus. In some computer systems 20, multiple system communication busses 30 may be used to provide access to different units of the system 20. For example, a computer system 20 may use a PCI bus to connect a CPU 22 to peripheral devices 28, 34, 36, 38, 40, 42 and concurrently to connect the CPU 22 to main memory 24 using an MCA bus. Other embodiments include a system bus 30 comprised of other bus architectures, or combination of bus architectures, such as an Accelerated Graphics Port (AGP) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), a Personal Computer Memory Card Industry Association (PCMCIA) bus, a NuBus, a TURBOchannel bus, a Multibus, a STD bus, or a Versa Module Europa (VME) bus.
It is immediately apparent from FIG. 1 that such a traditional computer system 20 is highly sensitive to any single point of failure. For example, if the main memory unit 24 fails to operate for any reason, the computer system 20 as a whole will cease to function. Similarly, should the system communications bus 30 fail, the computer system 20 as a whole will fail. A redundant, fault-tolerant system achieves an extremely high level of availability by using redundant components and data paths to insure uninterrupted operation. A redundant, fault-tolerant system may be provided with any number of redundant components. Some configurations include dual-mode redundant (DMR) systems, which include duplicates of certain hardware units found in FIG. 1, for example, duplicate, redundant CPUs 22 and main memory units 24 executing substantially identical instruction streams. Other configurations include triple-mode redundant (TMR) configurations, which include three of each of certain hardware units shown in FIG. 1, for example three redundant CPUs 22 and main memory units 24 executing substantially identical instruction streams. Yet other configurations are possible with even higher-level redundancies.
In brief overview, referring now to FIG. 2, one embodiment of a TMR, fault-tolerant computer system 20 is shown that includes three CPU boards 22, 22′, 22″ (generally 22), at least two I/ O subsystems 26, 26′, 26″, 26′″ (generally 26), redundant communications busses 30, 30′, 30″, 30′″ (generally 30), one or more first peripheral busses 64 a through 64 m (generally 64), and one or more peripheral devices 42 a through 42 n (generally 42).
Each of the CPU boards 22 communicates with the first I/O subsystem 26 through the first communications bus 30. Each of the CPU boards 22 communicates with the second I/O subsystem 26′ through the second, redundant communications bus 30′. In some embodiments, the CPU board 22 is a module, or chassis, while in yet other embodiments the CPU board 22 is a single chip.
In some embodiments, the system communication busses 30 are standard system busses such as those described for the computer system 20 illustrated in FIG. 1 (ISA, VESA, EISA, MCA, PCI, and the like). Examples of peripheral devices 42 include the peripheral devices previously identified in FIG. 1, e.g., a display device (e.g., a monitor), a hard disk drive, a CD ROM drive, one or more input device(s) (e.g., a keyboard or a computer pointing device), a printer, a plotter, and a floppy disk drive 40.
In another embodiment, the fault-tolerant computer system 20 includes more than two I/O subsystems (e.g., 26″ and 26′″ shown in phantom view) to allow the fault-tolerant computer system 20 to control additional peripheral devices 42. The additional I/O subsystems 26″, 26′″ are similarly in communication with the CPU boards 22 through additional communication busses 30″ and 30′″, also shown in phantom.
In one embodiment, the CPU boards 22 are redundant CPU boards 22 executing substantially identical instruction streams. By executing substantially identical instruction streams, the CPU boards 22 are configured in a “failover” mode. That is, at any instant in time, one CPU board 22, e.g., the second CPU board 22′, remains ready to replace the first CPU board 22 upon a failure of the first CPU board 22. As a consequence of the replacement, the second CPU board 22′ experiences no loss of data, as the second CPU board 22′ operates in place of the first CPU board 22.
Similarly, the I/O subsystems 26 are redundant components configured in failover mode. That is, at any instant in time, I/O communications between the CPU boards 22 and the peripheral devices 42 are communicated through one of the I/O subsystems 26, e.g., the first I/O subsystem 26, but are not simultaneously communicated through the second I/O subsystem 26′. The second, I/O subsystem 26′ remains ready to replace the first I/O subsystem 26 in the event of a failure. Failover of the I/O subsystems 26 is controlled by fault-tolerant control elements discussed later in more detail.
In one embodiment and as shown in FIG. 2, each I/O subsystems 26 communicates with a mass storage device 45 a, 45 b (generally 45). Each mass storage device 45 can be grouped with another mass storage device 45 to form a pair of devices 47, such as a pair of disks. To ensure integrity of the data that the fault-tolerant computer system 20 stores in one of the mass storage devices (e.g., 45 a), the fault-tolerant computer system 20 simultaneously writes the data to another mass storage device (e.g., 45 b). This process is known to those skilled in the art as “disk mirroring”. The mirroring of the data enables the fault-tolerant computer system 20 to access the same data from the second mass storage device 45 b following a failure of the first mass storage device 45 a, and vice-versa.
For example, a mass storage device 45 may be, without limitation, a hard disk, CD-ROM, magnetic disk, or magneto-optical drive. In one embodiment, the pair of mass storage devices 45 are part of a redundant array of independent disks (RAID arrays) used as failure-tolerant persistent mass storage. The fault-tolerant computer system 20 processes each write transaction to the mass storage devices 45 in parallel, writing it to each device 45 in the array. If the computer system 20 fails, then the mass storage device 45 with the most accurate set of contents available can be used as a master, copying all its contents to the other devices 45 in the array (RAID level 1). Another solution not only stores one copy of the transaction information across multiple mass storage devices 45, but also stores parity information concerning the transaction data (RAID level 5). In another embodiment, the mass storage devices 45 are Fibre Channel disks. In yet another embodiment, the mass storage devices 45 are a group of disks.
Frequently, only one I/O subsystem 26 has access to a particular mass storage device 45. Consequently, upon a failure of a particular mass storage device 45, the I/O subsystem 26 associated with that mass storage device 45 can no longer access the data previously stored on the failed mass storage device 45. Similarly, the I/O subsystem 26 associated with the failed mass storage device 45 a cannot access the mirrored mass storage device 45 b because the mirrored mass storage device 45 b is associated with an independent I/O subsystem 26′. Thus, upon a failure of a mass storage device 45 a, the I/O subsystem 26 associated with the failed mass storage device 45 is replaced by a redundant I/O subsystem 26′ to prevent loss of data (i.e., failover). For example, a failure of the first mass storage device 45 a causes the first CPU board 22 to connect to the redundant, second I/O subsystem 26′ through bus 30′.
In more detail and in one embodiment, each CPU board 22 contains at least one processor 44 and the main memory 24. In some embodiments, each CPU board 22 contains multiple processors 44, 44′, 44″, and 44′″ (generally 44). In multi-processor embodiments, each of the multiple processors 44 of a CPU board 22 (e.g., processor 44 and 44′ of CPU board 22) may process different instruction streams. Respective processors 44 on different CPU boards 22 (e.g., processor 44 on CPU board 22 and processor 44 on CPU board 22′) execute substantially identical instruction streams. In one embodiment, the processors 44 are selected from the “x86” family of processors manufactured by Intel Corporation of Santa Clara, Calif. The x86 family of processors includes the 80286 processor, the 80386 processor, the 80486 processor, and the Pentium®, Pentium® II, Pentium® III, Pentium® III Xeon™, Pentium IV processors, and the 64-bit Itanium™ family of processors. In another embodiment, the processors 44 are selected from the “680x0” family of processors manufactured by Motorola Corporation of Schaumburg, Ill. The 680x0 family of processors includes the 68000, 68020, 68030, and 68040 processors. Other processor families include the Power PC line of processors manufactured by Motorola Corporation, the Alpha line of processors manufactured by Compaq Corporation of Houston, Tex., and the Crusoe line of processors manufactured by Transmeta Corporation of Santa Clara, Calif. In yet another embodiment, the processor 44 is an Athlon processor, manufactured by Advanced Micro Devices (AMD) of Sunnyvale, Calif.
In one embodiment, the redundant CPU boards 22 execute in “lockstep,” that is, each CPU board 22 executes substantially identical copies of the operating system and application programs, substantially simultaneously, in cycle-by-cycle synchronism. In lockstep operation, the replicated CPU boards 22 store identical data in the replicated main memory 24 at all times. In some embodiments of a lockstep fault-tolerant computer system 20, a single reference clock source 48 (shown in phantom) is provided in communication with each of the CPU boards 22, and each of the CPU boards 22 synchronizes to the common clock source 48.
In other embodiments generally well known to those skilled in the art, the redundant CPU boards 22 execute identical operating systems and application programs and execute substantially equivalent instruction streams in a loosely synchronized, or “loose-stepped” manner. In a loose-step fault-tolerant computer system 20, each of the CPU boards 22 can include its own clock source 49, 49′, 49″ (generally 49) shown in phantom in FIG. 2, running asynchronously with respect to the clock sources 49 of the other CPU boards 22. Loose-step synchronization does not require the common clock source 48 used for lock-step embodiments.
In some embodiments of a loose-step fault-tolerant computer system 20, the CPU boards 22 maintain synchronization of the fault-tolerant computer system 20 by counting the instructions processed and initiating a synchronizing procedure after counting some quantum of the instructions. In other embodiments, the CPU boards 22 maintain synchronization of the fault-tolerant computer system 20 by monitoring events, such as memory references. In these embodiments any of the CPU boards 22 performing a monitored event before the remainder of the loose-stepped CPU boards 22 is stalled. Once the remainder of the CPU boards 22 perform the monitored event, the stalled CPU boards 22 are allowed to continue processing.
In more detail and referring now to FIG. 3, one embodiment of a DMR fault-tolerant computer system 20 is shown. The fault-tolerant computer system 20 includes at least a first redundant CPU 22 and a second redundant CPU 22′, a first and second communications channel 30 and 30′, and at least one I/O subsystem 26. The redundant CPUs 22, 22′ are in communication with the I/O subsystem 26 through the respective communications channel 30, 30′. The I/O subsystem 26 communicates with one or more peripheral devices 42 z through the I/O bus 58, and optionally through the peripheral busses 64 as shown in FIG. 2. I/O instructions can be generated at each of the CPUs 22, or the peripheral devices 42, and can be directed at any of the other CPUs 22 or peripheral devices 42. The I/O instructions include memory read or writes, configuration read or writes, mass storage device read or writes, or other special instructions.
The I/O subsystem 26 includes an I/O bus 58 and I/O bus interface 68. The I/O bus 58 interconnects one or more of the peripheral devices 42 to the I/O bus interface 68. In some embodiments, the I/O subsystem 26 includes a peripheral controller 72 shown in phantom.
In one embodiment as described in detail below, the I/O subsystem 26 includes I/O fault-tolerant logic 52 and optionally a voter delay buffer 98, shown in phantom. In one embodiment, the I/O fault tolerant logic 52 communicates with the I/O bus interface 68 and the voter delay buffer 98, if present. In another embodiment, the I/O fault-tolerant logic 52 is implemented within the I/O bus interface 68, such as on a single application specific integrated circuit (ASIC). The I/O fault tolerant logic 52 detects faults within the fault-tolerant computer system 20.
In one embodiment, the I/O fault-tolerant logic 52 includes a comparator 92 that performs comparisons of the I/O instruction streams. As previously described, for lockstep fault-tolerant embodiments, the CPU instructions are substantially identical where the same instructions occur for each of the redundant CPUs 22 on the same clock cycle. In some lockstep embodiments, voting of the I/O instructions from the CPU 22 is conducted during each clock cycle. In some embodiments, the I/O fault tolerant logic 52 compares the equivalent I/O instruction streams from each of the redundant CPUs 22 to identify a fault upon the detection of a miscompare.
In some embodiments the I/O fault-tolerant logic 52 also includes a buffer 94, shown in phantom. The buffer 94 can be used for holding I/O instructions from one or more of the redundant CPUs 22 in a loose-step fault-tolerant system 20. As previously described, the time of voting, or comparison, in a loose-step embodiment is determined by some event.
For either lock-step or loose-step embodiments, the I/O fault-tolerant logic 52 identifies errors when at least one of the inputs to the comparator 92 is different from the other equivalent, redundant inputs. In one embodiment, the comparator 92 substantially simultaneously performs a bit-by-bit comparison of the voted I/O instruction. When the comparator 92 determines it received input data from each of the redundant CPUs 22 that are identical at the time of voting, no errors are detected. In one embodiment, the redundant I/O instructions are stored in respective registers and voting occurs such that the equivalent bits of each register are compared by a plurality of comparators, one for each bit of the instruction.
The I/O fault-tolerant logic 52 provides the fault-tolerant computer system 20 with a fault-tolerant capability. In some embodiments, the I/O fault-tolerant logic 52 is implemented as a separate set of logic on the I/O subsystem 26. For example, the I/O fault-tolerant logic 52 may be provided as an ASIC, a field programmable gate array (FPGA), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a programmable logic device (PLD), or a read-only memory device (ROM). The I/O fault-tolerant logic 52 compares the results of each operation performed by the separate CPU boards 22 to the results of the same operation performed on one of the other CPU boards 22′, 22″. If the I/O fault logic 52 determines a discrepancy then a failure has occurred.
As indicated above, the DMR fault-tolerant computer system 20 may include a priority module 86 (shown in phantom) and the voter delay buffer 98 (shown in phantom). In one embodiment, the I/O fault-tolerant logic 52 includes a priority register 46, also shown in phantom. In one embodiment, each CPU board 22 contains a CPU bus interface 66 and CPU fault-tolerant logic 50. The CPU fault-tolerant logic 50 communicates with the CPU bus interface 66. The priority register 46 of the I/O fault-tolerant logic 52 communicates with the priority module 86 through a CPU PRIORITY signal 88. In some embodiments, the CPU fault-tolerant logic 50 on each redundant CPU 22 communicates with the I/O fault-tolerant logic 52 through a respective command line 96, 96′.
In one embodiment of a fault-tolerant computer system 20, the priority module 86 assigns a priority to each of the redundant CPUs 22. The assigned priority is stored in the priority register 46. The I/O fault-tolerant logic 52 or the priority module 86 may update or change the assigned priority during operation. The priority can be established by a number of factors, such as historical performance of the CPU 22 or prior/current diagnostic results from the maintenance and diagnostics module 84.
As described above, in one embodiment of a lockstep system 20, the I/O fault-tolerant logic comparator 92 performs a bit-by-bit cycle compare procedure on the data output streams passing into the I/O subsystem 26 on the I/O busses 30 and 30′. In another embodiment of a loose-step system 20, the I/O subsystem 26 includes buffer 94, shown in phantom, before the comparator 92, to hold the transactions of the data output streams that will be compared from each of the redundant CPU boards 22. When each of the transactions to be compared from each of the respective redundant CPU boards 22 arrives at the buffer 94, the transactions are provided to the comparator 92. In either embodiment, lockstep or loose-stepped, the comparator 92 may be, for example, an XOR gate or any other known component for comparing two bit values. If the cycle-compare procedure detects a difference between the two data output streams, this may be an indication that one of the CPU boards 22 and 22′ is malfunctioning. Accordingly, the I/O subsystem 26 responds by issuing a “STOP” command to both the first CPU 22 and the second CPU 22′ over a first command line 96 and a second command line 96′ respectively.
In one embodiment, the I/O fault tolerant logic 52 generates a command to take the suspected faulty CPU 22 off line. The command is communicated to the determined faulty CPU 22 over the respective command line 96. When voting results in a miscompare within a DMR system 20, additional information is required to determine which CPU 22 is faulty and should be taken off line.
In one embodiment, the I/O fault-tolerant logic 52 determines which of the two redundant CPUs 22 is faulty by requesting that the maintenance and diagnostics module 84 of each redundant CPU 22 perform diagnostics to verify the integrity of the respective CPU 22. In some embodiments, when a fault is detected, the DMR fault-tolerant system 20 restricts the propagation of faulty data by issuing a stop command to the CPUs 22 until diagnostics are completed, and by directing commands issued by the CPUs from the time of the issued faulty I/O instruction to the time that the stop command is executed by the CPUs 22. Once a fault is detected, the I/O instructions in the “pipeline” from each of the two redundant CPUs 22 are directed to a respective first and second first-in-first-out buffer (FIFO) buffer 102, 102′ within the voter delay buffer 98. If one of the two CPUs, for example CPU 22, diagnoses a failure, that CPU 22 is taken off line for repair and replacement and any commands in the related FIFO 102 are discarded, whereas the commands associated with the “good” CPU 22′ are reissued from the respective FIFO 102′.
In another embodiment, the I/O fault-tolerant logic 52 can determine which of the two redundant CPUs 22 is suspected faulty by defining the CPU 22 with the lower priority value stored in the priority register 46 as being the faulty CPU. In yet another embodiment, combinations of results from the maintenance and diagnostics modules 84 and the priority values can be used by the fault- tolerant logic 50, 52 to determine which CPU 22 is suspected faulty.
In operation, the fault-tolerant computer system 20 can best be described with reference to the flow diagram of FIG. 4. The data output streams on the I/O busses 30 and 30′ are bit-by-bit compared by the comparator 92 (step 110) to provide a comparative reading from which it can be determined if there are differences between the monitored data output streams. If there are no such differences detected, the comparator 92 continues to monitor the data output streams. If the comparator 92 detects differences, the I/O fault-tolerant logic 52 issues a STOP command (step 112). In some embodiments, the issuance of a STOP command prompts the initiation of a CPU diagnostic procedure on each of the redundant CPUs 22 to check the status of each CPU 22. Subsequently, the data output streams on the I/O busses 30 and 30′ are diverted to the voter delay buffer 98 (step 114).
The first CPU board 22 executes its ongoing diagnostic procedure (step 116). If the diagnosis indicates that the first CPU board 22 is malfunctioning, the first CPU board 22 is isolated (step 118) and operation of the computer system 20 continues with the second CPU board 22′. The data stored in the second FIFO buffer 102′ is output over the system I/O bus 30 (step 120) and thereafter subsequently processed data from the second CPU board 22′ is output over the system I/O bus 30.
Contemporaneously with the ongoing diagnosis procedure in the first CPU board 22 (step 116), the second CPU board 22 also executes its diagnostic procedure (step 122). If, on the other hand, the resulting diagnosis indicates that the second CPU board 22′ is malfunctioning, the second CPU board 22′ is isolated (step 124) and operation of the computer system 20 continues with the first CPU board 22. The data stored in the first FIFO buffer 102 is output over the system I/O bus 30 (step 126) and subsequent processed data from the first CPU board 22 is output over the system I/O bus 30.
If the diagnostic procedures fail to detect problems with either the first CPU board 22 or the second CPU board 22′, the relative CPU priorities determined by the priority module 86 and maintained within the priority register 46 are used as the determinative factor. The relative CPU priorities are read from each of the priority registers 46, 46′ to establish which of the first CPU board 22 or the second CPU board 22′ has the higher priority (step 128). As discussed above, the relative priorities of the CPU boards 22 have been determined by one or more criteria, such as their operational histories or the comparative cumulative record of their internal error corrections. If the second CPU board 22′ has been assigned the higher priority, for example, the computer system 20 selects the first CPU board 22 as the malfunctioning CPU board 22 and continues to operate with only the second CPU board 22′ (step 130). In this event, the data stored in the second FIFO buffer 102′ is output (step 132). On the other hand, if the first CPU board 22 has been assigned the higher priority, the computer system 20 selects the second CPU board 22′ as the malfunctioning CPU board 22 and the operation of the computer system 20 continues with the first CPU board 22 (step 130). In this case, the data stored in the first FIFO buffer 102 is output (step 132).
One embodiment of a TMR fault-tolerant computer system 20 is similar to that shown in FIG. 2 with the internal detail of the CPU 22 and the I/O subsystem 26 shown in FIG. 3. In a TMR system, the I/O fault-tolerant logic 52 votes, or compares each of the input values of each I/O transaction generated by the three redundant CPUs 22 and received at the I/O bus interface 68 to determine if the three input instructions are not the same. In one embodiment, the voting constitutes a bit-by-bit comparison of each of the bits of the three redundant I/O instructions, performed within the comparator 92. Here, the comparison determines if all the inputs are the same or, conversely, if one of the inputs is different from the other and, from that, identifies which one of the three differs from the others. In one embodiment, the comparator 92 is implemented in hardware. In another embodiment, the comparator 92 includes combinatorial logic. In another embodiment, the comparator 92 is implemented in software. In one embodiment, when the comparator 92 determines a miscompare among the three input I/O instructions, the comparator 92 also identifies which of the three CPUs 22, referred to as the minority CPU 22, is not in agreement with the other two CPUs 22, referred to as the majority CPUs 22. Using the I/O fault-tolerant logic 52, the majority value is determined to be the correct valid instruction and the minority is determined to be faulty. The I/O fault tolerant logic 52 commands that the suspected faulty CPU 22 be taken off line.
In some embodiments of a TMR fault-tolerant system, there is no priority module 86 and no voter delay buffer 98 since identification of the faulty CPU 22 is determined to be the minority CPU in a triple-valued compare. In some embodiments of a TMR fault-tolerant computer system 20, the priority module and priority register 46 can also be used to make further determinations of identifying a failed CPU 22 in a miscompare. In other embodiments of a TMR fault-tolerant computer system 20 the maintenance and diagnostics module 84 and voter delay buffer 98 can also be used to make further determinations of identifying a failed CPU 22 in a miscompare. In yet other embodiments, the TMR fault-tolerant computer system 20 will function as the previously identified DMR fault-tolerant computer system 20 when one of the three CPUs 22 has been taken off-line, leaving two remaining on-line CPUs 22.
In operation, referring to FIG. 5, in one embodiment of a TMR fault-tolerant computer system 20, the I/O fault-tolerant logic 52 compares I/O transactions from each of the CPU boards 22, 22′, 22″ (step 140). In the TMR system 20, the I/O fault-tolerant logic comparator 92 performs a voting function by comparing each of the instruction streams by a bit-by-bit comparison and identifying a minority CPU board 22 that produces an I/O instruction stream that does not match the I/O instruction stream from the majority (step 142)—the other two CPU boards 22′, 22″ in a TMR system. The minority CPU board 22 is then taken off-line to avoid the propagation of errors into the system and to allow for diagnostics, repair and replacement of the identified minority CPU board 22 if necessary (step 144).
Discussing the operation in further detail, when the I/O fault-tolerant logic 52 issues a STOP command in response to the detection of a miscompare, the I/O subsystem 26 stops transmitting output data on the I/O bus 58 and routes the data output streams on the busses 30 and 30′ to a voter delay buffer 98 via a delay buffer line 100. Specifically, the data received from the first CPU board 22 is sent to a first FIFO buffer 102, and the data received from the second CPU board 22′ is sent to a second FIFO buffer 102′. This action prevents the peripheral devices from being sent data which may have been corrupted by the malfunctioning CPU board 22, and also saves data which otherwise may have been lost or discarded while the malfunctioning CPU board 22 was being identified. In one embodiment, the fault- tolerant logic 50, 52 identifies a faulty CPU 22 and notifies the voter delay buffer which CPU 22 is faulty. The voter delay buffer 98 then releases the buffered commands from the respective FIFO 102, 102′ for the valid CPU 22 and normal processing continues.
In a preferred embodiment, the maintenance and diagnostic subsystems 84 and 84′ continually run their respective diagnostic procedures. It should be understood that, even after the STOP command has been issued to the CPU boards 22 and 22′, the I/O subsystem 26 continues to forward input data streams sent by the peripheral devices to the CPU boards 22 and 22′. The CPU boards 22 and 22′ continue to process the data while running the diagnostic procedures, in accordance with normal operational procedures. Thus, from the point of view of the peripheral devices, the fault-tolerant computer system 20 functions normally.
Referring to FIG. 6, another embodiment of the fault-tolerant computer system 20, also referred to below as a fault-tolerant server, is illustrated. The fault-tolerant server 20 includes a first computing element 150 and a second computing element 150′ (generally 150). In one embodiment, each computing element 150 is an independent motherboard that includes the CPU board 22 of FIG. 2 and FIG. 3 and a “local” I/ O subsystem 152, 152′ (generally 152). As used above and below, a “local” I/O subsystem 152 is an I/O subsystem that is associated with a particular CPU board 22.
Each computing element 150 also includes a “local” mass storage device 154, 154′ (generally 154). As used above and below, a “local” mass storage device 154 is a mass storage device that is associated with a local I/O subsystem, and thus associated with a particular CPU board 22 and computing element 150. In one embodiment, each respective local mass storage device 154 is located on the same board as the respective computing element 150. In another embodiment, the local mass storage device 154 is an external component of the computing element 150.
The local mass storage device 154 also has a disk controller (not shown) associated with the mass storage device 154 and with the particular computing element 150. In one embodiment, the disk controller is implemented in the local I/O subsystem 152. In another embodiment, the disk controller is implemented as an independent component in communication with the I/O subsystem 152 and the local mass storage device 154.
If a fault-tolerant server includes two independent computing elements that include devices local to the computing element (i.e., a local I/O subsystem and a local mass storage device), each CPU board typically does not have access to each local I/O subsystem and each mass storage device, unlike the fault-tolerant computer system 20 of FIG. 2. Thus, a typical fault-tolerant server having two independent computing elements cannot use a first CPU board located on the first computing element to access a mass storage device located on the second computing element.
To enable one of the CPU boards 22 to access a mass storage device 154 that is local to the other CPU board 22, the fault-tolerant server 20 additionally includes a backplane 158. In general, a backplane is a circuit board or framework that supports other circuit boards, devices, and the interconnections among devices, and provides power and data signals to supported devices. A computer system may also have expansion cards plugged into sockets of a backplane to increase the capabilities of the computer system. Backplanes are often described by those skilled in the art as being either “active” or “passive”. Active backplanes contain, in addition to the sockets, logical circuitry that performs computing functions. In contrast, passive backplanes contain almost no computing circuitry.
In particular, the backplane 158 enables one computing element 150 and, more particularly, one CPU board 22 to access either local I/ O subsystem 152, 152′ and thus either mass storage device 154, 154′ even though the mass storage devices 154, 154′ are local to one computing element 150, 150′. In one embodiment, the computing elements 150 generally plug into a socket in the backplane 158, thereby facilitating the swapping of one computing element 150 with another computing element 150′ (e.g., replacing a computing element 150 upon a failure).
In more detail and also referring to FIG. 7, each CPU board 22, 22′ communicates with its respective local I/ O subsystem 152, 152′ over a communications link. To communicate with the respective local I/O subsystem 152, the communications link includes a first communications channel 162, 162′ (162′ not shown) (generally 162), a switching fabric 166, 166′ (166′ not shown) (generally 166), and a second communications channel 164, 164′ (164′ not shown) (generally 164). To communicate with the local I/O subsystem 152 located on the opposite computing element 150, the communications link also includes a backplane 158 and a respective backplane link 161, 161′ (generally 161).
In one embodiment, the communications link or some or all of the components that make up the communications link (e.g., the communications channel 162, 164, each backplane link 161) are standard system busses such as those described for the computer system 20 illustrated in FIG. 1 (ISA, VESA, EISA, MCA, PCI, and the like). In other embodiments, the communications link or some or all of the components that make up the communications link may be a “twisted-pair” cable, coaxial cable, fiber optic cable, or wireless links, such as radio links or free-optics links.
In one embodiment, the first CPU board 22 communicates with the first switching fabric 166 over a first communications line 163 a. The first CPU board 22 communicates with the second computing element 150′ (i.e., the second switching fabric 166′) over a second communications line 163 b, the first backplane link 161, the backplane 158, and the second backplane link 161′. The first switching fabric 166 communicates with the first local I/O subsystem 152 over a third communications line 165 a. The first switching fabric 166 communicates transmission received from the second computing element 150′ to the first local I/O subsystem 152 over the fourth communications line 165 b.
Similarly, the second CPU board 22′ (not shown) communicates with the second switching fabric 166′ (not shown) over a first communications line 163 a′ (not shown). The second CPU board 22′ communicates with the first computing element 150 (i.e., the first switching fabric 166) over a second communications line 163 b′ (not shown), the second backplane link 161′ (not shown), the backplane 158, and the first backplane link 161. The second switching fabric 166′ communicates with the second local I/O subsystem 152′ (not shown) over a third communications line 165 a′. The second switching fabric 166 communicates transmission received from the first computing element 150 to the second local I/O subsystem 152′ over the fourth communications line 165 b′ (not shown).
It should be noted that the first communications lines 163 a, 163 a′ (generally 163 a), the second communications lines 163 b, 163 b′ (generally 163 b), the third communications lines 165 a, 165 a′ (generally 165 a), and/or the fourth communications lines 165 b, 165 b′ (generally 165 b) may each be, in one embodiment, a standard system bus such as those described for the computer system 20 illustrated in FIG. 1 (ISA, VESA, EISA, MCA, PCI, and the like). In other embodiments, the communications lines 163 a, 163 b, 165 a, 165 b may be a “twisted-pair” cable, coaxial cable, fiber optic cable, or wireless links, such as radio links or free-optics links.
In some embodiments, the backplane 158 includes an active logic module 176. In certain of these embodiments, the logic module 176 provides the computing elements 150 with system information required on power-up, such as the system identification number, or system ID. For example, the logic module 176 provides the first CPU board 22 with a system ID of 0 (i.e., CPU 0) and provides the second CPU board 22′ with a system ID of 1 (i.e., CPU 1). Additionally, the logic module 176 may provide a common clock source to the CPU boards 22 on each computing element 150.
The backplane 158 may include a connector interface (not shown) that facilitates the physical and electrical connection of the server 20 to external resources. In some embodiments, the connector interface is an external interface that provides a connection to an external network via, for example, an RJ-45 connector or coaxial cable connection. Similarly, the external connector interface may connect to an external modem of the fault-tolerant server 20 and thus provides a network connection to the internet. The connector interface can also connect to one or more peripheral devices, such as a keyboard or mouse via, for example, a DB-9 connector, a DB-25 connector, or a USB port. In certain embodiments, the external connector interface facilitates “blind mating” between the backplane 158 and the computing elements 150. That is, the computing elements 150 need only make connection with the backplane 158 in order to have access to all network and external physical resources. In other embodiments, the connector interface is internal to the computing element 150.
Each computing element 150 also includes a mass storage device controller 178 (also referred to throughout as a disk controller) to interface with the local mass storage device 154. In some embodiments, the local mass storage device 154 is a disk drive capable of communicating using the Fibre Channel protocol (i.e., “Fibre Channel disks”). In another embodiment, each computing element 150 includes multiple local mass storage devices (e.g., a first local mass storage device 154 a and a second local mass storage device 154 b) that communicate with the mass storage device controller 178.
For embodiments in which Fibre Channel disks 154 are provided, the disks 154 may be connected in a loop topology with the I/O subsystem 152. This arrangement is generally referred to as a Fibre Channel Arbitrated Loop (FC-AL). As is well known for FC-ALs, an I/O subsystem may communicate with multiple FC-ALs. For example, the first local I/O subsystem 152 may directly communicate with the disk drives 154 present on the first FC-AL and may also directly communicate with the remote disk drives 154 present on the remote FC-AL. This communication from one I/O subsystem 152 to either or both FC-ALs may be useful when one of the local I/O subsystems 152 fails because the other local I/O subsystem 152 can communicate with the FC-AL 154 associated with the failed local I/O subsystem 152 to continue normal operation.
In certain embodiments, the communication between the I/O subsystem 152 and the FC-AL is through the backplane 158. In other embodiments, the I/O subsystem 152 communicates directly with the remote FC-AL using a very high density metric (VHDM) connector (not shown). In a further embodiment, a subset of the connector pins in the VHDM connector provides electrical communication between the I/O subsystem 152 and the remote FC-AL. Additionally, another subset of the connector pins of the VHDM connector routes the switching fabric 166 located on one computing element 150 to the switching fabric 166 located on the other computing element 150.
Each computing element 150 also includes an I/O synchronization bus, or sync bus, 180. The sync bus 180 communicates with the backplane 158 and enables synchronization of all of the local I/O subsystems 152. In one embodiment, the sync bus 180 synchronizes state information about each CPU board 22 between the local I/O subsystems 152. The state information of a CPU board 22 described above includes, for example, if the CPU board 22 is an “on-line” CPU board 22 (i.e., operating correctly), a “broken” CPU board 22 (i.e., operating incorrectly), or an “offline” CPU board 22 (i.e., executing diagnostics or ready to be brought into service).
It should be noted that, although the description above with respect to FIG. 7 may describe the first computing element 150, the second computing element 150′ includes the same components as the first computing element 150 and the description applies to the second computing element 150 as well with minor modifications.
Referring to FIG. 8, each local I/O subsystem 152 (i.e., the I/O fault tolerant logic 52 described above) detects faults within the fault-tolerant server 20 by comparing each of the instruction streams that each CPU board 22 produces. The switching fabric 166 used in conjunction with the backplane 158 enables synchronization of each local I/O subsystem 152 in the reception of the I/O instruction streams provided by each CPU board 22. In one embodiment, the switching fabric 166 uses the system ID described above to determine which communications line (e.g., first communications line 163 a, second communications line 163 b) to route to the backplane link 161 (i.e., to the backplane 158) and which communications line (e.g., first communications line 163 a, second communications line 163 b) to route to the local I/O subsystem 152 that is located on the same computing element 150 as the switching fabric 166.
In particular and as shown in FIG. 8, the first switching fabric 166 provides a communications path between the first CPU board 22 and the first local I/O subsystem 152 using the first communications line 163 a and the third communications line 165 a of the first communications channel 162. The first switching fabric 166 also provides a communications path connecting the first communications line 163 a to a first delay module 184. Data, e.g., I/O instructions, are delayed by the delay module 184 prior to transmission of the data to the first local I/O subsystem 152.
The first switching fabric 166 also provides a communications path between the first CPU board 22 and the second local I/O subsystem 152′ on the second computing element 150′ using the second communications line 163 b of the first communications channel 162 and the first backplane link 161. The first switching fabric 166 routes all I/O instructions transmitted on the second communications line 163 b to the second local I/O subsystem 152′ through the backplane 158.
Similarly, the second switching fabric 166′ provides a communications path between the second CPU board 22′ and the second local I/O subsystem 152′ using the second communications line 163 b′ of the first communications channel 162′. The second switching fabric 166′ also provides a communications path from the second communications line 163 b′ to a second delay module 184′ where data is held prior to transmission to the second local I/O subsystem 152′. The first and second delay modules 184, 184′ (generally 184) are described in greater detail below.
The second switching fabric 166′ provides a communications path between the second CPU board 22′ and the first local I/O subsystem 152 using the first communications line 163 a′ of the first communications channel 162′. The second switching fabric 166′ routes all I/O instructions transmitted on the first communications line 163 a′ to the first local I/O subsystem 152 through the backplane 158.
To insure that the first CPU board 22 and the second CPU board 22′ generate identical instruction streams and therefore execute in lockstep operation and also referring to FIG. 9, the local I/O subsystem 152 that receives the I/O instructions from the CPU boards 22 compares each I/O instruction. To enable such a comparison, the fault-tolerant server 20 establishes (step 204) communication between the first computing element 150 and the communications link described above. Similarly, the fault-tolerant server 20 establishes (step 208) communication between the second computing element 150′ and the communications link. In particular, the computing element 150 establishes communications between the CPU board 22 and the first and second local I/ O subsystems 152, 152′ using the communication paths that the switching fabric 166 and the backplane 158 provides to each CPU board 22, as described above.
In one embodiment, the first CPU board 22 establishes communications between the local I/O subsystems 152 by transmitting a test message to the first local I/O subsystem 152 and the second local I/O subsystem 152′ over the first switching fabric 166, the backplane link 161, the backplane 158, the backplane link 161′, and the second switching fabric 166′. Once the first local I/O subsystem 152 and the second local I/O subsystem 152′ each receives the particular test message, the first local I/O subsystem 152 and the second local I/O subsystem 152′ each transmit a response message to the first CPU board 22. In one embodiment, each response message identifies the local I/O subsystem 152 that sends the response message to the CPU board 22. In some embodiments, the response message includes the system ID for the computing element 150 which transmitted the response message.
Once a communications link is established to each local I/O subsystem 152, in one embodiment the first CPU board 22 then generates (step 212) a first I/O instruction, such as to store a datum in the first local mass storage device 154. In another embodiment, the first CPU board 22 receives the first I/O instruction from a peripheral device 42 (shown in FIG. 3), such as a modem. The first CPU board 22 transmits the I/O instruction (e.g., “store datum”) to the first switching fabric 166 over the first communications line 163 a.
As the two computing elements 150 are executing in lockstep, the second computing element 150′ simultaneously generates (step 216) (or receives from a peripheral device 42) a second I/O instruction that is identical to the first instruction (e.g., to store a datum in the first local mass storage device 154). The second CPU board 22′ transmits the I/O instruction to the second switching fabric 166′ over the first communications line 163 a′. The first switching fabric 166 and the second switching fabric 166′ then transmit (step 220) the first I/O instruction and the second I/O instruction, respectively, to the first local I/O subsystem 152. More specifically, the second switching fabric 166′ transmits in step 220 the second I/O instruction to the first local I/O subsystem 152 via the first communications line 163 a′ and the backplane 158.
Transmission of I/O instructions to the opposite computing element 150 over the backplane 158 results in an intrinsic delay in the reception of the instruction by the receiving I/O subsystem. As described above, the local I/O subsystems 152 have to concurrently compare the instructions that each CPU board 22 generates to ensure that each CPU board 22 is operating in lockstep. Thus, to compensate for the inherent delay in the transmission to the other computing element 150, each switching fabric 166 delays the transmission of the I/O instruction to the local I/O subsystem 152 (located on the same computing element 150 as the switching fabric 166) with the delay module 184 noted above. In one embodiment, the delay module 184 is tuned to the backplane 158 so that the delay module 184 provides a delay that is equivalent to the amount of time an instruction takes to reach the local I/O subsystem 152 on the opposite computing element 150.
Once the first local I/O subsystem 152 receives the first and second I/O instructions, the fault-tolerant logic module 52 (not shown) of the first local I/O subsystem 152 uses the comparator 92 (shown in FIG. 3) described above to compare (step 224) the first and second I/O instructions (e.g., bit by bit comparison). If the I/O fault-tolerant logic module 52 on the first computing element 150 determines (step 228) that differences exist between the first instruction and the second instruction, the first local I/O subsystem 152 issues (step 232) a “STOP” command to both the first CPU board 22 and the second CPU board 22′. In particular, the first local I/O subsystem 152 transmits the “STOP” command to the first switching fabric 166 over the third communications line 165 a and then to the first CPU board 22 over the first communications line 163 a. To transmit the “STOP” command to the second CPU board 22′, the first local I/O subsystem 152 transmits the “STOP” command to the second switching fabric 166′ over the fourth communications line 165 b, the first backplane link 161, the backplane 158, and the fourth communications line 165 b′. If the first local I/O subsystem 152 does not detect differences in step 224, the first local I/O subsystem 152 executes (step 236) the I/O instruction, such as by storing the datum in the first local mass storage device 154.
Additionally, in one embodiment the fault-tolerant server 20 detects errors in communications that the communications channel 162, the backplane 158, the local mass storage devices 154, the local I/O subsystems 152, and/or the second communications channel 164 introduce. In one embodiment, each CPU board 22 introduces a parity bit to the I/O instructions. In a further embodiment, each local I/O subsystem 152 and/or each local mass storage devices 154 also introduces a parity bit to any communication to either computing element 150. The communications channel 162, 164 performs parity checking on any incoming instruction stream (e.g., from either CPU board 22, from either switching fabric 166, from the backplane 158).
More specifically and in one embodiment, the first switching fabric 166 performs parity checking on the first I/O instruction when the switching fabric 166 receives the instruction from the first communications line 163 a. Similarly and in further embodiments, the first switching fabric 166 performs parity checking on the second I/O instruction upon reception of the second I/O instruction from the backplane 158. If the first switching fabric 166 detects a parity error in the second I/O instruction, the switching fabric 166 alerts the CPU board 22 that the backplane 158 may be faulty (assuming no other error discovered in prior parity checks of the communication of the second I/O instruction before reaching the first switching fabric 166). Thus, in one embodiment the error detection enables the fault-tolerant server 20 to isolate faults of a CPU board 22, the backplane 158, or the switching fabric 166. Besides parity checking (e.g., even parity, odd parity), the fault-tolerant server 20 may support other protocols for ensuring transmission accuracy, such as, without limitation, Microcom Networking Protocol (MNP), V.42, Hamming coding, and the like.
In another embodiment, the first CPU board 22 provides a first I/O instruction to read a datum from the first local mass storage device 154. As the second CPU board 22′ is executing in lockstep with the first CPU board 22, the second CPU board 22′ provides a second I/O instruction to read the datum from the first local mass storage device 154. In this embodiment, the first I/O subsystem 152 accesses the datum from the first local mass storage device 154 and transmits the datum to each CPU board 22. More specifically and in one embodiment, the first local I/O subsystem 152 transmits the datum to the second CPU board 22′ over the backplane 158.
In one embodiment, mirroring software updates the contents of one of the local mass storage devices 154 (e.g., the second local mass storage device 154′) so that the contents are identical to the contents of the other local mass storage device 154 (e.g., the first local mass storage device 154). In other words, the second local mass storage device 154′ “mirrors” the first local mass storage device 154.
Although the description above and below may focus on the first computing element 150 and the components therein (e.g., the first CPU board 22, the first local I/O board 152), it should be clear that the description also applies to additional computing elements 150 (e.g., the second computing element 150′) and the respective components therein (e.g., the second CPU board 22′, the second local I/O subsystem 152′).
Referring to FIG. 10, the fault-tolerant server 20 may be organized in a cabinet fashion, with each computing element 150 being a 1U rack-mounted motherboard. In general, a rack-mounted motherboard is a motherboard that is built for installation in a metal frame or cabinet of standard width (typically 19 inches or 23 inches) and mounting arrangements. In general, a “U” as used above is a standard unit of measure for designating the height in computer enclosures and rack cabinets. In one embodiment, a U equals 1.75 inches. For example, a 4U chassis is 7 inches high and a 40U rack cabinet is 70 inches high.
As shown in FIG. 10, the fault-tolerant server 20 includes two 1U rack-mounted motherboards 150; thus the fault-tolerant server 20 is a 2U server. Although described as a 2U fault-tolerant server 20, it should be noted that the fault-tolerant server 20 can use any number of 1U rack-mounted motherboards 150 (e.g., the third computing element 150″ shown in shadow). In other embodiments, the rack-mounted motherboards 150 may be any reasonable height (e.g., 1U, 2U, 3U, 4U).
Having shown the preferred embodiments, one skilled in the art will realize that many variations are possible within the scope and spirit of the claimed invention. It is thus the intention to limit the invention only by the scope of the claims.

Claims (22)

1. A fault-tolerant server comprising:
(a) a communications link comprising a switching fabric, a first communications channel, and a second communications channel;
(b) a first computing element in electrical communication with the communications link, the first computing element providing a first output to the communications link;
(c) a second computing element in electrical communication with the communications link, the second computing element providing a second output to the communications link;
(d) a first local input-output (I/O) module in electrical communication with the first computing element and the communications link; and
(e) a second local I/O module in electrical communication with the second computing element and the communications link,
wherein at least one of the first local I/O module and the second local I/O module compares the first output and the second output and indicates a fault of at least one of the first computing element and the second computing element upon the detection of a miscompare of the first output and the second output, and
wherein the first local I/O module is in electrical communication with the second local I/O module via a sync bus to synchronize the first local I/O module and the second local I/O module, the synchronization of the first local I/O module and the second local I/O module providing a verification of state information about the first computing element and the second computing element.
2. The fault-tolerant server of claim 1 wherein each computing element further comprises a respective Central Processing Unit (CPU) and a respective local mass storage device.
3. The fault-tolerant server of claim 2 wherein the switching fabric comprises:
a first switching fabric in electrical communication with the CPU of the first computing element; and
a second switching fabric in electrical communication with the CPU of the second computing element, wherein each respective switching fabric is in electrical communication with at least one of the first local I/O module and the second local I/O module.
4. The fault-tolerant server of claim 1 further comprising a priority module to assign a priority to each respective computing element.
5. The fault-tolerant server of claim 4 wherein each local I/O module further comprises I/O fault-tolerant logic to determine whether at least one of the first computing element and the second computing element is faulty based on the priority.
6. The fault-tolerant server of claim 1 wherein each local I/O module further comprises I/O fault-tolerant logic to determine whether the first output and the second output are equivalent.
7. The fault-tolerant server of claim 6 wherein each I/O fault-tolerant logic comprises a comparator.
8. The fault-tolerant server of claim 6 wherein each I/O fault-tolerant logic further comprises a buffer to hold at least one of the first output and the second output from at least one of the CPUs.
9. The fault-tolerant server of claim 1 further comprising a voter delay buffer to store at least one of the first output and the second output upon a miscompare of the first output and the second output.
10. The fault-tolerant server of claim 1 further comprising a first delay module in electrical communication with the first local I/O module to delay transmission of at least one output to the first local I/O module and a second delay module in electrical communication with the second local I/O module to delay transmission of at least one output to the second local I/O module.
11. The fault-tolerant server of claim 1 wherein the first computing element and the second computing element further comprise a 1U rack-mount motherboard.
12. The fault-tolerant server of claim 1 wherein each respective local I/O module is located on a same motherboard as the respective computing element.
13. A method for a first computing element and a second computing element to execute in lockstep in a fault-tolerant server, the method comprising the steps of:
(a) establishing communication between the first computing element and a communications link, the communications link comprising a switching fabric, a first communications channel, and a second communications channel;
(b) establishing communication between the second computing element and the communications link;
(c) transmitting, by the first computing element, a first output to the communications link;
(d) transmitting, by the second computing element, a second output to the communications link; and
(e) comparing, by at least one of a local input-output (I/O) module of the first computing element and a local I/O module of the second computing element, the first output and the second output and indicating a fault of at least one of the first computing element and the second computing element in response thereto,
wherein the local I/O module of the first computing element is in electrical communication with the local I/O module of the second computing element via a sync bus to enable synchronization of the local I/O modules, the synchronization of the local I/O modules providing a verification of state information about the first computing element and the second computing element.
14. The method of claim 13 further comprising the step of transmitting a stop command to each computing element when the first output does not equal the second output.
15. The method of claim 13 further comprising detecting an error introduced by the communications link.
16. The method of claim 13 further comprising assigning a priority to each respective computing element.
17. The method of claim 16 further comprising determining whether at least one of the first computing element and the second computing element is faulty based on the priority.
18. The method of claim 16 further comprising determining whether the first output and the second output are equivalent.
19. The method of claim 13 further comprising storing at least one of the first output and the second output from at least one of the computing elements for a predetermined amount of time.
20. The method of claim 13 further comprising storing at least one of the first output and the second output upon a miscompare of the first output and the second output.
21. The method of claim 13 wherein the transmitting of the first output and the transmitting of the second output to the communications link occur simultaneously.
22. An apparatus for enabling a first computing element and a second computing element to execute in lockstep in a fault-tolerant server, the apparatus comprising:
(a) means for establishing communication between the first computing element and a communications link, the communications link comprising a switching fabric, a first communications channel, and a second communications channel;
(b) means for establishing communication between the second computing element and the communications link;
(c) means for transmitting, by the first computing element, a first output to the communications link;
(d) means for transmitting, by the second computing element, a second output to the communications link;
(e) means for comparing, by at least one of a local input-output (I/O) module of the first computing element and a local I/O module of the second computing element, the first output and the second output and indicating a fault of at least one of the first computing element and the second computing element in response thereto; and
(d) means for synchronizing the local I/O module of the first computing element and the local I/O module of the second computing element, the synchronization of the local I/O modules providing a verification of state information about the first computing element and the second computing element.
US09/832,466 2001-04-11 2001-04-11 Apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep Expired - Lifetime US6928583B2 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US09/832,466 US6928583B2 (en) 2001-04-11 2001-04-11 Apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep
AU2002257131A AU2002257131A1 (en) 2001-04-11 2002-04-10 An apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep
PCT/US2002/011192 WO2002084488A2 (en) 2001-04-11 2002-04-10 An apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US09/832,466 US6928583B2 (en) 2001-04-11 2001-04-11 Apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep

Publications (2)

Publication Number Publication Date
US20020152418A1 US20020152418A1 (en) 2002-10-17
US6928583B2 true US6928583B2 (en) 2005-08-09

Family

ID=25261733

Family Applications (1)

Application Number Title Priority Date Filing Date
US09/832,466 Expired - Lifetime US6928583B2 (en) 2001-04-11 2001-04-11 Apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep

Country Status (3)

Country Link
US (1) US6928583B2 (en)
AU (1) AU2002257131A1 (en)
WO (1) WO2002084488A2 (en)

Cited By (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030182594A1 (en) * 2002-03-19 2003-09-25 Sun Microsystems, Inc. Fault tolerant computer system
US20040078649A1 (en) * 2002-05-14 2004-04-22 Nec Corporation Computer system
US20040153731A1 (en) * 2002-07-10 2004-08-05 Nec Corporation Information processing apparatus
US20040153763A1 (en) * 1997-12-19 2004-08-05 Grochowski Edward T. Replay mechanism for correcting soft errors
US20040193954A1 (en) * 2001-09-21 2004-09-30 Bayerische Motoren Werke Aktiengesellschaft Method for transmitting messages between bus users
US20040193735A1 (en) * 2002-09-12 2004-09-30 Pavel Peleska Method and circuit arrangement for synchronization of synchronously or asynchronously clocked processor units
US20050060605A1 (en) * 2003-09-16 2005-03-17 Gibart Anthony Gerard High speed synchronization in dual-processor safety controller
US20050223275A1 (en) * 2004-03-30 2005-10-06 Jardine Robert L Performance data access
US20050223274A1 (en) * 2004-03-30 2005-10-06 Bernick David L Method and system executing user programs on non-deterministic processors
US20050278567A1 (en) * 2004-06-15 2005-12-15 Honeywell International Inc. Redundant processing architecture for single fault tolerance
US20060020852A1 (en) * 2004-03-30 2006-01-26 Bernick David L Method and system of servicing asynchronous interrupts in multiple processors executing a user program
US20060195849A1 (en) * 2002-09-12 2006-08-31 Pavel Peleska Method for synchronizing events, particularly for processors of fault-tolerant systems
US20060235937A1 (en) * 2005-04-18 2006-10-19 Dell Products L.P. System and method for processing commands in a storage enclosure
US20060236168A1 (en) * 2005-04-01 2006-10-19 Honeywell International Inc. System and method for dynamically optimizing performance and reliability of redundant processing systems
US20060242456A1 (en) * 2005-04-26 2006-10-26 Kondo Thomas J Method and system of copying memory from a source processor to a target processor by duplicating memory writes
US20060248409A1 (en) * 2003-06-23 2006-11-02 Dietmar Baumann Method and device for monitoring a distributed system
US7328371B1 (en) * 2004-10-15 2008-02-05 Advanced Micro Devices, Inc. Core redundancy in a chip multiprocessor for highly reliable systems
US20080091927A1 (en) * 2004-10-25 2008-04-17 Bernd Mueller Method And Device For A Switchover In A Computer System Having At Least Two Processing Units
US20080155306A1 (en) * 2005-12-21 2008-06-26 Combs William E Method and system for controlling command execution
US20080196037A1 (en) * 2007-02-13 2008-08-14 Thales Process for maintaining execution synchronization between several asynchronous processors working in parallel and in a redundant manner
US20090070564A1 (en) * 2005-01-25 2009-03-12 Del Vigna Jr Paul Method and system of aligning execution point of duplicate copies of a user program by exchanging information about instructions executed
US20090230255A1 (en) * 2008-03-11 2009-09-17 Lemonovich John E Cab signal receiver demodulator employing redundant, diverse field programmable gate arrays
US20100281239A1 (en) * 2009-04-29 2010-11-04 Ranganathan Sudhakar Reliable execution using compare and transfer instruction on an smt machine
US8015390B1 (en) * 2008-03-19 2011-09-06 Rockwell Collins, Inc. Dissimilar processor synchronization in fly-by-wire high integrity computing platforms and displays
US20120304024A1 (en) * 2010-02-16 2012-11-29 Freescale Semiconductor, Inc. Data processing method, data processor and apparatus including a data processor
US9823983B2 (en) 2014-09-25 2017-11-21 Nxp Usa, Inc. Electronic fault detection unit
US9842014B2 (en) 2012-11-22 2017-12-12 Nxp Usa, Inc. Data processing device, method of execution error detection and integrated circuit
US10063567B2 (en) 2014-11-13 2018-08-28 Virtual Software Systems, Inc. System for cross-host, multi-thread session alignment
US20210311522A1 (en) * 2020-04-07 2021-10-07 Airbus Operations Sas Method and system for synchronizing computing units of an aircraft
US11263136B2 (en) 2019-08-02 2022-03-01 Stratus Technologies Ireland Ltd. Fault tolerant systems and methods for cache flush coordination
US11281538B2 (en) 2019-07-31 2022-03-22 Stratus Technologies Ireland Ltd. Systems and methods for checkpointing in a fault tolerant system
US11288123B2 (en) 2019-07-31 2022-03-29 Stratus Technologies Ireland Ltd. Systems and methods for applying checkpoints on a secondary computer in parallel with transmission
US11288143B2 (en) 2020-08-26 2022-03-29 Stratus Technologies Ireland Ltd. Real-time fault-tolerant checkpointing
US11429466B2 (en) 2019-07-31 2022-08-30 Stratus Technologies Ireland Ltd. Operating system-based systems and method of achieving fault tolerance
US11586514B2 (en) 2018-08-13 2023-02-21 Stratus Technologies Ireland Ltd. High reliability fault tolerant computer architecture
US11620196B2 (en) 2019-07-31 2023-04-04 Stratus Technologies Ireland Ltd. Computer duplication and configuration management systems and methods
US11641395B2 (en) 2019-07-31 2023-05-02 Stratus Technologies Ireland Ltd. Fault tolerant systems and methods incorporating a minimum checkpoint interval

Families Citing this family (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6948092B2 (en) * 1998-12-10 2005-09-20 Hewlett-Packard Development Company, L.P. System recovery from errors for processor and associated components
US6971043B2 (en) * 2001-04-11 2005-11-29 Stratus Technologies Bermuda Ltd Apparatus and method for accessing a mass storage device in a fault-tolerant server
JP2003015900A (en) * 2001-06-28 2003-01-17 Hitachi Ltd Follow-up type multiplex system and data processing method capable of improving reliability by follow-up
JP3606281B2 (en) * 2002-06-07 2005-01-05 オムロン株式会社 Programmable controller, CPU unit, special function module, and duplex processing method
US20050114735A1 (en) * 2003-11-20 2005-05-26 Smith Zachary S. Systems and methods for verifying core determinacy
US7290169B2 (en) * 2004-04-06 2007-10-30 Hewlett-Packard Development Company, L.P. Core-level processor lockstepping
US7237144B2 (en) * 2004-04-06 2007-06-26 Hewlett-Packard Development Company, L.P. Off-chip lockstep checking
US7296181B2 (en) * 2004-04-06 2007-11-13 Hewlett-Packard Development Company, L.P. Lockstep error signaling
DE102005037222A1 (en) * 2004-10-25 2007-02-15 Robert Bosch Gmbh Mode signal evaluating method for computer system, involves generating mode signal and changes in mode signal in computer system, where changes in mode signal and mode signal are used for evaluation of signal in computer system
JP2006178616A (en) * 2004-12-21 2006-07-06 Nec Corp Fault tolerant system, controller used thereform, operation method and operation program
JP4168403B2 (en) * 2004-12-21 2008-10-22 日本電気株式会社 Fault tolerant system, control device used therefor, access control method, and control program
PT1764694E (en) * 2005-09-16 2008-09-08 Siemens Transportation Systems Redundant control method and apparatus for fail safe computers
WO2009015689A1 (en) * 2007-07-31 2009-02-05 Telefonaktiebolaget Lm Ericsson (Publ) All optical batcher banyan switch, batcher switch, banyan switch and contention manager
IT1391785B1 (en) * 2008-11-21 2012-01-27 St Microelectronics Srl ELECTRONIC SYSTEM FOR DETECTION OF FAILURE
JP4709268B2 (en) * 2008-11-28 2011-06-22 日立オートモティブシステムズ株式会社 Multi-core system for vehicle control or control device for internal combustion engine
JP6098778B2 (en) * 2012-03-29 2017-03-22 日本電気株式会社 Redundant system, redundancy method, redundancy system availability improving method, and program
TWI480741B (en) * 2012-12-12 2015-04-11 Inventec Corp A sever motherboard
CN104714439B (en) * 2013-12-16 2018-03-27 雅特生嵌入式计算有限公司 Safety relay case system
GB2555628B (en) * 2016-11-04 2019-02-20 Advanced Risc Mach Ltd Main processor error detection using checker processors
US10514990B2 (en) * 2017-11-27 2019-12-24 Intel Corporation Mission-critical computing architecture
US10946866B2 (en) 2018-03-31 2021-03-16 Intel Corporation Core tightly coupled lockstep for high functional safety
US11120642B2 (en) 2018-06-27 2021-09-14 Intel Corporation Functional safety critical audio system for autonomous and industrial applications
US11520297B2 (en) 2019-03-29 2022-12-06 Intel Corporation Enhancing diagnostic capabilities of computing systems by combining variable patrolling API and comparison mechanism of variables
US11132268B2 (en) * 2019-10-21 2021-09-28 The Boeing Company System and method for synchronizing communications between a plurality of processors
EP3936949A1 (en) * 2020-07-09 2022-01-12 Siemens Aktiengesellschaft Redundant automation system and method for operating a redundant automation system

Citations (100)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US1200155A (en) 1915-06-28 1916-10-03 James Stanley Wright Paper-clip.
US3192362A (en) 1961-08-22 1965-06-29 Sperry Rand Corp Instruction counter with sequential address checking means
US3208028A (en) 1963-04-30 1965-09-21 Ind Electronic Hardware Corp Multilayer circuitry with interrupted lines
US3212048A (en) 1963-04-30 1965-10-12 Ind Electronic Hardware Corp Multilayer circuitry with spring strips
US3252056A (en) 1963-05-03 1966-05-17 Unelco Ltd Unelco Limitee Electrical distribution system
US3292131A (en) 1963-12-20 1966-12-13 William J Smith Device for interconnection of electrical apparatus
US3533082A (en) 1968-01-15 1970-10-06 Ibm Instruction retry apparatus including means for restoring the original contents of altered source operands
US3533065A (en) 1968-01-15 1970-10-06 Ibm Data processing system execution retry control
US3593307A (en) 1968-09-20 1971-07-13 Adaptronics Inc Redundant, self-checking, self-organizing control system
US3665173A (en) 1968-09-03 1972-05-23 Ibm Triple modular redundancy/sparing
US3681578A (en) 1969-11-21 1972-08-01 Marconi Co Ltd Fault location and reconfiguration in redundant data processors
US3688274A (en) 1970-12-23 1972-08-29 Ibm Command retry control by peripheral devices
US3783250A (en) 1972-02-25 1974-01-01 Nasa Adaptive voting computer system
US3879712A (en) 1972-06-03 1975-04-22 Plessey Handel Investment Ag Data processing system fault diagnostic arrangements
US3923359A (en) 1971-07-09 1975-12-02 Pressey Handel Und Investments Multi-layer printed-circuit boards
US4030074A (en) 1974-06-03 1977-06-14 Centro Studi E Laboratori Telecomunicazioni System for checking two data processors operating in parallel
US4099234A (en) 1976-11-15 1978-07-04 Honeywell Information Systems Inc. Input/output processing system utilizing locked processors
US4176258A (en) 1978-05-01 1979-11-27 Intel Corporation Method and circuit for checking integrated circuit chips
US4228496A (en) 1976-09-07 1980-10-14 Tandem Computers Incorporated Multiprocessor system
US4241381A (en) 1979-04-04 1980-12-23 Amp Incorporated Bus bar assembly for circuit cards
US4323966A (en) 1980-02-05 1982-04-06 The Bendix Corporation Operations controller for a fault-tolerant multiple computer system
US4358823A (en) 1977-03-25 1982-11-09 Trw, Inc. Double redundant processor
US4366535A (en) 1978-03-03 1982-12-28 Cselt - Centro Studi E Laboratori Telecomunicazioni S.P.A. Modular signal-processing system
US4369494A (en) 1974-12-09 1983-01-18 Compagnie Honeywell Bull Apparatus and method for providing synchronization between processes and events occurring at different times in a data processing system
US4375683A (en) 1980-11-12 1983-03-01 August Systems Fault tolerant computational system and voter circuit
US4453215A (en) 1981-10-01 1984-06-05 Stratus Computer, Inc. Central processing apparatus for fault-tolerant computing
US4492419A (en) 1979-11-07 1985-01-08 Denckert Lennart Holger Electric distribution center
US4503535A (en) 1982-06-30 1985-03-05 Intel Corporation Apparatus for recovery from failures in a multiprocessing system
US4507784A (en) 1982-05-21 1985-03-26 International Computers Limited Data processing systems
US4567654A (en) 1984-04-02 1986-02-04 Emhart Industries, Inc. Bussing block
US4583224A (en) 1982-11-08 1986-04-15 Hitachi, Ltd. Fault tolerable redundancy control
US4597084A (en) 1981-10-01 1986-06-24 Stratus Computer, Inc. Computer memory apparatus
US4610013A (en) 1983-11-08 1986-09-02 Avco Corporation Remote multiplexer terminal with redundant central processor units
US4622667A (en) 1984-11-27 1986-11-11 Sperry Corporation Digital fail operational automatic flight control system utilizing redundant dissimilar data processing
US4644498A (en) 1983-04-04 1987-02-17 General Electric Company Fault-tolerant real time clock
US4648031A (en) 1982-06-21 1987-03-03 International Business Machines Corporation Method and apparatus for restarting a computing system
US4654846A (en) 1983-12-20 1987-03-31 Rca Corporation Spacecraft autonomous redundancy control
US4686677A (en) 1985-08-02 1987-08-11 Unisys Corporation Apparatus and method for detecting time-related faults
US4736377A (en) 1986-02-11 1988-04-05 Bradley Telcom Corp. Method for determining reliability of high speed digital transmission by use of a synchronized low speed side channel
US4739498A (en) 1984-08-31 1988-04-19 Messerschmitt-Bolkow-Blohm Gmbh Arrangement for the automatic reconfiguration of an intact equipment combination
US4750177A (en) 1981-10-01 1988-06-07 Stratus Computer, Inc. Digital data processor apparatus with pipelined fault tolerant bus protocol
US4799140A (en) 1986-03-06 1989-01-17 Orbital Sciences Corporation Ii Majority vote sequencer
US4816990A (en) 1986-11-05 1989-03-28 Stratus Computer, Inc. Method and apparatus for fault-tolerant computer system having expandable processor section
US4869673A (en) 1987-12-02 1989-09-26 Amp Incorporated Circuit panel assembly with elevated power buses
US4905181A (en) 1987-04-20 1990-02-27 Wang Laboratories, Inc. Interactive system with state manager subsystem
US4907232A (en) 1988-04-28 1990-03-06 The Charles Stark Draper Laboratory, Inc. Fault-tolerant parallel processing system
US4916695A (en) 1987-04-16 1990-04-10 Telefonaktiebolaget L M Ericsson Stored program controlled real time system including three substantially identical processors
US5017145A (en) 1988-04-27 1991-05-21 Nippon Telegraph & Telephone Corporation Matrix switching device and method of manufacturing the same
US5020024A (en) 1987-01-16 1991-05-28 Stratus Computer, Inc. Method and apparatus for detecting selected absence of digital logic synchronism
US5070430A (en) 1989-09-25 1991-12-03 Siemens Aktiengesellschaft Electrical installation composed of individual subassemblies
US5089958A (en) 1989-01-23 1992-02-18 Vortex Systems, Inc. Fault tolerant computer backup system
US5136704A (en) 1989-06-28 1992-08-04 Motorola, Inc. Redundant microprocessor control system using locks and keys
US5193180A (en) 1991-06-21 1993-03-09 Pure Software Inc. System for modifying relocatable object code files to monitor accesses to dynamically allocated memory
US5220668A (en) 1990-09-21 1993-06-15 Stratus Computer, Inc. Digital data processor with maintenance and diagnostic system
US5226152A (en) 1990-12-07 1993-07-06 Motorola, Inc. Functional lockstep arrangement for redundant processors
US5231640A (en) 1990-07-20 1993-07-27 Unisys Corporation Fault tolerant processor/memory architecture
US5247522A (en) 1990-11-27 1993-09-21 Digital Equipment Corporation Fault tolerant bus
US5249187A (en) 1987-09-04 1993-09-28 Digital Equipment Corporation Dual rail processors with error checking on I/O reads
US5251303A (en) 1989-01-13 1993-10-05 International Business Machines Corporation System for DMA block data transfer based on linked control blocks
US5255367A (en) * 1987-09-04 1993-10-19 Digital Equipment Corporation Fault tolerant, synchronized twin computer system with error checking of I/O communication
US5263034A (en) 1990-10-09 1993-11-16 Bull Information Systems Inc. Error detection in the basic processing unit of a VLSI central processor
US5271023A (en) 1991-06-03 1993-12-14 Motorola, Inc. Uninterruptable fault tolerant data processor
US5270699A (en) 1991-08-13 1993-12-14 Rockwell International Corporation Fault tolerant signaling
US5283870A (en) 1991-10-04 1994-02-01 Bull Hn Information Systems Inc. Method and apparatus for avoiding processor deadly embrace in a multiprocessor system
US5295258A (en) 1989-12-22 1994-03-15 Tandem Computers Incorporated Fault-tolerant computer system with online recovery and reintegration of redundant components
US5313627A (en) 1992-01-02 1994-05-17 International Business Machines Corp. Parity error detection and recovery
US5317726A (en) 1987-11-09 1994-05-31 Tandem Computers Incorporated Multiple-processor computer system with asynchronous execution of identical code streams
US5321706A (en) 1990-06-27 1994-06-14 International Business Machines Corporation Method and apparatus for checking the address and contents of a memory array
US5361267A (en) 1992-04-24 1994-11-01 Digital Equipment Corporation Scheme for error handling in a computer system
US5379381A (en) 1991-08-12 1995-01-03 Stratus Computer, Inc. System using separate transfer circuits for performing different transfer operations respectively and scanning I/O devices status upon absence of both operations
US5384906A (en) 1987-11-09 1995-01-24 Tandem Computers Incorporated Method and apparatus for synchronizing a plurality of processors
US5388242A (en) 1988-12-09 1995-02-07 Tandem Computers Incorporated Multiprocessor system with each processor executing the same instruction sequence and hierarchical memory providing on demand page swapping
US5392302A (en) 1991-03-13 1995-02-21 Quantum Corp. Address error detection technique for increasing the reliability of a storage subsystem
US5404361A (en) 1992-07-27 1995-04-04 Storage Technology Corporation Method and apparatus for ensuring data integrity in a dynamically mapped data storage subsystem
US5423024A (en) 1991-05-06 1995-06-06 Stratus Computer, Inc. Fault tolerant processing section with dynamically reconfigurable voting
US5428766A (en) 1992-12-01 1995-06-27 Digital Equipment Corporation Error detection scheme in a multiprocessor environment
US5430866A (en) 1990-05-11 1995-07-04 International Business Machines Corporation Method and apparatus for deriving mirrored unit state when re-initializing a system
US5537535A (en) 1993-09-20 1996-07-16 Fujitsu Limited Multi-CPU system having fault monitoring facility
US5586253A (en) 1994-12-15 1996-12-17 Stratus Computer Method and apparatus for validating I/O addresses in a fault-tolerant computer system
US5600784A (en) 1993-12-01 1997-02-04 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US5696895A (en) 1995-05-19 1997-12-09 Compaq Computer Corporation Fault tolerant multiple network servers
US5701410A (en) 1996-09-09 1997-12-23 Ford Motor Company Method and system for detecting fault conditions on multiplexed networks
US5701457A (en) 1994-09-08 1997-12-23 Hitachi, Ltd. Method of designated time interval reservation access process of online updating and backing up of large database versions without reserving exclusive control
US5751932A (en) * 1992-12-17 1998-05-12 Tandem Computers Incorporated Fail-fast, fail-functional, fault-tolerant multiprocessor system
US5758065A (en) 1995-11-30 1998-05-26 Ncr Corporation System and method of establishing error precedence in a computer system
US5781910A (en) 1996-09-13 1998-07-14 Stratus Computer, Inc. Preforming concurrent transactions in a replicated database environment
US5790397A (en) 1996-09-17 1998-08-04 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US5838899A (en) 1994-09-20 1998-11-17 Stratus Computer Digital data processing methods and apparatus for fault isolation
US5845060A (en) 1993-03-02 1998-12-01 Tandem Computers, Incorporated High-performance fault tolerant computer system with clock length synchronization of loosely coupled processors
US5862145A (en) 1996-09-12 1999-01-19 Advanced Micro Devices, Inc. Method and system for identifying an error condition due to a faulty cable connection in an ethernet network
US5896523A (en) 1997-06-04 1999-04-20 Marathon Technologies Corporation Loosely-coupled, synchronized execution
US5928339A (en) 1996-10-18 1999-07-27 Matsushita Electric Industrial Co., Ltd. DMA-transferring stream data apparatus between a memory and ports where a command list includes size and start address of data stored in the memory
US5978936A (en) * 1997-11-19 1999-11-02 International Business Machines Corporation Run time error probe in a network computing environment
US5983371A (en) 1997-07-11 1999-11-09 Marathon Technologies Corporation Active failure detection
US6032271A (en) 1996-06-05 2000-02-29 Compaq Computer Corporation Method and apparatus for identifying faulty devices in a computer system
US6049894A (en) 1995-02-22 2000-04-11 Adaptec, Inc. Error generation circuit for testing a digital bus
US6141769A (en) 1996-05-16 2000-10-31 Resilience Corporation Triple modular redundant computer system and associated method
US6223304B1 (en) * 1998-06-18 2001-04-24 Telefonaktiebolaget Lm Ericsson (Publ) Synchronization of processors in a fault tolerant multi-processor system
US6393582B1 (en) * 1998-12-10 2002-05-21 Compaq Computer Corporation Error self-checking and recovery using lock-step processor pair architecture
US6449732B1 (en) * 1998-12-18 2002-09-10 Triconex Corporation Method and apparatus for processing control using a multiple redundant processor control system

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3593173A (en) * 1968-12-30 1971-07-13 Forbro Design Corp Input limiting for bipolar operational transistor amplifier
US5675579A (en) * 1992-12-17 1997-10-07 Tandem Computers Incorporated Method for verifying responses to messages using a barrier message
US6252878B1 (en) * 1997-10-30 2001-06-26 Cisco Technology, Inc. Switched architecture access server

Patent Citations (105)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US1200155A (en) 1915-06-28 1916-10-03 James Stanley Wright Paper-clip.
US3192362A (en) 1961-08-22 1965-06-29 Sperry Rand Corp Instruction counter with sequential address checking means
US3208028A (en) 1963-04-30 1965-09-21 Ind Electronic Hardware Corp Multilayer circuitry with interrupted lines
US3212048A (en) 1963-04-30 1965-10-12 Ind Electronic Hardware Corp Multilayer circuitry with spring strips
US3252056A (en) 1963-05-03 1966-05-17 Unelco Ltd Unelco Limitee Electrical distribution system
US3292131A (en) 1963-12-20 1966-12-13 William J Smith Device for interconnection of electrical apparatus
US3533082A (en) 1968-01-15 1970-10-06 Ibm Instruction retry apparatus including means for restoring the original contents of altered source operands
US3533065A (en) 1968-01-15 1970-10-06 Ibm Data processing system execution retry control
US3665173A (en) 1968-09-03 1972-05-23 Ibm Triple modular redundancy/sparing
US3593307A (en) 1968-09-20 1971-07-13 Adaptronics Inc Redundant, self-checking, self-organizing control system
US3681578A (en) 1969-11-21 1972-08-01 Marconi Co Ltd Fault location and reconfiguration in redundant data processors
US3688274A (en) 1970-12-23 1972-08-29 Ibm Command retry control by peripheral devices
US3923359A (en) 1971-07-09 1975-12-02 Pressey Handel Und Investments Multi-layer printed-circuit boards
US3783250A (en) 1972-02-25 1974-01-01 Nasa Adaptive voting computer system
US3879712A (en) 1972-06-03 1975-04-22 Plessey Handel Investment Ag Data processing system fault diagnostic arrangements
US4030074A (en) 1974-06-03 1977-06-14 Centro Studi E Laboratori Telecomunicazioni System for checking two data processors operating in parallel
US4369494A (en) 1974-12-09 1983-01-18 Compagnie Honeywell Bull Apparatus and method for providing synchronization between processes and events occurring at different times in a data processing system
US4228496A (en) 1976-09-07 1980-10-14 Tandem Computers Incorporated Multiprocessor system
US4356550A (en) 1976-09-07 1982-10-26 Tandem Computers Incorporated Multiprocessor system
US4099234A (en) 1976-11-15 1978-07-04 Honeywell Information Systems Inc. Input/output processing system utilizing locked processors
US4358823A (en) 1977-03-25 1982-11-09 Trw, Inc. Double redundant processor
US4366535A (en) 1978-03-03 1982-12-28 Cselt - Centro Studi E Laboratori Telecomunicazioni S.P.A. Modular signal-processing system
US4176258A (en) 1978-05-01 1979-11-27 Intel Corporation Method and circuit for checking integrated circuit chips
US4241381A (en) 1979-04-04 1980-12-23 Amp Incorporated Bus bar assembly for circuit cards
US4492419A (en) 1979-11-07 1985-01-08 Denckert Lennart Holger Electric distribution center
US4323966A (en) 1980-02-05 1982-04-06 The Bendix Corporation Operations controller for a fault-tolerant multiple computer system
US4375683A (en) 1980-11-12 1983-03-01 August Systems Fault tolerant computational system and voter circuit
US4486826A (en) 1981-10-01 1984-12-04 Stratus Computer, Inc. Computer peripheral control apparatus
US4750177A (en) 1981-10-01 1988-06-07 Stratus Computer, Inc. Digital data processor apparatus with pipelined fault tolerant bus protocol
US4597084A (en) 1981-10-01 1986-06-24 Stratus Computer, Inc. Computer memory apparatus
US4654857A (en) 1981-10-01 1987-03-31 Stratus Computer, Inc. Digital data processor with high reliability
US4453215A (en) 1981-10-01 1984-06-05 Stratus Computer, Inc. Central processing apparatus for fault-tolerant computing
US4507784A (en) 1982-05-21 1985-03-26 International Computers Limited Data processing systems
US4648031A (en) 1982-06-21 1987-03-03 International Business Machines Corporation Method and apparatus for restarting a computing system
US4503535A (en) 1982-06-30 1985-03-05 Intel Corporation Apparatus for recovery from failures in a multiprocessing system
US4583224A (en) 1982-11-08 1986-04-15 Hitachi, Ltd. Fault tolerable redundancy control
US4644498A (en) 1983-04-04 1987-02-17 General Electric Company Fault-tolerant real time clock
US4610013A (en) 1983-11-08 1986-09-02 Avco Corporation Remote multiplexer terminal with redundant central processor units
US4654846A (en) 1983-12-20 1987-03-31 Rca Corporation Spacecraft autonomous redundancy control
US4567654A (en) 1984-04-02 1986-02-04 Emhart Industries, Inc. Bussing block
US4739498A (en) 1984-08-31 1988-04-19 Messerschmitt-Bolkow-Blohm Gmbh Arrangement for the automatic reconfiguration of an intact equipment combination
US4622667A (en) 1984-11-27 1986-11-11 Sperry Corporation Digital fail operational automatic flight control system utilizing redundant dissimilar data processing
US4686677A (en) 1985-08-02 1987-08-11 Unisys Corporation Apparatus and method for detecting time-related faults
US4736377A (en) 1986-02-11 1988-04-05 Bradley Telcom Corp. Method for determining reliability of high speed digital transmission by use of a synchronized low speed side channel
US4799140A (en) 1986-03-06 1989-01-17 Orbital Sciences Corporation Ii Majority vote sequencer
US4816990A (en) 1986-11-05 1989-03-28 Stratus Computer, Inc. Method and apparatus for fault-tolerant computer system having expandable processor section
US5020024A (en) 1987-01-16 1991-05-28 Stratus Computer, Inc. Method and apparatus for detecting selected absence of digital logic synchronism
US4916695A (en) 1987-04-16 1990-04-10 Telefonaktiebolaget L M Ericsson Stored program controlled real time system including three substantially identical processors
US4905181A (en) 1987-04-20 1990-02-27 Wang Laboratories, Inc. Interactive system with state manager subsystem
US5249187A (en) 1987-09-04 1993-09-28 Digital Equipment Corporation Dual rail processors with error checking on I/O reads
US5255367A (en) * 1987-09-04 1993-10-19 Digital Equipment Corporation Fault tolerant, synchronized twin computer system with error checking of I/O communication
US5317726A (en) 1987-11-09 1994-05-31 Tandem Computers Incorporated Multiple-processor computer system with asynchronous execution of identical code streams
US5384906A (en) 1987-11-09 1995-01-24 Tandem Computers Incorporated Method and apparatus for synchronizing a plurality of processors
US4869673A (en) 1987-12-02 1989-09-26 Amp Incorporated Circuit panel assembly with elevated power buses
US5017145A (en) 1988-04-27 1991-05-21 Nippon Telegraph & Telephone Corporation Matrix switching device and method of manufacturing the same
US4907232A (en) 1988-04-28 1990-03-06 The Charles Stark Draper Laboratory, Inc. Fault-tolerant parallel processing system
US5388242A (en) 1988-12-09 1995-02-07 Tandem Computers Incorporated Multiprocessor system with each processor executing the same instruction sequence and hierarchical memory providing on demand page swapping
US5251303A (en) 1989-01-13 1993-10-05 International Business Machines Corporation System for DMA block data transfer based on linked control blocks
US5089958A (en) 1989-01-23 1992-02-18 Vortex Systems, Inc. Fault tolerant computer backup system
US5136704A (en) 1989-06-28 1992-08-04 Motorola, Inc. Redundant microprocessor control system using locks and keys
US5070430A (en) 1989-09-25 1991-12-03 Siemens Aktiengesellschaft Electrical installation composed of individual subassemblies
US5295258A (en) 1989-12-22 1994-03-15 Tandem Computers Incorporated Fault-tolerant computer system with online recovery and reintegration of redundant components
US5430866A (en) 1990-05-11 1995-07-04 International Business Machines Corporation Method and apparatus for deriving mirrored unit state when re-initializing a system
US5321706A (en) 1990-06-27 1994-06-14 International Business Machines Corporation Method and apparatus for checking the address and contents of a memory array
US5231640A (en) 1990-07-20 1993-07-27 Unisys Corporation Fault tolerant processor/memory architecture
US5220668A (en) 1990-09-21 1993-06-15 Stratus Computer, Inc. Digital data processor with maintenance and diagnostic system
US5263034A (en) 1990-10-09 1993-11-16 Bull Information Systems Inc. Error detection in the basic processing unit of a VLSI central processor
US5247522A (en) 1990-11-27 1993-09-21 Digital Equipment Corporation Fault tolerant bus
US5226152A (en) 1990-12-07 1993-07-06 Motorola, Inc. Functional lockstep arrangement for redundant processors
US5392302A (en) 1991-03-13 1995-02-21 Quantum Corp. Address error detection technique for increasing the reliability of a storage subsystem
US5423024A (en) 1991-05-06 1995-06-06 Stratus Computer, Inc. Fault tolerant processing section with dynamically reconfigurable voting
US5271023A (en) 1991-06-03 1993-12-14 Motorola, Inc. Uninterruptable fault tolerant data processor
US5193180A (en) 1991-06-21 1993-03-09 Pure Software Inc. System for modifying relocatable object code files to monitor accesses to dynamically allocated memory
US5379381A (en) 1991-08-12 1995-01-03 Stratus Computer, Inc. System using separate transfer circuits for performing different transfer operations respectively and scanning I/O devices status upon absence of both operations
US5270699A (en) 1991-08-13 1993-12-14 Rockwell International Corporation Fault tolerant signaling
US5283870A (en) 1991-10-04 1994-02-01 Bull Hn Information Systems Inc. Method and apparatus for avoiding processor deadly embrace in a multiprocessor system
US5313627A (en) 1992-01-02 1994-05-17 International Business Machines Corp. Parity error detection and recovery
US5361267A (en) 1992-04-24 1994-11-01 Digital Equipment Corporation Scheme for error handling in a computer system
US5404361A (en) 1992-07-27 1995-04-04 Storage Technology Corporation Method and apparatus for ensuring data integrity in a dynamically mapped data storage subsystem
US5428766A (en) 1992-12-01 1995-06-27 Digital Equipment Corporation Error detection scheme in a multiprocessor environment
US5751932A (en) * 1992-12-17 1998-05-12 Tandem Computers Incorporated Fail-fast, fail-functional, fault-tolerant multiprocessor system
US5845060A (en) 1993-03-02 1998-12-01 Tandem Computers, Incorporated High-performance fault tolerant computer system with clock length synchronization of loosely coupled processors
US5537535A (en) 1993-09-20 1996-07-16 Fujitsu Limited Multi-CPU system having fault monitoring facility
US5600784A (en) 1993-12-01 1997-02-04 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US6038685A (en) 1993-12-01 2000-03-14 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US5956474A (en) 1993-12-01 1999-09-21 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US5701457A (en) 1994-09-08 1997-12-23 Hitachi, Ltd. Method of designated time interval reservation access process of online updating and backing up of large database versions without reserving exclusive control
US5838899A (en) 1994-09-20 1998-11-17 Stratus Computer Digital data processing methods and apparatus for fault isolation
US5586253A (en) 1994-12-15 1996-12-17 Stratus Computer Method and apparatus for validating I/O addresses in a fault-tolerant computer system
US6049894A (en) 1995-02-22 2000-04-11 Adaptec, Inc. Error generation circuit for testing a digital bus
US5696895A (en) 1995-05-19 1997-12-09 Compaq Computer Corporation Fault tolerant multiple network servers
US5758065A (en) 1995-11-30 1998-05-26 Ncr Corporation System and method of establishing error precedence in a computer system
US6141769A (en) 1996-05-16 2000-10-31 Resilience Corporation Triple modular redundant computer system and associated method
US6032271A (en) 1996-06-05 2000-02-29 Compaq Computer Corporation Method and apparatus for identifying faulty devices in a computer system
US5701410A (en) 1996-09-09 1997-12-23 Ford Motor Company Method and system for detecting fault conditions on multiplexed networks
US5862145A (en) 1996-09-12 1999-01-19 Advanced Micro Devices, Inc. Method and system for identifying an error condition due to a faulty cable connection in an ethernet network
US5781910A (en) 1996-09-13 1998-07-14 Stratus Computer, Inc. Preforming concurrent transactions in a replicated database environment
US5790397A (en) 1996-09-17 1998-08-04 Marathon Technologies Corporation Fault resilient/fault tolerant computing
US5928339A (en) 1996-10-18 1999-07-27 Matsushita Electric Industrial Co., Ltd. DMA-transferring stream data apparatus between a memory and ports where a command list includes size and start address of data stored in the memory
US5896523A (en) 1997-06-04 1999-04-20 Marathon Technologies Corporation Loosely-coupled, synchronized execution
US5983371A (en) 1997-07-11 1999-11-09 Marathon Technologies Corporation Active failure detection
US5978936A (en) * 1997-11-19 1999-11-02 International Business Machines Corporation Run time error probe in a network computing environment
US6223304B1 (en) * 1998-06-18 2001-04-24 Telefonaktiebolaget Lm Ericsson (Publ) Synchronization of processors in a fault tolerant multi-processor system
US6393582B1 (en) * 1998-12-10 2002-05-21 Compaq Computer Corporation Error self-checking and recovery using lock-step processor pair architecture
US6449732B1 (en) * 1998-12-18 2002-09-10 Triconex Corporation Method and apparatus for processing control using a multiple redundant processor control system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Nakamikawa et al., "High Performance Fault Tolerant Computer and its Fault Recovery," IEEE, vol. 4, pp. 2-6 (1997).
Patent Cooperation Treaty, International Search Report, International Application No. PCT/US02/11192, mailed on May 5, 2003, 8 pages.

Cited By (57)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7340643B2 (en) * 1997-12-19 2008-03-04 Intel Corporation Replay mechanism for correcting soft errors
US20040153763A1 (en) * 1997-12-19 2004-08-05 Grochowski Edward T. Replay mechanism for correcting soft errors
US7310746B2 (en) * 2001-09-21 2007-12-18 Bayerische Motoren Werke Aktiengesellschaft Method for transmitting messages between bus users
US20040193954A1 (en) * 2001-09-21 2004-09-30 Bayerische Motoren Werke Aktiengesellschaft Method for transmitting messages between bus users
US20030182594A1 (en) * 2002-03-19 2003-09-25 Sun Microsystems, Inc. Fault tolerant computer system
US7124319B2 (en) * 2002-03-19 2006-10-17 Sun Microsystems, Inc. Delay compensation for synchronous processing sets
US20040078649A1 (en) * 2002-05-14 2004-04-22 Nec Corporation Computer system
US7243257B2 (en) * 2002-05-14 2007-07-10 Nec Corporation Computer system for preventing inter-node fault propagation
US20040153731A1 (en) * 2002-07-10 2004-08-05 Nec Corporation Information processing apparatus
US20040193735A1 (en) * 2002-09-12 2004-09-30 Pavel Peleska Method and circuit arrangement for synchronization of synchronously or asynchronously clocked processor units
US20060195849A1 (en) * 2002-09-12 2006-08-31 Pavel Peleska Method for synchronizing events, particularly for processors of fault-tolerant systems
US20060248409A1 (en) * 2003-06-23 2006-11-02 Dietmar Baumann Method and device for monitoring a distributed system
US7502973B2 (en) * 2003-06-23 2009-03-10 Robert Bosch Gmbh Method and device for monitoring a distributed system
US20050060605A1 (en) * 2003-09-16 2005-03-17 Gibart Anthony Gerard High speed synchronization in dual-processor safety controller
US7287184B2 (en) * 2003-09-16 2007-10-23 Rockwell Automation Technologies, Inc. High speed synchronization in dual-processor safety controller
US7434098B2 (en) 2004-03-30 2008-10-07 Hewlett-Packard Development Company, L.P. Method and system of determining whether a user program has made a system level call
US20050246581A1 (en) * 2004-03-30 2005-11-03 Hewlett-Packard Development Company, L.P. Error handling system in a redundant processor
US20050223275A1 (en) * 2004-03-30 2005-10-06 Jardine Robert L Performance data access
US20050223274A1 (en) * 2004-03-30 2005-10-06 Bernick David L Method and system executing user programs on non-deterministic processors
US20060020852A1 (en) * 2004-03-30 2006-01-26 Bernick David L Method and system of servicing asynchronous interrupts in multiple processors executing a user program
US7426656B2 (en) * 2004-03-30 2008-09-16 Hewlett-Packard Development Company, L.P. Method and system executing user programs on non-deterministic processors
US20050246587A1 (en) * 2004-03-30 2005-11-03 Bernick David L Method and system of determining whether a user program has made a system level call
US20050278567A1 (en) * 2004-06-15 2005-12-15 Honeywell International Inc. Redundant processing architecture for single fault tolerance
US7392426B2 (en) * 2004-06-15 2008-06-24 Honeywell International Inc. Redundant processing architecture for single fault tolerance
US7328371B1 (en) * 2004-10-15 2008-02-05 Advanced Micro Devices, Inc. Core redundancy in a chip multiprocessor for highly reliable systems
US20080091927A1 (en) * 2004-10-25 2008-04-17 Bernd Mueller Method And Device For A Switchover In A Computer System Having At Least Two Processing Units
US20090070564A1 (en) * 2005-01-25 2009-03-12 Del Vigna Jr Paul Method and system of aligning execution point of duplicate copies of a user program by exchanging information about instructions executed
US7752494B2 (en) * 2005-01-25 2010-07-06 Hewlett-Packard Development Company, L.P. Method and system of aligning execution point of duplicate copies of a user program by exchanging information about instructions executed
US20060236168A1 (en) * 2005-04-01 2006-10-19 Honeywell International Inc. System and method for dynamically optimizing performance and reliability of redundant processing systems
US7797394B2 (en) * 2005-04-18 2010-09-14 Dell Products L.P. System and method for processing commands in a storage enclosure
US20060235937A1 (en) * 2005-04-18 2006-10-19 Dell Products L.P. System and method for processing commands in a storage enclosure
US7590885B2 (en) * 2005-04-26 2009-09-15 Hewlett-Packard Development Company, L.P. Method and system of copying memory from a source processor to a target processor by duplicating memory writes
US20060242456A1 (en) * 2005-04-26 2006-10-26 Kondo Thomas J Method and system of copying memory from a source processor to a target processor by duplicating memory writes
US20080155306A1 (en) * 2005-12-21 2008-06-26 Combs William E Method and system for controlling command execution
US7577870B2 (en) * 2005-12-21 2009-08-18 The Boeing Company Method and system for controlling command execution
US20080196037A1 (en) * 2007-02-13 2008-08-14 Thales Process for maintaining execution synchronization between several asynchronous processors working in parallel and in a redundant manner
US8205201B2 (en) * 2007-02-13 2012-06-19 Thales Process for maintaining execution synchronization between several asynchronous processors working in parallel and in a redundant manner
US7850127B2 (en) * 2008-03-11 2010-12-14 Ansaldo Sts Usa, Inc. Cab signal receiver demodulator employing redundant, diverse field programmable gate arrays
US20090230255A1 (en) * 2008-03-11 2009-09-17 Lemonovich John E Cab signal receiver demodulator employing redundant, diverse field programmable gate arrays
US8015390B1 (en) * 2008-03-19 2011-09-06 Rockwell Collins, Inc. Dissimilar processor synchronization in fly-by-wire high integrity computing platforms and displays
US20100281239A1 (en) * 2009-04-29 2010-11-04 Ranganathan Sudhakar Reliable execution using compare and transfer instruction on an smt machine
US8082425B2 (en) 2009-04-29 2011-12-20 Advanced Micro Devices, Inc. Reliable execution using compare and transfer instruction on an SMT machine
US20120304024A1 (en) * 2010-02-16 2012-11-29 Freescale Semiconductor, Inc. Data processing method, data processor and apparatus including a data processor
US9052887B2 (en) * 2010-02-16 2015-06-09 Freescale Semiconductor, Inc. Fault tolerance of data processing steps operating in either a parallel operation mode or a non-synchronous redundant operation mode
US9842014B2 (en) 2012-11-22 2017-12-12 Nxp Usa, Inc. Data processing device, method of execution error detection and integrated circuit
US9823983B2 (en) 2014-09-25 2017-11-21 Nxp Usa, Inc. Electronic fault detection unit
US10063567B2 (en) 2014-11-13 2018-08-28 Virtual Software Systems, Inc. System for cross-host, multi-thread session alignment
US11586514B2 (en) 2018-08-13 2023-02-21 Stratus Technologies Ireland Ltd. High reliability fault tolerant computer architecture
US11281538B2 (en) 2019-07-31 2022-03-22 Stratus Technologies Ireland Ltd. Systems and methods for checkpointing in a fault tolerant system
US11288123B2 (en) 2019-07-31 2022-03-29 Stratus Technologies Ireland Ltd. Systems and methods for applying checkpoints on a secondary computer in parallel with transmission
US11429466B2 (en) 2019-07-31 2022-08-30 Stratus Technologies Ireland Ltd. Operating system-based systems and method of achieving fault tolerance
US11620196B2 (en) 2019-07-31 2023-04-04 Stratus Technologies Ireland Ltd. Computer duplication and configuration management systems and methods
US11641395B2 (en) 2019-07-31 2023-05-02 Stratus Technologies Ireland Ltd. Fault tolerant systems and methods incorporating a minimum checkpoint interval
US11263136B2 (en) 2019-08-02 2022-03-01 Stratus Technologies Ireland Ltd. Fault tolerant systems and methods for cache flush coordination
US20210311522A1 (en) * 2020-04-07 2021-10-07 Airbus Operations Sas Method and system for synchronizing computing units of an aircraft
US11599141B2 (en) * 2020-04-07 2023-03-07 Airbus Operations Sas Computing unit assembly and a method and a system for synchronizing computing units of an aircraft
US11288143B2 (en) 2020-08-26 2022-03-29 Stratus Technologies Ireland Ltd. Real-time fault-tolerant checkpointing

Also Published As

Publication number Publication date
AU2002257131A8 (en) 2008-03-13
AU2002257131A1 (en) 2002-10-28
US20020152418A1 (en) 2002-10-17
WO2002084488A3 (en) 2008-01-31
WO2002084488A2 (en) 2002-10-24

Similar Documents

Publication Publication Date Title
US6928583B2 (en) Apparatus and method for two computing elements in a fault-tolerant server to execute instructions in lockstep
US6971043B2 (en) Apparatus and method for accessing a mass storage device in a fault-tolerant server
US7065672B2 (en) Apparatus and methods for fault-tolerant computing using a switching fabric
US6496940B1 (en) Multiple processor system with standby sparing
US7890797B2 (en) Vehicle including a processor system having fault tolerance
US5005174A (en) Dual zone, fault tolerant computer system with error checking in I/O writes
US5751932A (en) Fail-fast, fail-functional, fault-tolerant multiprocessor system
US5675807A (en) Interrupt message delivery identified by storage location of received interrupt data
US6157967A (en) Method of data communication flow control in a data processing system using busy/ready commands
US5964835A (en) Storage access validation to data messages using partial storage address data indexed entries containing permissible address range validation for message source
JPH03184130A (en) Error processing of software
EP0747817A2 (en) Data communication method in a fail-fast, fail-functional, fault-tolerant multiprocessor system
KR100258080B1 (en) Dual control equipment based on input/output operation
CA2178408A1 (en) Method and apparatus for delivering interrupts in a processing system
JPH0581056A (en) Duplex system for electronic computer
JPH0581055A (en) Duplex system for electronic computer

Legal Events

Date Code Title Description
AS Assignment

Owner name: STRATUS TECHNOLOGIES INTERNATIONAL, S.A.R.L., LUXE

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GRIFFIN, GERRY;MCLOUGHLIN, MICHAEL;REEL/FRAME:011886/0495

Effective date: 20010410

AS Assignment

Owner name: STRATUS TECHNOLOGIES BERMUDA LTD., BERMUDA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:STRATUS TECHNOLOGIES INTERNATIONAL, S.A.R.L.;REEL/FRAME:012918/0543

Effective date: 20020222

AS Assignment

Owner name: JPMORGAN CHASE BANK, AS ADMINISTRATIVE AGENT, TEXA

Free format text: SECURITY AGREEMENT;ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:012919/0001

Effective date: 20020222

AS Assignment

Owner name: JPMORGAN CHASE BANK, AS ADMINISTRATIVE AGENT, TEXA

Free format text: SECURITY INTEREST;ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:014242/0225

Effective date: 20031216

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: STRATUS TECHNOLOGIES BERMUDA, LTD., BERMUDA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:JPMORGAN CHASE BANK, N.A., FORMERLY KNOWN AS JPMORGAN CHASE BANK;REEL/FRAME:017388/0942

Effective date: 20060329

AS Assignment

Owner name: GOLDMAN SACHS CREDIT PARTNERS L.P.,NEW JERSEY

Free format text: PATENT SECURITY AGREEMENT (FIRST LIEN);ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:017400/0738

Effective date: 20060329

Owner name: DEUTSCHE BANK TRUST COMPANY AMERICAS,NEW YORK

Free format text: PATENT SECURITY AGREEMENT (SECOND LIEN);ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:017400/0755

Effective date: 20060329

Owner name: GOLDMAN SACHS CREDIT PARTNERS L.P., NEW JERSEY

Free format text: PATENT SECURITY AGREEMENT (FIRST LIEN);ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:017400/0738

Effective date: 20060329

Owner name: DEUTSCHE BANK TRUST COMPANY AMERICAS, NEW YORK

Free format text: PATENT SECURITY AGREEMENT (SECOND LIEN);ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:017400/0755

Effective date: 20060329

FPAY Fee payment

Year of fee payment: 4

AS Assignment

Owner name: STRATUS TECHNOLOGIES BERMUDA LTD.,BERMUDA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:JPMORGAN CHASE BANK;REEL/FRAME:024185/0931

Effective date: 20100405

AS Assignment

Owner name: JEFFERIES FINANCE LLC, AS ADMINISTRATIVE AGENT,NEW

Free format text: SUPER PRIORITY PATENT SECURITY AGREEMENT;ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:024202/0736

Effective date: 20100408

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., A

Free format text: INDENTURE PATENT SECURITY AGREEMENT;ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:024202/0766

Effective date: 20100408

Owner name: STRATUS TECHNOLOGIES BERMUDA LTD.,BERMUDA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:GOLDMAN SACHS CREDIT PARTNERS L.P.;REEL/FRAME:024213/0375

Effective date: 20100408

Owner name: JEFFERIES FINANCE LLC, AS ADMINISTRATIVE AGENT, NE

Free format text: SUPER PRIORITY PATENT SECURITY AGREEMENT;ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:024202/0736

Effective date: 20100408

Owner name: STRATUS TECHNOLOGIES BERMUDA LTD., BERMUDA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:GOLDMAN SACHS CREDIT PARTNERS L.P.;REEL/FRAME:024213/0375

Effective date: 20100408

FPAY Fee payment

Year of fee payment: 8

AS Assignment

Owner name: STRATUS TECHNOLOGIES BERMUDA LTD., BERMUDA

Free format text: RELEASE OF SUPER PRIORITY PATENT SECURITY AGREEMENT;ASSIGNOR:JEFFERIES FINANCE LLC;REEL/FRAME:032776/0555

Effective date: 20140428

Owner name: STRATUS TECHNOLOGIES BERMUDA LTD., BERMUDA

Free format text: RELEASE OF PATENT SECURITY AGREEMENT (SECOND LIEN);ASSIGNOR:WILMINGTON TRUST NATIONAL ASSOCIATION; SUCCESSOR-IN-INTEREST TO WILMINGTON TRUST FSB AS SUCCESSOR-IN-INTEREST TO DEUTSCHE BANK TRUST COMPANY AMERICAS;REEL/FRAME:032776/0536

Effective date: 20140428

Owner name: SUNTRUST BANK, GEORGIA

Free format text: SECURITY INTEREST;ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:032776/0595

Effective date: 20140428

Owner name: STRATUS TECHNOLOGIES BERMUDA LTD., BERMUDA

Free format text: RELEASE OF INDENTURE PATENT SECURITY AGREEMENT;ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.;REEL/FRAME:032776/0579

Effective date: 20140428

FEPP Fee payment procedure

Free format text: PAT HOLDER CLAIMS SMALL ENTITY STATUS, ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: LTOS); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

FPAY Fee payment

Year of fee payment: 12

AS Assignment

Owner name: STRATUS TECHNOLOGIES IRELAND LTD., IRELAND

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:052210/0411

Effective date: 20200301

AS Assignment

Owner name: TRUST BANK (AS SUCCESSOR BY MERGER TO SUNTRUST BANK), GEORGIA

Free format text: SECURITY INTEREST;ASSIGNOR:STRATUS TECHNOLOGIES IRELAND LIMITED;REEL/FRAME:052316/0371

Effective date: 20200301

AS Assignment

Owner name: CERBERUS BUSINESS FINANCE AGENCY, LLC, AS COLLATERAL AGENT, NEW YORK

Free format text: GRANT OF SECURITY INTEREST IN PATENT RIGHTS;ASSIGNORS:STRATUS TECHNOLOGIES IRELAND LIMITED;STRATUS TECHNOLOGIES BERMUDA LTD.;REEL/FRAME:057254/0557

Effective date: 20210608

AS Assignment

Owner name: STRATUS TECHNOLOGIES BERMUDA LTD., BERMUDA

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (057254/0557);ASSIGNOR:CERBERUS BUSINESS FINANCE AGENCY, LLC;REEL/FRAME:061354/0599

Effective date: 20220829

Owner name: STRATUS TECHNOLOGIES IRELAND LIMITED, IRELAND

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (057254/0557);ASSIGNOR:CERBERUS BUSINESS FINANCE AGENCY, LLC;REEL/FRAME:061354/0599

Effective date: 20220829