WO1997043712A2 - Systeme informatique redondant a trois modules - Google Patents

Systeme informatique redondant a trois modules Download PDF

Info

Publication number
WO1997043712A2
WO1997043712A2 PCT/US1997/008320 US9708320W WO9743712A2 WO 1997043712 A2 WO1997043712 A2 WO 1997043712A2 US 9708320 W US9708320 W US 9708320W WO 9743712 A2 WO9743712 A2 WO 9743712A2
Authority
WO
WIPO (PCT)
Prior art keywords
bus
processor
control device
interrupt
transaction information
Prior art date
Application number
PCT/US1997/008320
Other languages
English (en)
Other versions
WO1997043712A3 (fr
Inventor
James L. Petivan
Donald C. Lundell
Jonathan K. Lundell
Original Assignee
Resilience Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Resilience Corporation filed Critical Resilience Corporation
Priority to EP97926550A priority Critical patent/EP0916119B1/fr
Priority to AT97926550T priority patent/ATE210316T1/de
Priority to AU31288/97A priority patent/AU3128897A/en
Priority to DE69708881T priority patent/DE69708881T2/de
Priority to IL12705997A priority patent/IL127059A0/xx
Priority to JP54114297A priority patent/JP2002515146A/ja
Publication of WO1997043712A2 publication Critical patent/WO1997043712A2/fr
Publication of WO1997043712A3 publication Critical patent/WO1997043712A3/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/18—Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits
    • G06F11/181—Eliminating the failing redundant component
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/18—Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits
    • G06F11/182—Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits based on mutual exchange of the output between redundant processing components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/18—Error detection or correction of the data by redundancy in hardware using passive fault-masking of the redundant circuits
    • G06F11/187—Voting techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/1629—Error detection by comparing the output of redundant processing systems
    • G06F11/165—Error detection by comparing the output of redundant processing systems with continued operation after detection of the error
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing

Definitions

  • the invention relates in general to fault tolerant computer systems, and more particularly, to redundant fault tolerant computer systems.
  • Fault tolerant computer systems employ redundant elements, to guard against system failure. If one redundant element, fails then another element takes over for the failed element.
  • a typical triple modular redundant computer system may employ three processors. If one processor fails, then another of the three processors is available to take over for the failed processor. See Fault Tolerant Computer Svstem Design, by Dhiraj K. Pradhan, Prentice Hall, Inc., 1995.
  • Figure 1 is a generalized block diagram of an exemplary conventional computer system
  • FIG. 2 is a generalized block diagram of a system module in accordance with a presently preferred embodiment of the invention.
  • FIG 3 is a generalized block diagram of a triple modular redundant (TMR) computer system, which includes three system modules of the type illustrated in Figure 2, in accordance with a presently preferred embodiment of the invention;
  • TMR triple modular redundant
  • FIG 4 is a more detailed block diagram of the TMR system of Figure 3 which generally represents the physical layout of the system and which illustrates isolation of processor (CPU) bus and I/O bus on each module;
  • Figure 5 A is a generalized block diagram of a triple redundant system clock employed in the embodiments of Figures 3 and 4;
  • Figure SB is a more detailed block diagram of a triple redundant system clock employed in the embodiment of Figures 3 and 4 showing respective synchronous processors in respective PLL feedback paths;
  • Figure 6 is a generalized block diagram of a comparator logic portion of a bridge logic unit of one of the modules of the embodiments of Figures 3-5;
  • FIG 7 is a more detailed block diagram of a bridge logic unit of one of the modules of the embodiments of Figures 3-5;
  • Figure 8 is an illustrative timing diagram pertaining to a processor
  • Figure 9 is an illustrative timing diagram pertaining to a processor Read operation in a TMR system employing bridge logic units like that in Figure 7;
  • Figure 10 is an illustrative timing diagram pertaining to a processor
  • FIG 11 is an illustrative timing diagram pertaining to a DMA Read operation in a TMR system employing bridge logic units like that in Figure 7;
  • Figure 12 is an illustrative timing diagram pertaining to a DMA Write operation in a TMR system employing bridge logic units like that in Figure 7.
  • a presently preferred embodiment of the invention is based upon the architecture of the Sun Microsystems, Inc., SPARCstation 5 (TM) (SS5), a RISC workstation that runs the Solaris 2 (TM) operating system, a Unix (TM) variant.
  • the SS5 is highly integrated. Referring to the drawing of Figure 1, there is shown a generalized block diagram of a conventional SS5 computer system 10.
  • the computer system includes a processor 12 plus system memory 14 and Input/Output (I/O) components 16 and 18.
  • the processor 12 and the I/O components 16 and 18 communicate over an SBus 20 which may include SBus expansion slots 22. Details of SBus operation are described in, SBus
  • the presently preferred embodiment includes three redundant system modules, each of which essentially implements a SPARCstation 5 computer system.
  • each system board includes a bridge logic unit that allows the three system boards to exchange data, compare signals in order to detect faults, share I/O devices, and connect to a common backplane.
  • FIG. 2 there is shown a generalized block diagram of a system module 46 in accordance with a current implementation of the invention.
  • the system module 46 includes a processor 48 and system memory 50.
  • the module 46 also includes Input/Output (I/O) controllers 54 and a bridge logic unit 52.
  • the processor 48 and bridge logic unit 52 are connected via a bus 56, in this case an SBus.
  • the I/O controllers 54 and bridge logic unit 52 are connected via a bus 57, in this case an SBus.
  • the I/O controllers 54 and bridge logic unit 52 are connected via a bus 57, in this case an SBus.
  • the system module 46 is coupled to an interconnect unit 38 which interconnects three system modules, and which interconnects actual I/O connectors (not shown) to the three system modules.
  • the interconnect unit 38 is implemented as a PC board and may be referred to as a "backplane.”
  • the interconnect unit 38 includes first connections 40 whereby the respective bridge logic unit 52 of each module can broadcast the module's "own” I/O bus transaction information for delivery to other modules.
  • the interconnect 38 unit also includes second connections 42 whereby the respective bridge logic unit of each module can receive bus transaction, information broadcast by the bridge logic unit of neighboring "upstream” and "downstream" modules.
  • the present interconnect unit also has disposed on it the physical I/O connectors (details not shown) which communicate with the I/O controllers on the system modules. It will be appreciated, of course, that the physical location of the I/O connectors and the I/O controllers may be changed without departing from the scope of the invention.
  • FIG. 3 there is shown a block diagram of a triple modular redundant (TMR) computer system 44 in accordance with the presently preferred embodiment of the invention.
  • the presently preferred embodiment includes three system modules 46 A 46B and 46C.
  • Each module includes a processor 48A 48B, 48C, system memory 50A 50B, 50C, bridge logic units 52A, 52B, 52C and I/O controllers 54A, 54B, 54C, and three independent processor buses 56A 56B, 56C and three independent I/O buses 57A 57B and 57C.
  • FIG. 4 there is shown another view of the TMR computer system 44 showing the layout of the current embodiment.
  • Processor (CPU) 48A is coupled to processor bus 56A
  • Processor (CPU) 48B is coupled to processor bus 56B.
  • Processor (CPU) 48C is coupled to processor bus 56C.
  • I/O controller 54A (within dashed lines) is coupled to I/O bus 57A
  • I/O controller 54B (v ⁇ thin dashed lines) is coupled to I/O bus 57B.
  • I/O controller 54C (within dashed lines) is coupled to I/O bus 57C.
  • a bus interface control unit 53 controls the transfer of information between the processor buses 56A 56B and 56C and the I/O buses 57A 57B and 57C.
  • the bus interface control unit 53 ensures that the processors 48A 48B and 48C operate synchronously during info ⁇ nation transfers with the I/O buses. That is, the bus interface control unit ensures that the processors perform the same operations in lock-step during normal operation.
  • the bus interface control unit 53 includes first, second and third bridge logic units 52A 52B and 52C.
  • Processor bus 56A and I/O bus 57A are coupled to bridge logic unit 52A.
  • Processor bus 56B and I/O bus 57B are coupled to bridge logic unit 52B.
  • Processor bus 56C and I/O bus 57C are coupled to bridge logic unit 52C.
  • the interconnect unit 38 interconnects the individual bridge logic units 52 A 52B, 52C disposed on different modules.
  • each module 46A 46B, 46C actually includes two I/O controllers connected to the I/O bus 57A 57B or 57C disposed on that module.
  • these controllers commonly referred to as a "MACIO” 58A, 58B, 58C and a "SLAVIO" 60A 60B, 60C.
  • the MACIO is an STP2000 produced by SunMicro Electronics of Mountain View, California.
  • the MACIO controls SCSI and Ethernet I/O functions.
  • the SLAVIO is an STP2001 available from SunMicro Electronics.
  • the SLAVIO controls Serial I/O, Timers, Interrupts, BootPROM and NVRAM.
  • each respective module also includes expansion slots (not shown) which permit other I/O controllers, such as network interfaces, to be added to the system.
  • the additional I/O controllers in the expansion slots typically are connected directly to I/O devices, such as network routers.
  • Each module has access to the I/O controllers of the other modules. More particularly, information transferred to or from one or more I/O devices directly connected to a module via that module's own I/O controllers is sent to neighboring modules via backplane interconnect circuitry 38. Thus, information transacted with I/O devices is shared among the modules. The shared information is compared, as described below, in order to detect possible system faults.
  • the bridge logic unit of each module compares the information transacted by that module with information transacted by a neighboring module.
  • the unit of comparison is a bus transaction. More specifically, a given block of information that is compared to determine whether there is a fault comprises the data, address and control information transacted during a given bus cycle.
  • the above-described unit of comparison shall be referred to herein as a "transaction". Examples of transactions in accordance with the current embodiment include processor Read, processor Write, DMA Read and DMA Write. Note that different information may be compared for different transactions since different control and data are presented during different transactions.
  • the system of the presently preferred embodiment performs three pairwise comparisons. Each such comparison involves a comparison of transaction information from two (a pair) of the three modules.
  • the bridge logic unit of each module receives transaction information from its upstream neighbor module over backplane interconnects, and compares it to its own bus cycle info ⁇ nation.
  • module A is downstream of module B
  • module B is downstream of module C
  • module C is downstream of module A
  • the bridge logic of each module sends its own bus cycle information downstream over backplane interconnects for similar comparison by the downstream bridge logic.
  • the bridge logic detecting the mismatch signals a fault to all three processors.
  • the three processors all evaluate the fault, as explained below, in an effort to diagnose the problem.
  • the bus cycle comparisons involve three pairwise comparisons.
  • the pairwise comparisons are performed using bridge logic units and backplane interconnects, described below, that operate in parallel with (rather than in the path of) the bus control and bus data/address paths disposed on the different modules. That is, the different bridge logic units and the various backplane interconnects of the interconnect unit used to make comparisons generally are not in the path of the different buses and control lines on the different modules.
  • the fault detection logic does not form part of the actual data path.
  • An advantage of the disposition of the fault detection logic generally outside the modules' actual control and data paths is that no additional delay is imposed on the signal path since there is no voting logic in the signal path. Also, there is no voting logic hardware in the signal path that could fail and possibly corrupt the data.
  • the I/O controllers of the TMR computer system operate independently of each other. They are not synchronized. That is, different operations may be initiated during the same time interval on the three I/O buses. This contrasts with operations on the processor buses which are always identical and synchronized during normal operation.
  • the bridge logic units and the backplane interconnects facilitate fault tolerant operation of the overall system despite the interconnection of synchronous processors and asynchronous I/O controllers and I/O devices. For example, in order to read data from an I/O device during normal synchronous system operation, all three processors synchronously direct a read request to the I/O controller local to the module that controls the target I/O device.
  • the subject I/O controller responds by reading the requested information from the target I/O device via the bus local to the one particular module.
  • the bridge logic unit and the backplane interconnects cooperate to send the read information to the other two modules.
  • the transaction information incident to the read transaction is subject to pairwise comparisons, and the read information is sent to all three processors. If an error is detected, then each of the three processors runs a diagnostic program to evaluate possible causes of the error.
  • all three processors synchronously direct a write request to the I/O controller local to the module that controls the target I/O device.
  • the information to be written is actually provided to the I/O controller interface by the processor local to the target I/O device.
  • the subject I/O controller responds by causing the write info ⁇ nation to be written to the target I/O device.
  • the bridge logic unit and the backplane interconnects cooperate to send the write information provided by each processor to the bridge logic units of the neighboring modules.
  • the transaction information incident to the write transaction is subject to pairwise comparisons. If an error is detected, then each of the three processors runs a diagnostic program to evaluate possible causes of the error.
  • each module includes respective reference clock oscillators 64A 64B, 64C and respective phase-locked loop circuits 66A 66B, 66C together with respective frequency dividers 67 A 67B and 67C.
  • the reference clocks operate at the frequency of the bus clock. For an SBus, a typical reference clock is nominally 16-25 MHz.
  • Each module's reference clock signal is broadcast over the backplane, via lines 38', to the other modules.
  • a phase locked loop circuit PLL
  • SCLK local clock
  • the local clock (SCLK) signals serve as the bus clock (both as processor bus clock and as I/O bus clock) on modules 46A, 46B and 46C.
  • the PLLs are configured to lock the local SCLK signals to the same frequency as the designated oscillator 64 A, 64B or 64 C.
  • the PLLs can be locked to some multiple of the oscillator frequency without departing from the invention.
  • respective processors (CPUs) 48A, 48B and 48C are disposed in respective feedback loops with corresponding PLLs 66A, 66B and 66C.
  • the designated reference oscillator 64A, 64B or 64C provides a 22MHz reference clock signal.
  • the outputs of the respective PLLs 66A 66B and 66C are provided at 55 MHz to the respective processors 48A 48B and 48C.
  • Each respective processor 48A 48B and 48C includes internal circuitry (not shown) which multiplies the received 55 MHz signals to 110 MHz which is the internal processor frequency in the presently preferred embodiment.
  • Internal processor circuitry then divides down the 110 MHz processor clock signal to 22 MHz which is the bus frequency in the present embodiment.
  • Each processor 48 A 48B and 48C provides its respective 22MHz signal as feedback to its corresponding PLL 66A, 66B or 66C.
  • Sychronization of the three PLLs 66A ⁇ 66B and 66C is initially achieved through a synchronizing reset operation.
  • each module initially uses its own local reference oscillator.
  • the modules perform a power- on selftest (POST) routine to ensure that the hardware is functioning properly before synchronizing the modules.
  • the modules employ multiplexer switches 68A 68B, 68C to select a common reference (by convention, the reference clock from module A, B or C in priority order). This common reference is used by all three modules in the phase-locked loop process described above. That is, each of the PLLs lock onto the common reference, and then perform a synchronizing reset, which is described below after which all three modules are running in synchronization with the selected reference clock source.
  • all three should be set to the same initial state, and the inputs to all three modules should be identical (except, of course, in the case of an actual error). Furthermore, it should be possible to replace a faulty module and bring the replacement module into synchronization with the working modules.
  • processors can be initialized programmatically, e.g., registers and cache contents.
  • main memory ordinarily can be initialized programmatically.
  • some processor state is not accessible to the programmer. For example, the memory refresh counter and the free-running translation lookaside buffer (TLB) replacement counter may not be programmatically accessible. Unfortunately, having these counters out of synchronization may lead to processors being out of synchronization.
  • TLB translation lookaside buffer
  • processors such as the Intel Pentium (TM)
  • TM Intel Pentium
  • a processor's JTAG logic can be used to initialize -l i ⁇ the processor as required. Note that resort to JTAG logic is not necessary for Micro SPARC 2 implementation of the present embodiment.
  • processors In addition to starting in the same state, all processors should receive the same inputs. This calls for several synchronizing actions. For example, external interrupts should be synchronized and presented to all processors at the same time. This is accomplished by the bridge logic unit as explained in detail below. DMA requests also should be synchronized. The bridge logic units also accomplish this synchronization, along with other DMA overhead. Processor reads from I/O devices should be stable and deterministic. In order to ensure stability and determinism, relevant signals such as SBus
  • the processor buses 56A 56B and 56C are isolated from their - corresponding I/O buses 57 A 57B and 57C. Consequently, fault tolerance strategies may be applied to the processors 48 A 48B and 48C which are different from fault tolerance strategies applied to the I/O controllers 54 A 54B and 54C and I/O devices. More particularly, in the presently preferred embodiment, the fault tolerance strategy applied to the processors involves synchronous processor operation. Whereas, the fault tolerance strategies applied to the I/O controllers and devices involve asynchronous I/O controller operation and asynchronous I/O device operation. Nevertheless, bridge logic comparison operations are employed to identify possible faults arising from (synchronous) processor malfunction and to help identify possible faults arising from (asynchronous) I/O malfunctions.
  • the interconnect circuitry enables the sharing of fault information among processors on the different modules so that they may individually run diagnostic programs to evaluate the fault. More specifically, the interconnect circuitry interconnects the three modules over the backplane.
  • the current implementation employs an SBus which has a bus transaction with a three clock cycle rninimum.
  • the system of the present embodiment takes advantage of this multiple clock cycle bus transaction to pipeline the transfer of the bus cycle information over the backplane, sharing (multiplexing) the same backplane interconnects for address and data.
  • the interconnect circuitry performs multiple functions. Bridge logic units on each module compare bus transaction information in order to identify possible faults. This comparison is done as three pairwise compares, each module comparing its own transactions information with that of its upstream neighbor. Errors are reported with interrupts and status registers.
  • the interconnect unit also provides access between processors and I/O controllers and devices, and provides access between system memory and I/O controllers and devices.
  • the bridge logic units provide a mechanism for exchanging information between modules without requiring a processor local to one module to write to an I/O controller or device local to another module.
  • any bridge logic unit detects an error (that is, its local transaction information does not match its upstream neighbor's transaction information), it broadcasts an error signal over the backplane to all the bridges logic units. In response, each bridge logic unit latches the transaction information (data, address, control signals) associated with its own transaction as well as that of its upstream neighbor's transaction, and interrupts its local processor.
  • bus transactions can be further serialized to reduce the backplane pin count in future architectures with wider buses, for instance. This can be accomplished, for example, by transmitting the bus cycle information over the backplane at a higher clock rate.
  • the backplane connections could be implemented as high speed fiber optic connections.
  • each respective, bridge logic unit connects directly to its local bus, and separates the bus into two sections: a processor bus section and an I/O device bus section.
  • AS address strobe
  • the AS signal alerts the bridge logic units of the start of a Read/Write transaction.
  • each bridge logic unit sends the relevant bus signals (address, data, and control) over the backplane to its downstream neighbor. Simultaneously, each receives the corresponding information from its upstream neighbor, and compares its own local transaction information to that sent by its upstream neighbor. Likewise, each downstream neighbor bridge logic unit compares its own transaction information against that sent by its upstream neighbor.
  • the module A bridge logic sends a copy of its own transaction information downstream to module C for comparison with the module C transaction information.
  • the module B bridge logic unit sends a copy of its own transaction information downstream to the module A for comparison with the module A transaction information.
  • the module C bridge logic unit sends a copy of its own transaction information downstream to module B for comparison with the module B transaction information.
  • the transaction information need not be reassembled at the destination bridge logic unit in order to perform comparisons since the destination bus logic unit performs the same serialization with its own bus cycle.
  • each bridge logic unit serializes its own transaction information and passes it downstream. Simultaneously, its upstream neighbor does the same.
  • the comparison logic of any given module has its own serialized version of its own module's transaction information which it sends downstream. This serialized version of its own transaction information is compared with the serialized transaction information received from the given module's upstream neighbor. The comparisons, therefore, are performed on the serialized transaction information.
  • the bridge logic unit on each module includes comparator logic for comparing transaction information (data, address and control signals) transacted by its "own" module during a given bus transaction with the transaction information (data, address and control signals) transacted by a neighboring module during the same bus transaction. If the bridge comparator logic on any module detects a mismatch between the transaction information transacted on its own and an upstream neighbor module, then the detecting bridge module stores the bus information (or state) transacted locally with the bus information (or state) transacted by the upstream neighbor module. The detecting bridge logic unit also signals the error to the bridge logic units on the other modules over dedicated backplane connections. The respective bridge logic units interrupt their corresponding processors at the highest priority interrupt level (level 15 in the SPARC architecture). Identical error handler programs in the processors examine the saved transaction information (or state) of the mismatched cycle to diagnose which module is in error.
  • FIG. 6 there is shown a generalized block diagram of the comparator logic and transaction latch portions of a bridge logic unit of one of the three modules of the TMR computer system of the present embodiment.
  • the local I/O Bus (56) is implemented as an SBus which connects a local processor (not shown) to local I/O devices (not shown).
  • the backplane interconnect unit 38 connects the bridge logic units of the three modules.
  • the comparator logic portion of each bridge logic unit includes a transaction latch 70 which captures the state (address, data, control) of each I/O bus transaction. Each bridge logic unit sends the captured transaction information over the backplane 38 to its downstream neighbor module.
  • each bridge logic unit receives transaction information that has been captured by its upstream neighbor module and that has been transferred to such unit over the backplane interconnect unit 38.
  • the comparator logic of each bridge logic unit also includes a transaction comparator 72 which compares the local module's transaction information with the transaction information received from the local module's upstream neighbor module.
  • each bridge logic unit further includes control/status logic 74 which gives software access to the comparator logic portion.
  • control/status logic 74 can be instructed to enable or disable comparisons on each module. For instance, comparisons by a given module can be disabled if the given module is diagnosed as faulty.
  • Errors detected by the transaction comparator 72 of a given bridge logic unit are reported to the control/status logic 74 of the given unit which reports the error over the backplane interconnect unit 38 to the other modules.
  • error reports from the other modules are received by the control/status logic 74 over the backplane interconnect unit 38 as shown.
  • an error detected by any enabled module is reported to the respective control/status logic 74 of the other modules.
  • the control/status logic on each module instructs its transaction latch 70 to "freeze" (retain) transaction info ⁇ nation for the transaction for which an error has been detected.
  • Another function of the bridge logic units is to allow the three processors to share each other's associated I/O devices.
  • corresponding I/O devices on the three modules are addressed at different locations in the bus address space.
  • the three processors may synchronously read an I/O device on module B, for instance.
  • the processor 48B on module B actually carries out the read from the module B I/O device.
  • the results of that read (data and acknowledge, in the case of an SBus) are transferred via the backplane to modules A and C and are delivered to processors 48A, 48B and 48C.
  • the bridge logic units on each module perform comparisons to test for errors described above.
  • processor writes although all three processors synchronously request a write, only the local processor actually writes to its local I/O device. That is, only the processor local to the same module as a target I/O device can actually write to that device.
  • An advantage of this arrangement is that a faulty processor cannot corrupt an I/O device on another module.
  • each module During a DMA (direct memory access) write to system memory, the DMA data is actually captured by a bridge logic unit buffer memory. The bridge logic unit on each module then performs a DMA write to its local system memory. To guard against renegade DMA devices, each module is restricted (by bridge logic) to its own private region of DMA virtual address space. DMA reads do not require such data buffering.
  • DMA address space is logically divided into four quadrants, based on two DMA address bits. Each module is restricted to one of the quadrants. One quadrant ordinarily is unused.
  • the interconnect circuitry permits sharing of I/O information
  • the I/O controllers and I/O devices local to the three modules are advantageously isolated from each other.
  • the I/O devices and the I/O controller 54B are isolated from modules A and C even though transaction information is shared among modules. This sharing of transaction information facilitates e ⁇ or diagnoses and is necessary in order to pass information from the I/O devices to the processors.
  • the failure of an I/O device or an I/O controller associated with (or local to) one module will not corrupt data transacted on the other modules.
  • Interrupts are inherently asynchronous events. Hence synchronization of interrupt operations is necessary before execution of interrupts by the processor buses.
  • the interrupt controller is resident in the SLAVIO I/O controller which sends the encoded interrupt level to its local bridge logic unit.
  • each of the three bridge logic units receives an interrupt request level from its respective interrupt controller.
  • the interrupt request levels of the three modules are exchanged over the backplane, and the highest priority of the requests is presented to all three processors.
  • An interrupt handler computer program determines which I/O device is interrupting and vectors the interrupt appropriately in a manner which forms no part of the present invention and which will be readily understood by those skilled in the art.
  • a faulty module can potentially disable an entire system.
  • faulty bridge logic might continuously broadcast a highest level interrupt (level 15 in the current implementation).
  • the remaining good modules can disable the faulty module. Disabling is achieved by ignoring the faulty interrupt broadcast (by turning off the corresponding "InSync" bit of an internal register in the respective bridge logic units), or by actively shutting down the faulty module.
  • Each module is built to shut itself down upon the command of any two modules.
  • the interrupted processors run a computer software implemented diagnostic procedure to determine the source of the error.
  • the diagnostic procedure ordinarily is run by all three modules in synchronization. However, it will be appreciated that a faulty module may have caused the error and may be no longer in sync with the other two modules.
  • System software receives the interrupt and invokes the diagnostic procedure to interpret the transaction information stored in the respective transaction latches of the respective modules in order to determine which module is in error.
  • the transaction information captured by each module is compared by the bridge logic units on two of the three modules. Depending on which module is in error, the combination of miscompares serve to identify the offending module, according to the following table:
  • the software may instruct the system operator to remove and replace the module with a good module.
  • the software executes a resynchronization procedure to bring the replaced module into synchronization with the modules already in (synchronous) operation.
  • the above described pairwise error detection processes are useful in identifying faults. However, further analysis often is required to understand the nature of the fault and to institute appropriate corrective action. In other words, the result of the pairwise analysis may not point unambiguously to one faulty module.
  • a more detailed diagnostic procedure described below is employed to handle such ambiguities in a current implementation of the invention.
  • the error source location and the error detection location may be on the same module or on different modules. That is, there may be ambiguity as to whether a detected fault is due to a fault on the module that detects the fault or is due to a fault on the module that transferred the transaction information to the module that detects the fault.
  • the nature of these ambiguities and diagnostic processes employed to evaluate them are described below in more detail.
  • error is first identified as to the nature of its source.
  • Elements of an I/O transaction being compared can originate independently from all three modules, or from a single module. For example, when the processors read a register from an I/O device on a specific module, the transaction address has three independent sources (the three processors) that should all be identical. The data that is read, however, originates from a single source, the addressed I/O device.
  • the diagnostic software can determine whether the miscompare originated with one module, all three modules, or some combination of the two.
  • An error is detected by comparison logic on each module that compares I/O transaction information on that module with the corresponding transaction information sent from its upstream neighbor. There are three pairwise comparisons performed, one on each module. The transaction information from each module participates in two of the three comparisons: its own, and that of its downstream neighbor. Unambiguous errors are detected (as described above) by the comparator on the faulty module itself, and by the comparator on its downstream neighbor. The third comparator, located on the upstream neighbor and comparing the two neighbors, will not detect an error.
  • module A For example, suppose that a transaction reads from module A and module A is faulty in such a way that it returns part of the data with a marginal signal level (that is, between the defined high and low levels).
  • the three modules receiving the data can interpret the marginal levels arbitrarily, and the resulting miscompare might point to the wrong module. If module B sees the marginal data as logical 0, and modules A & C see it as logical 1, the resulting compare will superficially identify module B as faulty. But detailed analysis will reveal that the offending data came from a single source, module A and that it is a fault candidate.
  • ambiguous errors for example, is the backplane interconnect unit and bridge logic units.
  • the described hardware cannot distinguish among faulty transmission of a transaction to the backplane, faulty reception from the backplane, or faulty compare logic, for example.
  • Byzantine resilience theory demonstrates that ambiguous e ⁇ ors are inherent in any triple-redundant system, and that they can be dealt with unambiguously only be having a minimum of four modules and meeting various other requirements.
  • the approach taken in the present embodiment is to recognize the possibility of ambiguity, and deal with it in the software diagnostic process rather than in hardware voting logic.
  • the strategy employed by the diagnostic procedure of the current embodiment involves analyzing patterns of miscompares.
  • a single miscompare may not necessarily signify an error.
  • multiple miscompares are evaluated in order to discern a pattern which may suggest which module(s) are suspect.
  • the suspect modules are then "scored", in a manner described below, in order to determine which of the suspect modules is likely to be faulty.
  • the general procedure employed in the current embodiment for diagnosing faults that may be ambiguous is as follows: a. Determine which modules are reporting a miscompare. b. Determine the source of the miscomparing data (all three modules or a single module). c. If the miscompared data originates from all three modules, and two modules report the error, treat the error as unambiguous, and label the suspect module in accordance with the pairwise error analysis described above. d. If the miscompared data originates from all three modules, and only one module reports the error, treat the reporting module as suspect (a compare logic error). If the data source was faulty then we would expect two modules to report compare errors. e.
  • miscompared data originates from all three modules, and all three modules report the error, diagnose a common-mode multiple-module failure and mark all three modules as suspect. Reliable operation is not guaranteed in this case.
  • f If the miscompared data originates from one module, and all three modules report the error, diagnose the originating module as suspect.
  • the miscompared data originates from one module, and only one module reports an error, consider both the originating module and the reporting module as suspect. Note that these may be the same module.
  • h. If the miscompared data originates from one module, and two modules report the error, consider the originating module and the common- error module as suspect. Note that these may be the same module.
  • the common-error module is the module that the above-described pair-wise error analysis suggests to be in error.
  • the originator of the data is not necessarily in error if there is a miscompare.
  • the error could have occu ⁇ ed in the reporting module. i. If exactly one module has been marked as suspect, disable it and instruct the operator to replace it. j. If two or three modules have been marked as suspect, evaluate the probability that each module is faulty.
  • the probability for each module is a combination of the probability of the suspected error mechanism (externally predefined for each error type), modified by whether the module has been suspected but not replaced previously. This evaluation depends upon the nature of the suspected error, forms no part of the present invention and will be readily understood by those skilled in the art, and therefore, need not be described in detail herein.
  • bridge logic unit A which compares its own error free bus cycle information with the erroneous transaction information of bridge logic unit B.
  • Bridge logic unit C which compares its own error free transaction information with that of bridge logic unit A, does not detect an error.
  • Ambiguous errors are also possible. For instance, e ⁇ ors in the bridge logic itself can be ambiguous.
  • bridge logic unit A transmits its own transaction information over the backplane to module C, for example, an error in a bridge logic A transmitter is indistinguishable from a corresponding error in a bridge logic C receiver (or comparator).
  • There are alternative ways to handle ambiguous errors In the case of a permanent bridge logic error, built-in selftest logic in the bridges can be used to determine which module is faulty.
  • Certain faults termed Byzantine faults, require four modules and a different interconnection scheme to be reliably tolerated.
  • An example in the present system is a potential fault when all three modules read from a single source (e.g., an I/O register on one of the modules). The data read from the single source is driven onto the backplane by the target module. If one of the data drivers is faulty, and drives an undefined signal level onto the bus (say between V m V_), the receiving modules will perceive it as some arbitrary (and possibly different) logic value.
  • the three synchronized processors read a register on module C, which exhibits the described fault, and further suppose that modules B and C see the bit as a logic 1, while module A sees it as a logic 0. The resulting error is detected, but since B and C agree on the value, it appears that A is at fault-even though the real fault lies with C's backplane driver.
  • the fault diagnosis software running in the processors of the three modules is aware of the possibility of Byzantine faults, and in cases where such a fault is possible, labels both modules A and C (in the example) as potentially faulty.
  • the recovery strategy is ultimately up to the operator, but one possibility is to replace both A and C while continuing to run with module B. 4.0 I/O Fault Tolerance
  • the I/O fault tolerance strategy is different from the processor fault tolerance strategy.
  • the I/O devices operate asynchronously; different I/O devices may initiate different I/O operations independently of each other.
  • the processors on the other hand, operate synchronously; during normal operation, all processors simultaneously process the same instructions.
  • an I/O fault tolerance strategy generally can be separately tailored to each I/O device or subsystem.
  • the basic architecture of the present invention provides three independent SCSI channels-one on each system module. Several levels of fault tolerance can be implemented depending on application requirements.
  • the SCSI subsystem can be implemented with no fault tolerance configured at all. This is not ordinarily desirable, since a faulty module would render its associated disks unavailable.
  • a disk can be connected to two of the system's SCSI channels by connecting a cable from one module to the disk and thence to a second module, with termination at the modules.
  • one of the modules acts as the SCSI controller
  • the other module on the cable is a standby controller and is not active.
  • the third module's SCSI channel is not used. If the first module fails, its duties as SCSI controller are taken over by the standby module. This configuration protects against module failure, but does not provide any protection against disk drive failure, or against certain kinds of cable failure.
  • two identical drives are connected to two independent SCSI channels.
  • the data on the two drives is duplicated. All writes are directed to both drives. If one drive fails, the mirror drive is available. This configuration protects only against disk errors that can be identified by hard failures or check sum errors.
  • Three-way mirroring is an extension of two-way mirroring, but allows for byte-for-byte data comparison on reads. Three-way mirroring provides full fault tolerance for the disk subsystem at a significant cost in drives.
  • Three-way mirroring exacts a performance penalty, especially when byte-for-byte compares are employed.
  • An advantage of three-way rhirroring over RAID, however, is that drive failure does not cause additional perfo ⁇ nance loss.
  • a third-party RAID box is employed.
  • the RAID box has two controllers, with independent SCSI channels, that are connected to two system modules.
  • the level of data integrity attained is dependent on the kind of RAID employed, but is typically not as high as three- way mirroring. On the other hand, disk capacity efficiency is quite high. Module failures and most controller failures are tolerated.
  • One disadvantage of RAID in general is that a drive failure can cause a significant loss in performance until the failed drive is replaced and rebuilt.
  • the currently prefe ⁇ ed embodiment provides a system with three independent Ethernet ports. These ports can be employed as three conventional ports, with fault tolerance, if any, being provided by existing IP (Internet Protocol) mechanisms. Alternatively, a low-level fault tolerant Ethernet mode is available. In this mode, the three ports are programmed to the same Ethernet address. Only one is active at a time; the other two are inactive standby ports. Together, the one active and two standby ports appear to the system software as a single Ethernet port. If the module containing the active port fails, another port is set active.
  • a daemon process mnning in synchronization on all three processors is configured to ping one or more remote hosts over the active Ethernet channel. If the ping fails, the daemon commands a switch over from the active port to one of the standby ports. More specifically, the daemon logic rotates the active port through all the available modules even if errors are not found. The ping responses are gathered over time and used to develop a figure of merit for each of the three ports. The figure of merit is used to determine the dwell time for each port in the rotation.
  • the presently preferred embodiment runs the Solaris operating system which uses a simple timer to cause an interrupt at a 100 Hz rate; this timer is used for timekeeping, process scheduling and other functions.
  • the system of the present embodiment employs three such timers — one per module. Each of the three timers is programmed to interrupt at the normal rate. Low level software in the timer code synthesizes a virtual timer from the three physical timers in such a way that the failure of any single timer (not interrupting at all, interrupting too fast, or interrupting too slowly) is ignored by the system.
  • the system timer is described in more detail below in Section 5.2.
  • a TMR system achieves fault tolerance through triplication; three identical components are provided as back up.
  • the system therefore, does not depend on the proper operation of any individual component for the overall system to operate properly.
  • the clock poses additional challenges. All three modules employ the same clock, at any given moment, in order to run synchronously. If the clock fails, then the entire system fails.
  • redundant clocks are employed. There can be problems, however, in identifying failure of individual redundant clocks since such failure can be gradual. Moreover, there can be difficulties in selecting an alternative redundant clock in the event of a clock failure.. For instance, a simple three-way voting scheme is generally not applicable to clock signals produced by different oscillators because of difficulty controlling the phase and drift of the output of an oscillator. Although complicated three-way clock voting techniques have been designed, they often introduce a variable amount of skew between the modules because of the skew introduced by the voting logic. This skew can be minimized, and possibly brought to within an acceptable range, but it generally cannot be eliminated.
  • the system of the presently preferred embodiment addresses the problem of clocking a TMR system by providing a clock with "zero" skew between modules, that will keep running independent of a failure on any one module.
  • the processor and I/O buses employ the same bus clock even though different operations may be simultaneously initiated on the synchronous processor buses, on the one hand, and on individual asynchronous I/O buses on the other hand.
  • the bridge logic units coordinate the operation of the synchronous processor buses and the asynchronous I/O buses.
  • each module has a clock source oscillator 64A, 64B, 64C that it drives out onto the backplane via transfer circuitry lines 38'.
  • Each module takes in the clock source from the other two modules.
  • One clock source is designated or elected as the clock source to be used and is selected by multiplexers 68A, 68B, 68C.
  • the manner in which a clock source is elected is implementation dependent and forms no part of the present invention. The principle requirement is that all three modules elect the same clock source.
  • the designated clock source is fed into a PLL 66A, 66B and 66C on each module.
  • the SCLK signal on each module is used as the bus clock for that module.
  • a PLL 66A, 66B and 66C will remain locked at that frequency and phase for a short while even if the elected clock source changes frequency or phase (e.g., if the elected clock source dies).
  • the PLLs 66A, 66B and 66C perform an important role in achieving fault tolerance. If an oscillator (clock) 64 A 64B and 64C fails, the oscillator failure can be readily detected as described below. If the failing oscillator happens to be the oscillator designated as the basis for clocks on all three modules, then another one of the oscillators is designated to replace it.
  • clock comparators In order to detect the failure of an oscillator (clock) 64 A 64B and 64C, clock comparators (not shown), described below, constantly monitor the clock sources from the three modules. In the presently preferred embodiment, there are three comparators, one on each module. Each comparator does a full comparison between the three oscillators. When a comparator detects that one of the clock sources has failed (i.e., its either running too fast or too slow), it issues an interrupt to the processor so that the software can run diagnostics and take the failed module off-line. If the failed clock source 64A, 64B or 64C is the elected clock source, then the hardware logic will also elect a new clock source.
  • the "inertia" of the PLL permitted it to clean up the clock.
  • the system of the present invention advantageously employs this same "inertia” to temporarily maintain clock operation even if the elected clock source fails.
  • the PLL "inertia” refers to the tendency of a PLL circuit to continue to operate at a locked-in frequency for a brief time interval despite removal or alteration of clock source frequency. If the PLL 66 A 66B and 66C on one of the modules 46A, 46B or 46C ever fails then the clock for that particular module fails.
  • the clock comparator (not shown) of the presently preferred embodiment comprises three counters, each one clocked by one of the clock sources from the three modules.
  • the comparator checks to see if the other two counters have at least counted as high as BCNT (bottom count, an implementation dependent constant; this value would typically be TCNT - 2). If they have, then all three clock sources are fine; all three counters are reset; and the counting starts over again. If neither of the other two counters has reached BCNT yet, then the clock source for the counter that reached TCNT is running too fast and is marked as failed. If one of the other counters has reached BCNT but the other one has not, then the clock source for the counter that did not reach BCNT is running too slow and is marked as failed.
  • the comparator can detect frequency differences of about +/-2% between the three clocks, but it will take 100 clock ticks before a failed clock is detected. If TANT is 10 and BCNT is 8, then the comparator can detect frequency differences of about ⁇ 20% between the three clocks, but it will only take 10 clock ticks before a failed clock is detected.
  • TANT and BCNT Multiple counters with different values of TANT and BCNT may be used for each clock source, so that small counters can quickly detect drastic frequency changes, and larger counters can eventually detect subtle frequency differences.
  • Essentially "zero" skew can be achieved because the elected clock source 64A 64B or 64C only passes through a switch (multiplexers 68A 68B or 68C which have essentially “zero” skew) and a PLL 66A, 66B or 66C (which have essentially “zero” skew) to generate the clock for each module.
  • a significant advantage of the architecture of the present embodiment is that it flexibly supports a wide variety of approaches to I/O fault tolerance. This capability is valuable because the nature of I/O demands different approaches for different kinds of I/O (disk, network, etc) and for different applications.
  • timer For instance, in a typical non-fault-tolerant workstation computer system, there is a system timer whose function it is to periodically interrupt the system, generally at 60 or 100 times per second, but sometimes at somewhat different rates.
  • the system software uses these interrupts to perform service procedures such as keeping track of time, and scheduling and allocating the resources of the workstation (task switching, for example).
  • the proper functioning of the timer is critical to correct system operation, and a fault tolerant implementation should address (and tolerate) the possibility of timer failure. 5.2.1 Timer Overview
  • the architecture of the current system implementation because of its triply-symmetrical nature, provides three hardware timers, one on each module.
  • a system timer is provided on each SLAVIO I/O controller.
  • Timer software uses these three hardware timers to synthesize a fault tolerant virtual timer, whose correct operation is guaranteed as long as two of the three hardware timers operate correctly.
  • the third timer can fail in any fashion, including not functioning at all, or interrupting at the wrong rate, either slower or faster than normal.
  • the processors are scheduled to be interrupted periodically by all three of the hardware timers, nominally at 100 Hz in the example implementation so that appropriate service routines can be executed on the processors.
  • all interrupts are broadcast across the backplane and used by all three modules. Interrupts are synchronized and the highest priority interrupt level from the three interrupt controllers is simultaneously passed to each of the three processors.
  • the software mnning in synchronization on the three processors follows the following process, and determines whether to issue a virtual, fault tolerant, timer event, equivalent to the single timer tick of a non- fault-tolerant system. Timer operation is basically as follows.
  • any one timer can exhibit arbitrary behavior, from not interrupting at all to interrupting continuously, without adversely affecting system operation.
  • a faulty timer is identified and can be disabled through conventional procedures, including, as a last resort, disabling the entire module on which it resides.
  • Special reset logic is important in the microSPARC-2 based TMR systems of the presently preferred embodiment because of a peculiarity of the reset logic inside the microSPARC-II (Processor).
  • Clocks are generated inside the Processor by dividing down the Processor master clock. Two of these clocks SCLK (the SBus clock) are selected by the user to be ⁇ 3, ⁇ 4 or ⁇ 5 based on the speed of the Processor, and GCLK (the graphics clock) is fixed at ⁇ 3. The counters used to divide these clocks are never reset.
  • the clock synchronization logic forces SCLK to be synchronous on all three modules, but it cannot do anything with GCLK.
  • the reset logic corrects for the inability of resetting the clock divider counters by taking advantage of two features put into the microSPARC-2 for reset timing reasons. The first is that no matter what state the Processor is in when it receives a reset, it will begin executing its reset logic state machine. The second is that a few clocks after the reset line goes inactive (the exact number of clocks depends on the relationship between GCLK and SCLK at the end of the reset), the Processor disables all clock outputs for four SCLK ticks by freezing the GCLK divider counter with GCLK high, and holding SCLK high without actually freezing the SCLK divider counter (it cannot freeze the SCLK divider counter because it is clocking its reset logic state machine with
  • SCLK SCLK
  • the reset logic is clocked on SCLK which is tightly synchronized on all three modules.
  • the logic issues a reset pulse that goes inactive synchronously on all three modules.
  • the relationship between GCLK and SCLK is an unknown state, but when the Processor eventually disables its clocks, the clocks on all the modules are in the same known state.
  • a second reset pulse is issued synchronously on all three modules.
  • the clock output disable is released synchronously on all the modules when the Processor begins executing its reset logic state machine, and the GCLK divider counter starts counting again, but this time it starts synchronously from a known state on all three modules.
  • the second reset pulse is held long enough to satisfy the reset timing requirements of the system (32 SCLK ticks in the present implementation), then synchronously released on all three modules. All clocks are now in the same state; therefore, the entire system is in a known state and three modules can run synchronously.
  • the bus interface control unit 53 best illustrated in Figure 4 includes three bridge logic units 52A 52B, and 52C, each associated with a different one of the system modules 46A 46B and 46C.
  • Each of the bridge logic units include circuitry: to control the transfer of transaction information among modules; to perform the pairwise comparisons; to temporarily store transaction information for use in diagnostic analysis; and to coordinate communication between the synchronous (during normal operation) processors and the asynchronous I/O controllers and I/O devices, such that the processors operate in lock-step while the I/O devices operate independently.
  • transaction information includes data, address and control signals that pertain to a given transaction such as a BPIO Read, BPIO Write, Processor Read, Processor Write, DMA Read or DMA Write.
  • the transfer of transaction information by any given bridge logic unit involves sending transaction information to a downstream module for comparison; receiving transaction information from an upstream module for comparison; receiving Processor Read data from other modules; and receiving DMA Write transaction information from other modules.
  • the pairwise comparisons involve comparisons of transaction information.
  • the combination of control, address and data signals transacted by module 46A, for any given transaction is compared with the corresponding combination of signals transacted by module 46B.
  • the transaction info ⁇ nation transacted by module 46B is compared with the transaction information transacted by module 46C.
  • the transaction information transacted by module 46C is compared with the transaction information transacted by module 46A.
  • each processor module 46A, 46B and 46C includes both a processor bus 56A 56B and 56C and an I/O bus 57A, 57B and 57C.
  • the bridge logic units 52A, 52B and 52C operate as bus interface units between respective processor buses and I/O buses.
  • the bridge logic units 52A 52B and 52C coordinate the communication of information between the processors 48A, 48B and 48C, coupled to the processor buses 56A 56B and 56C, and the I/O controllers 54A, 54B and 54C and I/O devices (not shown) coupled to the I/O buses 57A, 57B and 57C.
  • bridge logic unit 52A operates as a bus interface unit between processor bus 56A and I/O buses 57A, 57B and 57C.
  • Bridge logic unit 52B operates as a bus interface unit between processor 56B and I/O buses 57A, 57B and 57C.
  • Bridge logic unit 52C operates as a bus interface unit between processor 56C and I/O buses 57A, 57B and 57C.
  • the bridge logic units 52A, 52B and 52C include transfer circuitry 38 which transfers transaction information between the system modules 46A 46B and 46C for comparison.
  • first transaction information data, address and control
  • second transaction information involved in transfers across
  • the second processor bus 56B is transferred to the first bridge logic unit 52A for corresponding first transaction information.
  • third transaction information (data, address and control) involved in transfers across the third processor bus 56C is transferred to the second bridge logic unit 52B for comparison with the second transaction information.
  • the generalized block diagram of Figure 6 illustrates some of the major components of a representative one of the bridge logic units 52 A which comprises: a comparator 72, transaction latch 70 and control and status logic 74.
  • the illustrative block diagram of Figure 7 shows further details of these components and shows details of the data, address and control flow paths of the representative bridge logic unit 52A of the presently preferred embodiment.
  • the bridge logic units are an integrated component of the backplane interconnect unit.
  • control and status block 74 receives processor- originated control and address signals as input from the processor bus 56A and also receives I/O controller/device-originated control and address signals as input from the I/O bus 57 A.
  • the control and status block 74 outputs processor-directed control signals as output to the processor bus 56A and also outputs I/O controller/device-directed control signals as output to the I/O bus 57A and also provides bridge logic-directed control signals as output to the transaction latch 70 and to the transaction comparator 72.
  • the transaction latches (only the latch 70 of one module shown) temporarily store the transaction information so that it can be provided to corresponding processors for diagnostic analysis in the event that the pair-wise comparison process detects a possible error.
  • the transaction latches also perform an important role in isolating the independently operating I/O buses from the synchronously operating processor buses during certain bus transactions. More specifically, transaction data and transaction address information may be temporarily stored in transaction latches as control and status circuitry in each bridge logic unit coordinates the interaction of the processor and I/O buses. Thus, the transaction latches on each bridge logic unit in effect comprise a portion of the bus signal path between corresponding processor buses and I/O buses during certain bus transactions.
  • an I/O device local to one module requests a write to processor memory.
  • the I/O device initiating the DMA write is local to only one of the three modules.
  • the transaction latch temporarily holds the data to be written to the processors while the write data is transferred to the """" bridge logic units on the other modules.
  • the data to be written by the initiating I/O device has been received by all three bridge logic units, then it can be synchronously provided to the three processors.
  • a FIFO is employed to temporarily store data during DMA write operations.
  • the data portion of the local processor bus 56 A is labelled "PSD”.
  • PSD data portion of the local I/O bus 57
  • Address signals are provided via the "SA" bus lines.
  • SA SA bus lines.
  • SA bus lines There is only one SA bus on the local bridge logic unit 52 A and it is common to both the local processor bus 56A and to the local I/O bus 57A
  • processors 48A, 48B and 48C operate synchronously. That is, the same operations are performed on all three processor buses at all times during normal operation. Data provided by the respective processors 48A 48B and
  • PSD in this manner, it can be shorted (or connected) to ISD via switch 96. In this manner the data on PSD can enter the bus multiplexer 86 via ISD.
  • the I/O buses of the three modules are isolated from each other and from their corresponding processor buses and operate asynchronously (independently).
  • ISD is connected directly to the bridge logic and PSD can be selectively connected via a switch 96.
  • switch 96 is open, isolating PSD and ISD.
  • the switch 96 is closed, connecting PSD and ISD.
  • a backplane (BP) muliplexer 80 receives digital information from upstream and downstream modules; selects between information from an upstream module and information from a downstream module; and provides the selected information to the backplane/own data multiplexer 82 and to the up/own address multiplexer 84.
  • An up/down selection control signal UDISEL (1) controls the BP multiplexer 80 selection of upstream or downstream information. It will be appreciated that the information received by the BP multiplexer 80 may be comprised of data information or address information transferred from an upstream module or from a downstream module.
  • a bus multiplexer 86 receives data signals (ISD) and address signals (SA) from the bus. As explained above, there is only one S A bus, and it is common to both the processor and I/O buses. It is driven by the local processor 48 A
  • the multiplexer selects between the ISD and S A signals; and provides the selected signals to the BP/own data multiplexer 82, to the "own" (or local) data storage unit 83 and to the up/own address multiplexer 84.
  • the BP/own data multiplexer 82 selects between data signals provided over the backplane via BP multiplexer 80 and data signals provided via the bus multiplexer 86 by the module's "own" (or local) I/O bus.
  • the "own” data storage unit 83 stores data provided via the bus multiplexer by the module's own (local) bus. It will be appreciated that the "own” data storage unit 83 is part of the transaction latch 70 of Figure 6.
  • the up/own address multiplexer 84 selects between address signals provided over the backplane via BP multiplexer 80 and address signals provided via the bus multiplexer 86 by the module's own (or local) SA bus. The selections by the bus multiplexer 86 are controlled by the ISASEL (2) signal and by the ISDSEL (3) signal.
  • the own (or local) address storage unit 88 receives input from the module's own (or local) bus address SA lines.
  • the BPIO register 90 receives input from the ISD lines.
  • Signal selection by the BP/own data multiplexer 82 is controlled by the OBPSEL (5) control signal.
  • Signal selection by the up/own address multiplexer 84 is controlled by the ISASELP1 (4) control signal.
  • the BPIO registers can be used to temporarily store data and to pass such stored data between the modules.
  • the BPIO registers are not a part of the transaction latch, and they are only written to when software controlled processes write to them. Such software controlled processes, for example, might be used during diagnostic operations.
  • the own data storage unit 83 is part of the bridge pipeline. It holds data before it goes into the data compare 72-1. It loads data from the bus multiplexer 86 automatically. Address information passes through unit 83 but does not get used.
  • Storage unit 88 is used to store the address and feed to the address comparator 72-2.
  • the transaction latch 70 of Figure 6 includes three storage regions illustrated in Figure 7.
  • a transaction data error storage 70-1 receives input from the own data storage 83. This latch stores the data information when there is a miscompare (i.e., error).
  • the address error transaction storage 70-2 receives input from the own address storage 88. This latch stores address information when there is a miscompare.
  • a control error storage 70-3 receives input from the processor bus control lines. This latch stores control information when there is a miscompare. It will be appreciated that, even though it is the ISD that is directly connected to the bridge logic unit, processor bus signals are compared since they are the ones that are synchronized. Accordingly, the processor control lines are connected to the bridge logic units so that processor control signals can be compared.
  • the data information in storage 70-1, the address information in storage 70-2 and the control information in storage 70-3 collectively comprise transaction information which can be employed to diagnose error conditions.
  • the transaction comparison logic 72 of Figure 6 performs comparisons on three signals.
  • the comparison logic 72 of Figure 6 comprises three units: data compare 72-1, address compare 72-2 and control compare 72-3.
  • the data compare unit 72-1 compares data signals provide by data multiplexer 82 with data signals provided by the data storage unit 83. Note that multiplexer 82 serves two purposes: to get backplane data to 72-1; and to get backplane or own data to the DMA FIFO 92. When data is going into FIFO 92 it is not compared by 72-1.
  • the data compare is controlled by DVALDD (8).
  • An address compare unit 72-2 compares address signals provided by the address multiplexer 84 with address signals provided by the address storage 88.
  • address storage unit 88 stores the address at the start of the transaction at which point it is compared to the upstream address. After this first comparison, multiplexer 84 selects the own (local) address instead of the upstream address. During a typical SBus transaction the address should not change. If it does change, then there is a failure. By comparing the stored copy of the own address at the start of the transaction (88) to the current address on the local bus (SA) a change in the address provided on a module can be detected without having to constantly drive the address out overthe backplane. This permits multiplexing of address and data thereby greatly reducing the number of backplane interconnects needed. The address compare is controlled by AVALID (9).
  • a control compare unit 72-3 compares control signals provided by an upstream module via the backplane interconnect unit with control signals provided by the processor bus control signals produced on the local processor bus.
  • a DMA FIFO 92 receives input from the BP/own data multiplexer 82.
  • the role of the FIFO 92 can be explained briefly as follows.
  • data provided by a single I/O device is broadcast across the backplane 38 to the respective bridge logic unit on each module and is stored temporarily in the respective DMA FIFOs 92 (only one shown) on the different bridge logic units before being sent to the respective processors.
  • This intermediate storage in FIFOs is employed in the cu ⁇ ent embodiment, in part, because of the manner in which acknowledge signals are employed in the SBus protocol.
  • a "master" supplying data on a DMA transfer must be able to constantly supply the data, word after word, on each consecutive clock tick.
  • the bridge logic unit is the bus master on the processor bus.
  • the data is first be stored up in a FIFO and then driven onto the processor bus.
  • Another reason for the use of a FIFO in the current embodiment is the manner in which backplane transaction data broadcast between the modules is multiplexed with transaction information transferred to a downstream neighbor for pairwise comparisons. It will be appreciated that in the presently preferred embodiment, broadcast data and comparison data cannot be provided on the backplane at the same time. Consequently, data cannot be broadcast from an I/O device while data is being sent to the processor.
  • a DMA write is broken into two cycles: one that broadcasts the data, and the other that transfers the data to the processor and to the downstream neighbor for comparison.
  • data is loaded into the FIFO in the first cycle, and then is subsequently transferred out of the FIFO to the processor in the second cycle.
  • Signal selection circuitry 94 receives input from the DMA FIFO 92, the data comparison logic 70-1, 70-2, 70-3 and from the error storage 72-1, 72-2, 72-3.
  • the signal selection circuitry selects which signals to provide from the bridge logic unit to the ISD/PSD.
  • PSD can be shorted (connected) to ISD via switch 96.
  • switch 96 When PSD and ⁇ SD are connected, the data passes from the bridge logic unit to PSD via the switch 96 and ISD. This is the case for a DMA write (i.e., when the FIFO is used), for example.
  • switch 96 (and switch 98) contribute to the isolation of the local processor data bus (PSD) from the local I/O data bus (ISD).
  • Switch 96 operates under control of SWDCLS (6).
  • Switch 98 operates under control of SWACLS (7).
  • the control and status block 74 receives as input local processor control and address signals and receives as input I/O device control and address signals.
  • the control and status block 74 provides as output processor control signals, I/O device control signals and bridge logic unit control signals.
  • the control and address signals should be the same on the three modules because the processors are synchronous.
  • the I/O signals are isolated and independent. Therefore, block 74 receives the I/O signals from each module, not just the local one.
  • control and status block 74 A more detailed description of the control signals input to and output from the control and status block 74 is provided in the following charts. Subsequent sections of this application provide a more detailed explanation of the operation of the bridge logic unit during BPIO Read/Write, processor Read/Write and DMA Read/Write.
  • CPUW Information is only received on UDBP for comparison purposes; therefore, the upstream neighbor is always selected. Initially the address is driven onto the backplane by all the Bridges for comparison, and is received by each Bridge from its upstream neighbor. Next, the write data from the CPU is continuously driven onto the backplane by all the Bridges for comparison, and is received by each Bridge from its upstream neighbor.
  • DMA R Initially, the virtual address is broadcast across the backplane by the Bridge local to the I/O device that is doing the DMA. This information is received via UDBP from that module, by its upstream and downstream neighbor. Next, the read data is output by the CPU on each module and driven onto the backplane for comparison. Data is received on each module from its upstream neighbor.
  • the virtual address is broadcast across the backplane by the Bridge local to the I/O device that is doing the DMA. This information is received via UDBP from that module, by its upstream and downstream neighbor.
  • the write data is output by the I/O device and broadcast across the backplane. Likewise, this data is received via UDBP from that module.
  • the virtual address followed by the data is transferred to the CPU and driven onto the backplane for comparison. Each item is received from the upstream neighbor on UDBP for comparison.
  • ISASEL ISASEL, SBus mux
  • ISDSEL ISDSEL drives the data onto the backplane.
  • they are the inverse of each other except that in our implementation, there is a turn around period in which neither address nor data is driven out when switching between address and data.
  • CPU R/W ISASEL is only active for one clock tick at the start of the transaction. Then ISDSEL is active.
  • the state of the transaction information (data, address and control) from the "own" and upstream modules are latched inside the bridge logic unit of each module.
  • System software can then perform diagnostics on this information (and can run additional tests to get more information) to determine which module is at fault. After the software determines which module is probably at fault, it may take that module off-line so that the user can replace it with a new module. Registers inside the bridge logic units are set by the software to take a faulty board off-line.
  • a method and mechanism is provided to exchange data between modules using the bridge logic units.
  • This mechanism is to give all three processors synchronized access to corresponding I/O data that can differ in value.
  • the implementation of this capability uses the respective backplane Input/Output (BPIO) registers in the bridge logic units of each module.
  • BPIO backplane Input/Output
  • Each processor reads a corresponding I/O register or DRAM location on its own module. 3. Each processor writes the data to its local BPIO register.
  • Each processor reads module A's BPIO register; the module A bridge logic unit facilitates this by sending its BPIO register contents over the backplane.
  • Each processor likewise reads module B's BPIO register.
  • Each processor likewise reads module C's BPIO register.
  • Each processor now has identical copies of the three values from the three corresponding BPIO registers. Ifa particular I/O device is faulty, only its local processor is exposed to the fault during the read in step 2 above.
  • the processor drives out PSAS*, PSSLV*[0], SA[27:0] and the SBus master control signal (PSRD and PSSIZ[2:0]).
  • SA[27:0J, PSRD, and PSSIZ[2:0] out onto the backplane for comparison.
  • Bridge control signal ISASEL enables the address onto the backplane, and ISDSEL enables the data onto the backplane.
  • the target of the transaction is inside of the bridge logic unit; therefore, there is no need to drive out any SBus salve select line. If the I/O bus is busy, the bridge will issue a retry acknowledge to the processor, and the processor will try the transaction again.
  • the timing for accesses to registers inside the bridge logic unit is predetermined, it is hardwired into the bridge and is the same for each module; therefore, the target of the transaction (i.e., the bridge logic) does not have to issue ISACK*[2:0] for the bridge logic to know when the data is available. In the current implementation, the data is available immediately, and it is broadcast across the backplane to the bridge logic units on the other modules.
  • the bridge logic unit on each module then issues an acknowledge to its processor, then passes out the data from the targeted module to its processor.
  • the bridge logic unit of each module then sends the data that went to its processor across the backplane to its downstream neighbor for comparison. This data passes though the data comparison pipeline as indicated by the bridge control signal DVALID. Likewise, valid address information for comparison is indicated by AVALID. Since the comparators are not directly in the path of the signal they are comparing, it does not matter how long it takes before the comparisons actually take place; the comparisons do not introduce delay on the I/O bus (as normal three-way voting logic could.)
  • the following example involves a write to the own module with data comparisons disabled.
  • the processor drives out PSAS*, PSSLV*[0], SA [27:0], PSD [31 :0] and the SBus master control signals (PSRD and PSSIZ [2:0]).
  • PSRD PSRD
  • PSSIZ PSSIZ [2:0] out onto the backplane for comparison.
  • the target of the transaction is inside of the bridge logic unit; therefore, there is no need to drive out any SBus slave select line. If the I/O bus is busy, the bridge logic unit will issue a retry acknowledge to the processor, and the processor will try the transaction again.
  • the timing for the accesses to registers inside the bridge logic unit is predete ⁇ nined, it is hardwired into the bridge and is the same for each module; therefore, the target of the transaction (i.e., the bridge logic) does not have to issue ISACK* [2:0] for the bridge logic unit to know when the data has been written. In the current implementation, it takes two ticks of SCLK to write the date, the bridge logic unit on each module then issues an acknowledge to its Processor. 4.
  • the bridge logic unit of each module constantly drives out the data from its processor across the backplane to its downstream neighbor for comparison. This data passes through the data comparison pipeline of the bridge logic unit and is compared. (In the timing diagram of the Figure 8, data comparisons are actually disabled.)
  • the corresponding signals ISSLV*[n] go from the bridge logic unit to the I/O devices.
  • PSSLV*[0] is for the registers (e.g., BPIO) inside the bridge logic unit (there is no ISSLV*[0] since it is internal to the bridge logic unit).
  • Each target has a unique select line. These select lines are basically an extension of the address bits.
  • SWDCLS* Internal bridge logic unit control signal that controls when the switch between ISD [31 :0] and PSD [31 :0] is open.
  • the switch is open from the start of a DMA transaction (i.e., when an ISBG*[n] is driven active by the bridge logic) until
  • PSBG* is driven active by the processor.
  • the timing diagram of Figure 8 shows a simultaneous processor read from a local register on each module. The results of this read can be different for each module; therefore, it is done with comparisons disabled (i.e., DVALID never goes active).
  • I/O bus transactions can be originated by the processor or by an I/O device (DMA); they can be reads or writes; and processor-originated transactions can be directed to corresponding local-module I/O devices, or to a single I/O device on a specified module. Each of these transactions is described below.
  • the processor on each module reads or writes a corresponding I/O device registered on its local module— all in parallel, all in sync.
  • the bridge logic unit performs a comparison only, with no inter-module I/O data transfers required.
  • S A[31 :0] The address is driven by each CPU and received by its Bridge and I/O devices. One clock after PSAS goes active,
  • SA is driven out onto the backplane and is received by each module's downstream neighbor as UDBP (this is controlled by
  • the local SA that was compared to the upstream neighbor's SA is also latched inside the Bridge.
  • the latched SA is compared to the current SA.
  • PSSLV[4:0] The slave select lines from the CPU are compared in a similar fashion to SA. Basically, they are like additional address bits.
  • PSAS Driven by each CPU and received by its Bridge. Always compared.
  • PSBG Driven by each CPU and received by its Bridge in response to a bus request, PSBR, from the Bridge. Always compared, but should never go active during a CPU initiated transaction.
  • PSSIZ[2:0] Similar to PSRD.
  • processors may simultaneously access corresponding I/O devices local to their own modules.
  • a processor local to module A may access an I/O device local to module A
  • a processor local to module B may access an I/O device local to module B
  • a processor local to module C may access an I/O device local to module C.
  • These local access can be used to pass data between modules via BPIO registers and the backplane connections.
  • each processor can read its local I/O device and temporarily store the result in its local BPIO register. The processors can then read each others' BPIO registers in order to share the data.
  • the respective bridge logic on each module decodes the address signal issued by the processor on that module in conjunction with the read request.
  • the bridge logic on the target module performs the actual read of the addressed I/O device, and sends the result (acknowledgment and data) of the read over the backplane to the respective bridge logic on the other respective modules.
  • the respective bridge logic on each of the respective modules passes the (identical) data and acknowledgment signal to their respective processors.
  • the respective bridge logic of neighboring modules compare control and data signals produced and/or transferred between neighboring processors and the target I/O device.
  • FIG. 9 there is shown a timing diagram for an example of a processor read from the perspective of a processor on a different module than the target I/O device.
  • the processor drives out SBus address signals, SA[27:0] and slave select signals PSSLV[4:].
  • the processor drives out processor SBus read/write, PSRD, and processor address strobe, PSAS, and processor SBus size signals, PSSIZ[2:0]. It will be appreciated that, since in normal operation the three processors operate in synchronism, all three processors drive out these signals on their respective modules.
  • the higher order address bits are decoded by the bridge logic units in order to ascertain which module is local to the target I/O device or location.
  • the processors also send slave select signals directed to a particular I/O devices.
  • the bridge logic will drive SA[27:0], PSRD, and PSSIZ[2:0] out onto the backplane for comparison. That is, the bridge logic on each module will drive these signals to a neighboring downstream module for comparison.
  • address signals and data signals are multiplexed. Specifically, address signals are only driven out once at the start of a processor initiated read transaction. After that, data signals are presented on the same lines. However, if more dedicated signal lines were provided, such multiplexing might not be necessary.
  • bridge logic internal control signal I/O SBus address select, ISASEL
  • I/O SBus data select ISDSEL
  • the bridge logic on the module that has the target I/O device also drives out a slave select control signal for the target device identified in SA[27:0] and PSSLV[4:0].
  • control signals are continuously driven onto the backplane to the downstream neighbor. However, the signals are compared only when they are supposed to be valid in accordance with the SBus protocol.
  • the up/down backplane, UDBP[31:0], signal represents the above described passage of address information from the upstream module via the backplane at about clock tick T5.
  • the UDBP[31:0] signal also represents the above described passage of data from the upstream or downstream module (whichever is local to the I/O device from which the data is read), via the backplane, at about clock tick T14. Note that the UDBP[31 :0] signal also represents the passage of data from the upstream module for comparison at about clock tick T17.
  • the respective bridge logic units will issue a retry acknowledge on the PSACK[2:0] lines to their respective processors, and the processors, operating in synchronism, will try the transaction again.
  • the target I/O device will issue an I/O device acknowledge signal, ISACK, followed by the actual read data.
  • ISACK I/O device acknowledge signal
  • the time at which ISACK is driven is not predetermined but rather depends upon the I/O device.
  • ISACK does not change in the illustrated timing diagram because ISACK is driven on another module local to the target I/O device.
  • the exemplary timing diagrams are.for a module that is not local to the target I/O device.
  • the bridge logic unit of the module local to target I/O device broadcasts its I/O SBus acknowledge signal to the other modules which receive it from across the backplane as their upstream/downstream I/O device acknowledge signal, UDIS ACK, followed by the data across the backplane to the bridge logic on the other modules.
  • UDBP[31 :0] has the data output from the target I/O device.
  • the data valid signal, D VALID indicates when the data coming from the UDBP[31 :0] and the I/O SBus data, ISD, have made it through the comparison logic's comparison stage of the pipeline. It will be appreciated that the ISD data is the local data, and the UDBP data is the data local to the upstream module. These data should be identical if there are no faults. If there are differences between these data then there may be a fault in the system. The comparison logic will detect the possible fault.
  • the bridge logic on each module passes a processor SBus acknowledge signal, PSACK, followed at T16 by the delivery of the data, PSD, (read from the target device) to its respective processor.
  • PSACK processor SBus acknowledge signal
  • the AVALID signal is issued synchronously by the bridge logic unit on each module.
  • a DVALID signal is issued synchronously by the bridge logic unit on each module based on the ISACK signal of the target I/O device.
  • the AVALID and the DVALED signals respectively indicate that address and data signals have been transfe ⁇ ed over the backplane and are ready for comparison by the comparison logic.
  • the I/O device SBus address strobe is driven by the bridge logic unit local to the target device to address the target I/O device.
  • the ISRD signal is driven by each of the bridge logic units to enable the transfer of data to be read.
  • the UTDATSEL signal indicates which module is the source of data during a Read transaction.
  • the bridge logic of each module issues an ISDSEL signal which causes the downstream transfer of the data that was sent to its respective processor. This downstream transfer takes place across the backplane. The transfe ⁇ ed data is sent to its downstream neighbor for comparison. This data passes through the data comparison pipeline of the bridge logic and is eventually compared.
  • Valid/invalid data in the data comparison pipeline is indicated by the logical state of bridge control signal DVALID which is used by the bridge logic unit.
  • valid/invalid address info ⁇ nation in the comparison pipeline is indicated by the logical state of AVALID which is used by the bridge logic. Since the comparators are not directly in the path of the signals they are comparing, the time required to perform the comparisons is not critical. Thus, the comparisons do not introduce delay on the I/O bus as in certain conventional three-way voting logic.
  • each given module compares the following control, address and data signals transacted directly with such module to the corresponding control, address and data signals transacted directly with such module's upstream neighbor.
  • the upstream neighbor data and address signals are transferred downstream (from such upstream module to such given module) as part of the UDBP signals described above.
  • Upstream neighbor control signals are transferred downstream on dedicated lines.
  • the compared signals are: PSBG, PSAS, SA PSSLV, PSRD, PSSIZ, PSACK and PSD.
  • ISD Data is driven by each CPU and received by its Bridge via PSD. It is passed on through the Bridge to the I/O device via ISD. (In a presently preferred embodiment this is done by closing a switch, controlled by SWDCLS, that shorts together PSD and ISD.) The data is compared when ISACK is issued by the I/O device (i.e., when the data is latched into the I/O device according to SBus protocol).
  • the respective processors on each of the modules may synchronously request a write to a target I/O device local to only one of the multiple modules. Although all three processors transact all of the signals involved in the write operation, only the processor on the module local to the target I/O device actually performs the write. Thus, while each bridge logic unit decodes the I/O device address received from its local processor, only the bridge logic unit local to the target I/O device transmits the slave select and I/O device Sbus address strobe, ISAS, to the target I/O device.
  • the bridge logic unit on the target module performs the actual write to the addressed I/O device and sends the result of the write (acknowledgment) over the backplane to the bridge logic units on the other modules.
  • the respective bridge logic units of neighboring modules compare control, data and address signals produced and/or transferred between processors and the target I/O device. Referring to the illustrative drawing of Figure 10, there is shown a timing diagram for an example of a write to a target I/O device from the perspective of a respective bridge logic unit disposed on the same module as the target JIO device. Note that, in the illustrated example, data comparisons are disabled because of the addressing mode of the Write transaction.
  • the processor drives out, SBus address, SA[27:0], processor data, PSD[31:0], and processor SBus address strobe, PSAS.
  • the processor also drives out processor SBus master control signals, PSRD and PSSIZ[2:0]. It will be appreciated that each of the processors drives out these same signals provided that the three processors are in synchronism in a normal operational mode.
  • the bridge logic unit issues ISASEL which causes SA[27:0], PSRD, and PSSIZ[2:0] to be driven out onto the backplane for delivery to the downstream module for comparison.
  • the bridge logic unit local to the module that is local to the target I/O device also drives out a slave select signal (not shown) at approximately T2 or T4 depending on the target I/O device.
  • the bridge logic unit drives ISASEL active for one clock cycle to enable the address onto the backplane for comparison. It then drives ISDEL active to enable data onto the backplane for comparison. ISDEL is driven until an acknowledge is received from the target I/O device If the I/O bus is busy, the bridge logic will issue a retry acknowledge on PSACK[2:0] (not shown) to the processor, and the processor will try the transaction again.
  • the target I/O device issues an acknowledge signal.
  • the bridge logic unit local to the target I/O device broadcasts this acknowledge across the backplane to the bridge logic on the other modules.
  • the upstream/downstream slave control signal select, UTSLVSEL, on the module(s) receiving the acknowledge selects whether to use the acknowledge from the upstream or the downstream module.
  • the bridge logic unit on each module then passes the acknowledge to its respective processor.
  • the target I/O device issues an ISACK when it has written the data.
  • This acknowledge signal is broadcast over the backplane, received by each bridge logic unit, and is transferred to the processors.
  • the bridge logic unit of each module drives out the data from its processor across the backplane to its downstream neighbor for comparison. This data passes through the data comparison pipeline of the bridge logic and is eventually compared at approximately clock tick T10, about three ticks after
  • ISACK ISACK. Note that in the present embodiment data is not actually compared until an acknowledge is received from the target I/O device because it is only at the receipt of the acknowledge that the target I/O device latches in the data. Note that in the illustrative timing diagram of Figure 10, data comparisons are actually disabled. In certain cases, computer software simultaneously reads data from different modules when the data read from the different modules is known to be different. In these special cases, the comparisons typically are disabled.
  • the AVALID signal is an internal bridge logic unit control signal which indicates when the address signals are valid and ready for comparison.
  • the DVALID signal is an internal bridge logic unit control signal which indicates when the data are valid and ready for comparison.
  • the UDIS ACK signal is the ISACK signal transmitted across the backplane by the (upstream or downstream) module local to the target I/O module.
  • ISRD is the I/O device SBus read signal generated by the bridge logic unit based upon PSRD.
  • ISAS is the I/O SBus address strobe signal driven only by the bridge logic unit local to the target I/O device. 6.8.1 Comparisons during Processor write to a module- specific I/O device
  • each given module compares the following control, address and data signals transacted directly with such module to the corresponding control, address and data signals transacted directly with such module's upstream neighbor.
  • the upstream neighbor signals are transferred downstream (from such upstream module to such given module) as part of the UDBP signals described above.
  • the compared signals are: PSBG, PSAS, SA PSSLV, PSRD, PSSIZ, PSACK and PSD.
  • I/O device reads memory via DMA
  • ISD Data is driven by each CPU and received by its Bridge via PSD. It is passed on through the Bridge to the I/O device via ISD. (In a presently preferred embodiment this is done by closing a switch, controlled by SWDCLS, that shorts together PSD and ISD.) PSACK from the CPU is also transferred through the Bridge to ISACK of the I/O devices. (In a presently preferred embodiment this is done by closing a switch, controlled by SWACLS, that shorts together PSACK and ISACK. It is only closed for a DMA read.) The data is compared one clock after PSACK is issued by the CPU (i.e., when the data is latched into the I/O device according to SBus protocol).
  • PSRD The SBus read/write signal is driven by the I/O device and received by each Bridge via ISRD on the local module and UDISRD on the other modules. After each Bridge receives PSBG from its CPU, it drives out the read/write signal to its CPU via PSRD. Compared when PSBG is active and when PSAS is active if it happens to go active during a DMA read.
  • PSACK Driven by each CPU and received by its Bridge. Passed to the I/O device by its Bridge via ISACK Always compared.
  • an I/O device e.g. MACIO
  • DMA direct memory access
  • a requesting I/O device may issue a Bus Request signal to all of the bridge logic units.
  • the local bridge logic responds with Bus Grant, and accepts the DMA address from the I/O device. Note that although an I/O device local to one module issues the request, that request is passed to all of the bridge logic units and eventually to all of the processors acting in synchronism. More particularly, after issuing a bus grant, the local bridge logic unit passes the DMA address from the I/O device to the bridge logic units on the other modules, and the bridge logic units on each module synchronously pass the request to their processors.
  • Synchronous operation is maintained because each bridge logic unit initially receives the bus request from the I/O device at the same time and all three units simultaneously determine when that I/O device bus request is granted.
  • Each processor responds with a bus grant and accepts the DMA address from its respective bridge logic unit.
  • Each processor then reads its own local memory and sends the resulting data to its own local bridge logic unit.
  • the bridge logic units of neighboring modules compare control and data signals produced and/or transferred between processors and the requesting I/O device. Note that it is only the bridge logic unit local to the requesting I/O device that actually passes the data to the requesting I/O device.
  • FIG. 11 there is shown a timing diagram for an example of a four word DMA read from the perspective of a bridge logic unit in which the reading I/O device and the read from device (in this case processor memory) are local to the same module.
  • an I/O device drives out its SBus request
  • the arbitration logic of the local bridge logic unit (not shown) issues an I/O SBus grant, ISBGS to the requesting I/O device as shown at clock tick T3. Note that although each bridge logic unit generates internal signals granting the bus to the requesting I/O device, only the bridge logic unit local to the target I/O device actually drives the bus grant to the target I/O device. At approximately clock tick T4, the I/O device then drives out a virtual address onto the I/O bus data lines, ISD. Also, at about T4, the
  • I/O device drives out the SBus master control signals, ISSIZ and ISRD.
  • the bridge logic unit of the module local to the requesting I/O device issues ISDSEL which broadcasts the virtual address over the backplane on the UDBP of the receiving modules, and broadcasts the SBus master control signals to the other modules over the backplane.
  • ISDSEL which broadcasts the virtual address over the backplane on the UDBP of the receiving modules, and broadcasts the SBus master control signals to the other modules over the backplane.
  • Each bridge logic unit checks that the virtual address is valid for the I/O device requesting the DMA read. That is, each module is allocated a section of memory. I/O devices local to a given module can only perform DMA Reads of the portion of memory allocated to their module. If it is not valid, then the bridge logic units on all of the modules issue error acknowledges to the requesting I/O device. The I/O device actually receives the error acknowledge from its local bridge logic unit.
  • the respective bridge logic units on the different modules issue interrupts to their respective processors, and the transaction is terminated. If the virtual address is valid, however, then at approximately T8, the bridge logic issues a processor SBus request signal, PSBR, to its local processor. Eventually, at about T9 in this example, the respective processors issue respective processor SBus grant signals, PSBG, to their local bridge logic units. Then at about T10, the bridge logic units drive out the virtual address on PSD (which also is connected to ISD via a switch) and drive out SBus master control signals PSSIZ and PSRD to their respective processors. The virtual address is driven on PSD, not SA; therefore, it is driven onto the backplane when ISDSEL is active.
  • ISASEL is inactive throughout the DMA
  • the virtual address that was sent to the processors goes out onto the backplane for comparison and is received from the upstream module on UDBP.
  • the respective processors drive out acknowledge signals, PSACK, which are passed on to ISACK via a switch on the bridge logic unit.
  • PSACK acknowledge signals
  • the processor drives out PSD, which is passed on to ISD via a switch on the bridge logic unit.
  • the bridge logic unit that is local to the requesting device actually passes the acknowledge followed by data to the requesting I/O device.
  • each module drives out the data onto the backplane for delivery to its downstream neighbor for comparison. Note that the data/address are driven out onto the backplane via OBP.
  • UDBP Data/address
  • the downstream neighbor receives the data on UDBP.
  • the bridge logic units select the upstream or downstream information for the requested read data based upon UTDATSEL .
  • each given module compares the following control, address and data signals transacted directly with such module to the corresponding control, address and data signals transacted directly with such module's upstream neighbor.
  • the upstream neighbor signals are transfe ⁇ ed downstream (from such upstream module to such given) module as part of the UDBP signals described above.
  • the compared signals are: PSBG, PSAS, PSRD, PSSIZ, PSACK and PSD.
  • DMA Write PSD[31 :0] Data is received by each Bridge from the I/O device via ISD on the local module and UDBP on the other modules, and is stored in the DMA FIFO of each Bridge. After each Bridge receives PSBG from its CPU, the data is then driven out by each Bridge and received by each CPU via PSD. (In a presently prefe ⁇ ed embodiment PSD and ISD are actually shorted together by a switch controlled by SWDCLS, so the data is on both PSD and ISD.) Data is compared one clock tick after PSACK (i.e., when the data is valid according to SBus protocol).
  • PSAS Same as CPU read.
  • PSRD Same as DMA read.
  • PSSIZ Same as DMA read.
  • PSACK Driven by each CPU and received by its Bridge. ISACK was issued to the I/O device by its Bridge when the Bridge was receiving the DMA data into its DMA FIFO; therefore, the acknowledge received by the Bridge from the CPU is not passed on to the I/O device. Always compared. Note that SA and PSSLV are not compared during DMA because a presentiy prefe ⁇ ed embodiment only supports DMA between an I/O device and CPU memory, not between two I/O devices; therefore, the address information does not have to be valid.
  • an I/O device may employ DMA (direct memory access) techniques to write info ⁇ nation to processor memory.
  • DMA direct memory access
  • the I/O device requests access to processor memory. It will be appreciated, however, that the same basic principles apply when access to another I/O device is requested.
  • a requesting device may issue a bus request signal.
  • the bridge logic responds by issuing a bus grant signal and accepts the DMA virtual address from the requesting I/O device.
  • the local bridge logic which is on the same module as the requesting I/O device, passes the virtual address over the backplane to the bridge logic on the other modules.
  • the local bridge logic accepts the write data from the requesting I/O device and transfers it over the backplane to the bridge logic on the other modules.
  • Each respective bridge logic stores the write data in its respective local FIFO buffer.
  • the respective bridge logic pass the bus request to their local processors.
  • Each respective processor responds with a bus grant, and accepts the DMA virtual address from its local bridge logic.
  • Each processor then accepts the write data from its local bridge logic and writes it to its own main memory.
  • the bridge logic of neighboring modules compare control signals and data signals produced and/or transferred between neighboring processors and the requesting I/O device. Note that although the DMA in the current embodiment is targeted to read or write to main memory, it is consistent with the invention to implement a system in which one I/O device targets a DMA read or write to another I/O device.
  • a tirning diagram for an example of a four word DMA write from the perspective of the module local to the writing I/O device.
  • a writing I/O device drives out its SBus request strobe, OSBRS, to the bridge logic of all the modules.
  • OSBRS SBus request strobe
  • the arbitration logic of each bridge logic unit issues a I/O SBus grant strobe, ISBGS.
  • the bridge logic unit local to the target I/O device actually drives out the ISBGS signal to the target I/O device.
  • the requesting I/O device drives out a virtual address onto the SBus data lines, ISD. Also at T4, the I/O device drives out the I/O SBus master control signals: ISRD and ISSIZ.
  • the virtual address and the SBus master control signals are broadcast over the backplane by the bridge logic unit local to the I/O device and are received by the other modules. The respective bridge logic units check that the virtual address is valid for the I/O device requesting the DMA.
  • each respective bridge logic issues an error acknowledge directed to the requesting I/O device. Although only the bridge logic unit local to the requesting I/O device actually provides the error acknowledge to the device. Each bridge logic unit issues an interrupt to its respective processor and the transaction is terminated.
  • the respective bridge logic unit local to the module with the requesting I/O device begins issuing I/O SBus acknowledges, ISACKs, to the requesting I/O device, thereby receiving the data into its DMA FB7O.
  • the bridge logic broadcasts the I/O device data to the upstream and downstream modules which in turn begin receiving the data via UDBP.
  • the other bridge logic units store the data coming from the backplane into their respective DMA FIFOs.
  • the bridge logic of each respective module issues a processor SBus request, PSBR, to its respective processor.
  • the respective processors issue processor SBus grants, PSBG, to their respective bridge logic.
  • the respective bridge logic drives out the virtual address on PSD (which is connected to ISD via a switch), PSRD and PSSIZ.
  • the respective processors start issuing acknowledges, and the data, PSD, is transfe ⁇ ed out of the respective DMA FIFOs and into respective processors' memories.
  • Each respective bridge logic unit drives the data and control transferred into and out of its processor out across the backplane to its downstream neighbor for comparison. 6.10.1 Comparisons during I/O device writes to memory via DMA
  • each given module compares the following control, address and data signals transacted directly with such module to the corresponding control, address and data signals transacted directly with such module's upstream neighbor.
  • the upstream neighbor signals are transferred downstream (from such upstream module to such given module) as part of the UDBP signals described above.
  • the compared signals are: PSBG, PSAS, PSRD, PSSIZ, PSACK and PSD.
  • SPARCstation 5 which uses the microSPARC-II for the processor, the SBus for the bus protocol, and Solaris 2 for the operating system
  • the principles of the invention are applicable to any bus protocol and operating system, and to any processor that can be brought into a known state through some type of initialization mechanism such as a hardware and/or software reset.
  • Possible processors include but are not limited to the Pentium from Intel, Alpha from DEC, PowerPC from Motorola, etc.
  • Possible bus protocols include but are not limited to PCI, UPA and ISA.
  • Possible operating systems include but are not limited to Windows NT, and UNLX.

Abstract

L'invention a pour objet un système informatique tolérant aux fautes composé d'un premier module contenant un premier processeur, un premier bus de processeur et un premier bus entrée/sortie; d'un deuxième module contenant un deuxième processeur, un deuxième bus de processeur et un deuxième bus entrée/sortie; et d'un troisième module contenant un troisième processeur, un troisième bus de processeur et un troisième bus entrée/sortie. Le premier module dudit système comprend également un premier dispositif de commande qui coordonne le transfert des données relatives à la première transaction entre le premier bus de processeur et chacun des trois bus entrée/sortie, à savoir le premier, le deuxième ou le troisième; et le premier module contient une première logique de comparaison qui compare les données relatives à la première transaction avec les données correspondantes relatives à la deuxième transaction. Le deuxième module dudit système comprend également un deuxième dispositif de commande qui coordonne le transfert des données relatives à la deuxième transaction entre le deuxième bus de processeur et chacun des trois bus entrée/sortie, à savoir le premier, le deuxième ou le troisième; et le deuxième module contient une deuxième logique de comparaison qui compare les données relatives à la deuxième transaction avec les données correspondantes relatives à la troisième transaction. Le troisième module dudit système comprend également un troisième dispositif de commande qui coordonne le transfert des données de la troisième transaction entre le troisième bus de processeur et chacun des trois bus entrée/sortie; et le troisième module contient une troisième logique de comparaison qui compare les données de la troisième transaction avec les données correspondantes de la première transaction. Enfin, ledit système informatique tolérant aux fautes comprend des circuits de transfert assurant le transfert des données relatives aux première, deuxième et troisième transactions parmi le premier, le deuxième et le troisième modules.
PCT/US1997/008320 1996-05-16 1997-05-15 Systeme informatique redondant a trois modules WO1997043712A2 (fr)

Priority Applications (6)

Application Number Priority Date Filing Date Title
EP97926550A EP0916119B1 (fr) 1996-05-16 1997-05-15 Systeme informatique redondant a trois modules
AT97926550T ATE210316T1 (de) 1996-05-16 1997-05-15 Dreifach redundantes modulares rechnersystem
AU31288/97A AU3128897A (en) 1996-05-16 1997-05-15 Triple modular redundant computer system
DE69708881T DE69708881T2 (de) 1996-05-16 1997-05-15 Dreifach redundantes modulares rechnersystem
IL12705997A IL127059A0 (en) 1996-05-16 1997-05-15 Computer system
JP54114297A JP2002515146A (ja) 1996-05-16 1997-05-15 ３連モジュール冗長コンピュータシステム

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US1720196P 1996-05-16 1996-05-16
US60/017,201 1996-05-16
US60/037,363 1997-01-31
US08/853,670 1997-05-09

Publications (2)

Publication Number Publication Date
WO1997043712A2 true WO1997043712A2 (fr) 1997-11-20
WO1997043712A3 WO1997043712A3 (fr) 1998-05-14

Family

ID=21781294

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US1997/008320 WO1997043712A2 (fr) 1996-05-16 1997-05-15 Systeme informatique redondant a trois modules

Country Status (3)

Country Link
AU (1) AU3128897A (fr)
TW (1) TW320701B (fr)
WO (1) WO1997043712A2 (fr)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO1999066405A1 (fr) * 1998-06-15 1999-12-23 Sun Microsystems, Inc. Pont de processeur a acces de donnees dissemblables
WO1999066403A1 (fr) * 1998-06-15 1999-12-23 Sun Microsystems, Inc. Pont de processeur a registres de donnees dissemblables
WO1999066406A1 (fr) * 1998-06-15 1999-12-23 Sun Microsystems, Inc. Passerelle de processeur avec tampon d'ecriture retardee
WO1999066404A1 (fr) * 1998-06-15 1999-12-23 Sun Microsystems, Inc. Pont de systeme multiprocesseur a acces controle
EP0992911A2 (fr) * 1998-09-29 2000-04-12 Dr. Johannes Heidenhain GmbH Méthode et circuit de surveillance de la fiabilité des fréquences d'horloges dans un sytème redondant
EP1052563A2 (fr) * 1999-05-14 2000-11-15 Nec Corporation Système synchrone de traitement de signal
DE10023166A1 (de) * 2000-05-11 2001-11-15 Alcatel Sa Mehrrechner-System
WO2002056176A1 (fr) * 2001-01-16 2002-07-18 Thales Dispositif de synchronisation tolerant aux pannes pour reseau informatique temps reel
CN111213062A (zh) * 2017-09-14 2020-05-29 Bae系统控制有限公司 使用多核处理器减轻共模计算故障

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7725618B2 (en) * 2004-07-29 2010-05-25 International Business Machines Corporation Memory barriers primitives in an asymmetric heterogeneous multiprocessor environment

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0447577A1 (fr) * 1988-12-09 1991-09-25 Tandem Computers Incorporated Système de calculateur à haute performance à capacité de tolérance de fautes
WO1992003787A1 (fr) * 1990-08-14 1992-03-05 Siemens Aktiengesellschaft Systeme multicalculateurs de haute securite avec trois calculateurs
JPH05204692A (ja) * 1992-01-30 1993-08-13 Nec Corp 情報処理装置の故障検出・切離方式
JPH06250867A (ja) * 1993-03-01 1994-09-09 Nippon Telegr & Teleph Corp <Ntt> 耐故障計算機および耐故障計算処理方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0447577A1 (fr) * 1988-12-09 1991-09-25 Tandem Computers Incorporated Système de calculateur à haute performance à capacité de tolérance de fautes
WO1992003787A1 (fr) * 1990-08-14 1992-03-05 Siemens Aktiengesellschaft Systeme multicalculateurs de haute securite avec trois calculateurs
JPH05204692A (ja) * 1992-01-30 1993-08-13 Nec Corp 情報処理装置の故障検出・切離方式
JPH06250867A (ja) * 1993-03-01 1994-09-09 Nippon Telegr & Teleph Corp <Ntt> 耐故障計算機および耐故障計算処理方法

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
1985 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION, 25 - 28 March 1985, ST. LOUIS, MO, US, pages 802-806, XP000647411 OZGUNER F ET AL: "A RECONFIGURABLE MULTIPROCESSOR ARCHITECTURE FOR RELIABLE CONTROL OF ROBOTIC SYSTEMS" *
PATENT ABSTRACTS OF JAPAN vol. 017, no. 634 (P-1649), 24 November 1993 & JP 05 204692 A (NEC CORP), 13 August 1993, *
PATENT ABSTRACTS OF JAPAN vol. 018, no. 645 (P-1839), 7 December 1994 & JP 06 250867 A (NIPPON TELEGR & TELEPH CORP), 9 September 1994, *

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6587961B1 (en) 1998-06-15 2003-07-01 Sun Microsystems, Inc. Multi-processor system bridge with controlled access
WO1999066403A1 (fr) * 1998-06-15 1999-12-23 Sun Microsystems, Inc. Pont de processeur a registres de donnees dissemblables
WO1999066406A1 (fr) * 1998-06-15 1999-12-23 Sun Microsystems, Inc. Passerelle de processeur avec tampon d'ecriture retardee
WO1999066404A1 (fr) * 1998-06-15 1999-12-23 Sun Microsystems, Inc. Pont de systeme multiprocesseur a acces controle
WO1999066405A1 (fr) * 1998-06-15 1999-12-23 Sun Microsystems, Inc. Pont de processeur a acces de donnees dissemblables
US6138198A (en) * 1998-06-15 2000-10-24 Sun Microsystems, Inc. Processor bridge with dissimilar data registers which is operable to disregard data differences for dissimilar data write accesses
US6141718A (en) * 1998-06-15 2000-10-31 Sun Microsystems, Inc. Processor bridge with dissimilar data registers which is operable to disregard data differences for dissimilar data direct memory accesses
US6148348A (en) * 1998-06-15 2000-11-14 Sun Microsystems, Inc. Bridge interfacing two processing sets operating in a lockstep mode and having a posted write buffer storing write operations upon detection of a lockstep error
EP0992911A2 (fr) * 1998-09-29 2000-04-12 Dr. Johannes Heidenhain GmbH Méthode et circuit de surveillance de la fiabilité des fréquences d'horloges dans un sytème redondant
EP0992911A3 (fr) * 1998-09-29 2005-09-28 Dr. Johannes Heidenhain GmbH Méthode et circuit de surveillance de la fiabilité des fréquences d'horloges dans un sytème redondant
EP1052563A2 (fr) * 1999-05-14 2000-11-15 Nec Corporation Système synchrone de traitement de signal
EP1052563A3 (fr) * 1999-05-14 2008-03-19 NEC Electronics Corporation Système synchrone de traitement de signal
DE10023166A1 (de) * 2000-05-11 2001-11-15 Alcatel Sa Mehrrechner-System
WO2002056176A1 (fr) * 2001-01-16 2002-07-18 Thales Dispositif de synchronisation tolerant aux pannes pour reseau informatique temps reel
FR2819598A1 (fr) * 2001-01-16 2002-07-19 Thomson Csf Dispositif de synchronisation tolerant aux pannes pour reseau informatique temps reel
US7509513B2 (en) 2001-01-16 2009-03-24 Thales Fault-tolerant synchronisation device for a real-time computer network
CN111213062A (zh) * 2017-09-14 2020-05-29 Bae系统控制有限公司 使用多核处理器减轻共模计算故障
CN111213062B (zh) * 2017-09-14 2022-11-29 Bae系统控制有限公司 使用多核处理器减轻共模计算故障

Also Published As

Publication number Publication date
TW320701B (fr) 1997-11-21
AU3128897A (en) 1997-12-05
WO1997043712A3 (fr) 1998-05-14

Similar Documents

Publication Publication Date Title
US6141769A (en) Triple modular redundant computer system and associated method
US5185877A (en) Protocol for transfer of DMA data
US5153881A (en) Method of handling errors in software
EP0415545B1 (fr) Procédé de traitement d&#39;erreurs de logiciel
KR100566338B1 (ko) 폴트 톨러런트 컴퓨터 시스템, 그의 재동기화 방법 및 재동기화 프로그램이 기록된 컴퓨터 판독가능 기억매체
EP0747820B1 (fr) Méthode de synchronisation d&#39;une paire d&#39;unités de traitement centrale pour une opération en duplex et en lock-step
US4785453A (en) High level self-checking intelligent I/O controller
US5251227A (en) Targeted resets in a data processor including a trace memory to store transactions
US5068851A (en) Apparatus and method for documenting faults in computing modules
US6035414A (en) Reliability of crossbar switches in an information processing system
JP2573508B2 (ja) ディジタルロジック同期モニター方法および装置
JPH02118872A (ja) Ｉ／ｏの読み取りに対するエラーチェック機能を有したデュアルレールプロセッサ
US5065312A (en) Method of converting unique data to system data
JPH01154243A (ja) 耐欠陥性でない要素と耐欠陥性システムとのインターフェイス
US5048022A (en) Memory device with transfer of ECC signals on time division multiplexed bidirectional lines
US5163138A (en) Protocol for read write transfers via switching logic by transmitting and retransmitting an address
JP3595033B2 (ja) 高信頼化コンピュータシステム
EP3273353B1 (fr) Dispositif de traitement de données
WO1997043712A2 (fr) Systeme informatique redondant a trois modules
EP0747817B1 (fr) Méthode et dispositif de contrôle du flux de données dans un système à multiprocesseur à tolérance de fautes
JP3069585B2 (ja) データ処理装置における目標指定リセット法
JPH05313930A (ja) 高信頼度化情報処理装置
JPH0916426A (ja) ２ポートコンソールを持つフォールトトレラントコンピュータ
JPH03184155A (ja) 非存在メモリエラー処理の方法
JPH10322334A (ja) リトライバッファの試験システム

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AL AM AT AU AZ BA BB BG BR BY CA CH CN CU CZ DE DK EE ES FI GB GE GH HU IL IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK TJ TM TR TT UA UG US UZ VN YU AM AZ BY KG KZ MD RU TJ TM

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): GH KE LS MW SD SZ UG AT BE CH DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF

DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
121 Ep: the epo has been informed by wipo that ep was designated in this application
AK Designated states

Kind code of ref document: A3

Designated state(s): AL AM AT AU AZ BA BB BG BR BY CA CH CN CU CZ DE DK EE ES FI GB GE GH HU IL IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK TJ TM TR TT UA UG US UZ VN YU AM AZ BY KG KZ MD RU TJ TM

AL Designated countries for regional patents

Kind code of ref document: A3

Designated state(s): GH KE LS MW SD SZ UG AT BE CH DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF

WWE Wipo information: entry into national phase

Ref document number: 1997926550

Country of ref document: EP

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

WWP Wipo information: published in national office

Ref document number: 1997926550

Country of ref document: EP

NENP Non-entry into the national phase in:

Ref country code: CA

WWG Wipo information: grant in national office

Ref document number: 1997926550

Country of ref document: EP