WO1996033466A1

WO1996033466A1 - Performing input/output operations in a multiprocessor system

Info

Publication number: WO1996033466A1
Application number: PCT/EP1995/001454
Authority: WO
Inventors: Rolf Hilgendorf; Gottfried Goldrian; Rolf Fritz; Klaus-Jörg GETZLAFF
Original assignee: International Business Machines Corporation
Priority date: 1995-04-18
Filing date: 1995-04-18
Publication date: 1996-10-24
Also published as: JPH09507939A

Abstract

The invention concerns I/O operations in a multiprocessor system. These systems comprise a hierarchically structured I/O bus structure, wherein at the nodes of the I/O bus, adapters are provided which support I/O operations. The adapters decide if to accept an I/O operation, and in case of acceptance, they direct the transmission of the operation through the I/O bus and temporarily disconnect the line between this node and the foregoing node of the I/O bus means.

Description

D E S C R I P T I O N

PERFORMING INPUT/OUTPUT OPERATIONS IN Λ MULTIPROCESSOR

SYSTEM

The invention is directed to a multiprocessor system with a hierarchically structured Input/Output (I/O) bus, wherein at the nodes of the I/O bus, adapters are provided which support I/O operations. Further the invention relates to an improved method for performing I/O operations in a multiprocessor system.

In multiple processor computers known in the prior art, the processing units communicate with I/O adapters in order to start up or to perform I/O operations. Those systems provide appropriate I/O instructions which can be for example "Sense/Control instructions", where a Control instruction transfers one word of data to an I/O adapter, and a Sense instruction causes an adapter to transfer one word of data back to a processing unit.

In computer systems with a number of processing units being connected to a main storage and a plurality of I/O channels by means of a single system bus, all processing units suffer from storage or I/O accessibility, if one of them performs such a Sense/Control operation, since the bus is occupied for a few cycles to carry out the transfer operation.

In the prior art approaches, the system bus is held busy/occupied until the I/O adapter has completed the Sense/Control operation. For a Sense, this is until the requested data are delivered. For a Control, this ranges from one cycle after the data is put onto the bus until an Acknowledge signal becomes true. The exact time depends on the bus protocol as well as on the adapter design. A further disadvantage of those known approaches is that in systems which consist of several, hierarchically structured buses, each possibly with a different timing, the time where the bus nearest to the processing units is occupied, increases dramatically. For this reason, those systems can not comprise a linear increase of performance with an increasing number of processing units and can not take advantage of a more complex I/O bus structure.

It is therefore an object of the present invention to provide a multiple processor system with an improved I/O performance, and an improved method for the communication of multiple processing units with the I/O adapters of a hierarchical I/O bus structure which increases accessibility of the common system bus. Hereby a particular aim for a system design is to make the occupation of the I/O buses minimum duration.

This problem is solved by the features laid down in the independent claims. The advantages of the proposed solution become more evident with respect to the following part of the description, where a preferred embodiment of the invention is described in more detail with reference to the drawing in which

Fig. 1 is a schematic representation of a multiple processor computer system according to prior art approaches, which comprises a hierarchically structured I/O bus system;

Fig. 2 and 3 show block diagram charts which depict the propagation of signals through several levels in a hierarchical I/O system; and Fig. 4 is a timing chart for a bus protocol illustrating the improved method according to the invention for handling a complex I/O bus structure as shown in Fig. 1.

Fig. 1 shows a prior art multiple processor computer system which comprises a number of processing units CPU_1 - CPU_n which are connected to a memory unit MEM consisting of several memory banks. The processing units are also connected to a hierarchical I/O bus structure via a BUS ADAPTER_1 and by means of a single system BUS_1. At the ADAPTER_1, the I/O bus splits up into several tree-like sub-buses, each of them being connected to terminal bus partitions I/O ADAPT via a number of second-stage ADAPTERS_2. This structure shows, only exemplary, the architecture of complex hierarchically structured I/O bus systems, whereby the shown structure can be extended by adding more layers of sub-buses at the bottom of the shown structure.

Referring to Fig. 2 and Fig. 3, it is now assumed that one of the processing units CPU_1 - CPU_n delivered a Sense/Control command to the I/O system via BUS_1 and is now waiting for the answer to the Sense Command. In this situation, all the other processing units temporarily have no access to BUS_1, since BUS_1 is occupied by the requesting processing unit at this moment. Thus BUS_1 first has to be freed to allow other processing units to access the main memory MEM. Moreover, if the Sense/Control command propagates through several levels in the hierarchical I/O system, as shown in Fig. 2, every level is blocked until the answer of the I/O Adapt of destination is received.

According to prior art approaches, a possible solution in this situation can be that the Sense operation starts normally and frees BUS_1 after the Command is accepted by ADAPTER 1. The Command then is driven to its destination location, e.g. one of the I/O Adapters on BUS_3. But the answer returnes back only to ADAPTER_1 and is stored there temporarily. Thus the processing unit which needs the result has to poll frequently ADAPTER_1 to find out if the result has already arrived and in case of it, finally fetches it from a pre-defined storage location in ADAPTER_1. The described method, therefore, puts a heavy load onto BUS_1.

Alternatively, ADAPTER_1 may deliver an interrupt signal to the processing units. Depending on the system design, one or all processing units will react to the interrupt. If all processing units react, each of them has to stop its current program and to switch to an interrupt handler. The interrupt handler has to sense all interrupt sources known to him and this way determine if some action has to follow or not. Sooner or later, all except the one processor which waits for the answer will turn back and continue their current programs. The processing unit which is waiting will complete its Sense/Control operation and may continue as well.

If only one processing unit is selected to react, it has to stop its current operation and to find the source for the interrupt. Then, from the data which are gathered from the interrupting unit, it has to determine which processing unit has to receive the interrupt to process it further.

Thereupon, the above described methods have the limitation that ADAPTER_1 has to provide enough storage locations for the maximum number of Sense/Control instructions which can be concurrently active. Hereby a Sense instruction can be rejected by a reject mechanism in case of no location being available. In general, a Sense/Control instruction to an I/O adapter breaks up into two operations, a Sense from a processing unit and a Control from the adapter transferring the answer. According to the proposal of the invention a new operation for BUS_1, called Disconnected_Sense, is defined.

By means of the proposed method, the Sense/Control command is propagating through several levels in the hierarchical I/O system, where each level is freed by an early status signal (Fig. 3). In the first cycle of an operation, its command phase, the destination address of the I/O adapter, the operation code, the address of the issuing processing unit and the information that the operation shall perform as Disconnected_Sense is transferred to ADAPTER_1.

In the second cycle, a Control data word is transferred if the operation code specified is a control operation to an I/O adapter. After receiving the first cycle, ADAPTER_1 interprets the command and after receiving the second cycle it sends an early status signal back to the issuing processing unit. This early status may be an Accept, an Error or a Reject signal. Alternatively, the command is accepted and preliminary stored in the ADAPTER_1, anyway. A Reject signal is sent if there is already a Sense operation pending for one of the I/O adapters connected to the same ADAPTER_2 as the I/O adapter which is currently tried to be accessed.

Having transferred the early status, BUS_1 is disengaged for further operations. The processing unit which issued the Disconnected_Sense instruction falls into a quiescent state where it awaits a Control Command from ADAPTER_1.

ADAPTER_1 takes up the command and the data received, adds a sequence number for identification and sends all down to the appropriate ADAPTER_2, e.g. using BUS_2. After sending this package, BUS_2 is free again. This is accomplished by sending an early status signal from ADAPTER_2 to ADAPTER_1.

ADAPTER_2 inspects the command, and transfers it to the appropriate bus, e.g. BUS_3, to which the addressed I/O adapter is connected. This bus is held occupied until the I/O adapter of destination puts out the answer. Meanwhile ADAPTER_2 transforms the command received into a Control command which is sent back to ADAPTER_1. Hereby the address of the source processing unit is transformed as to be the destination address for the Control command.

In case of a Control command from the issuing processing unit, the answer of the I/O adapter of destination is a simple Status information. For a Sense command it is either the requested data word or an Error status. The received answer is put as data to the Control prepared by ADAPTER_2 and then sent to ADAPTER_1.

If the ADAPTER_1 receives the Control signal correctly, it requests BUS_1 and forwards the Control to the waiting processing unit. If the data transfer between ADAPTER_2 and ADAPTER_1 is erroneous, ADAPTER_1 sends a Control with a bad status indication to the waiting processing unit. In both cases, ADAPTER_1 resets an appropriate "sense pending" latch to allow for new sense commands.

In Fig. 4 a timing chart of a bus protocol illustrating the improved method according to the invention is shown. For the underlying I/O bus structure which comprises three bus layers BUS_!, BUS_2 and BUS_3, it is referred to the foregoing Figures. It is worthy of note that the three bus layers of this embodiment comprise different bus cycle times, respectively. But the invention can be also applied to a bus structure which provides an allover synchronous bus cycle. The drawing exemplary shows the pending signals on these buses in case of an exemplary Disconnected-Sense operation, dependent on the time.

It is further assumed that one of the processing units CPU_1 - CPU_n starts an I/O operation by delivering a Disconnected-Sense instruction via BUS_1. This instruction is followed by Disconnected Data which can be the content of a CPU register containing for example a memory address which defines a memory location in which incoming data from an I/O device of destination have to be stored. ADAPTER_1 takes up and interprets this command and sends an early status command back to the issuing processing unit.

The ADAPTER_1 having transferred the early state signal, BUS_1 becomes free and can be occupied by further operation signals. In addition, the processing unit which issued the Disconnected_Sense instruction falls into a quiescent state awaiting a control command from ADAPTER_1. Using BUS_2 ADAPTER_1 sends this package down to the appropriate ADAPTER_2 which lies on the way to the adapter of destination. After sending this package also BUS_2 becomes free of signals.

The protocol of BUS_3 is illustrated for two exemplary cases. In the first paradigm the issuing processing unit delivered a Control command wherein a Control command CC followed by a data package, e.g. the ordered I/O operation or an address in the main storage, where data fetched by means of I/O shall be stored, are on BUS_3. Here the I/O Adapt of destination delivers a status signal, e.g. an Accept of a command or a Not Accept due to an erroneous state or an already busy state of the Adapter. The I/O Adapter of detination delivers data which describe the internal state of the destinated I/O Adapter. The terminal I/O Adapteres can be SCSI controllers or Adapters, which communicate with I/O channels. The processing units themselves provide logical circuits for handling Disconnected-Sense operations. At the time, the issuing processing unit receives a correct early status word due to an acceptance of the command, a CPU internal Disconnected-Sense Pending latch is set. This latch being set instructs the private L2 cache memory of the CPU to accept a Control command from ADAPTER_1 which is addressed to this CPU. Herethrough the L2 cache memory stores data into a register pair of the CPU and resets the Disconnected-Sense Pending latch.

Additionally a CPU microcode polls the Disconnected-Sense Pending latch and as soon as the latch is reset, the microcode continues to use data received or a bad status signal, respectively. The microcode polling loop is time¬ out controlled, i.e. if the latch does not drop within a defined period of time, microcode polling is stopped and a special control, only for this purpose, is issued to clear the Disconnected-Sense Pending latch in ADAPTER_1 as already described above.

In normal operation, the sense-pending latch in ADAPTER_1 is set, when ADAPTER_1 drives a Sense/Control command on BUS_2. In case of a Reset, a sequence number comes back as answer and the sense-pending latch is reset.

In a problematic operation situation, like a Sense from a processing unit is not replied by the Adapter of destination, BUS_2 is temporarily blocked. Then the processing unit resets all of the pending latches of the Adapters on the bus path to the destination Adapter.

Claims

C L A I M S

1. Multiprocessor system comprising:

at least two processing means which support I/O operations;

main memory means for storing information which is processed by the processing means;

common bus means for transmitting information between the processing means and the memory means;

hierarchically structured I/O bus means for transmitting I/O information which is connected to the common bus means;

adapter means at the nodes of the I/O bus means for transmitting I/O information via lines of the hierarchical I/O bus means, including means for the analysis of I/O instructions and for generating responses, and means for driving I/O instructions onto the next lower bus level of the hierarchical I/O bus means.

2. The multiprocessor system according to claim 1, wherein the processing means provide an I/O instruction for disconnecting an I/O operation from the next upper bus level of the hierarchical I/O bus means.

3. The multiprocessor system according to claim 2, wherein the processing means comprise latch means for indicating that a disconnecting I/O instruction is delivered to the hierarchical I/O bus means.

4. The multiprocessor system according to any one of the claims 1 to 3, wherein the adapter means provide means for rejecting I/O instructions in case of the targeted adapter being busy.

5. The multiprocessor system according to any one of the preceding claims, wherein providing means for awaiting two phase responses from driving I/O instructions onto the next lower bus level of the hierarchical I/O bus means.

6. A computer system comprising the multiprocessor system according to any one of the preceding claims.

7. A method for performing I/O operations in a multiprocessor system, the system comprising processing means and main memory means which are interconnected by common bus means, wherein the common bus means are connected to a hierarchically structured I/O bus means, and wherein adapter means are provided at the nodes of the I/O bus means which support I/O operations and transmit I/O information via lines of the hierarchical I/O bus means;

the method comprising the following steps:

analysing I/O instructions being delivered from a processing means;

generating responses to these instructions; driving the I/O instructions onto the next lower bus level of the hierarchical I/O bus means;

awaiting two phase responses from there.

8. The method according to claim 7, wherein the processing means provide an I/O instruction for disconnecting an I/O operation from the next upper bus level of the hierarchical I/O bus means.

9. The method according to claim 8, wherein each level of the propagated hierarchical I/O bus means is freed by an early status signal which is delivered by each of the concerned adapter means.

10. The method according to any one of claims 7 to 9, wherein the different stages of the hierarchical I/O bus structure comprise different bus cycle times.