PCI Interrupt Routing
From Open Watcom
As usual, PCI interrupt routing in PC compatibles was made incredibly complicated by the quest for backwards compatibility combined with some questionable technical decisions.
The PCI bus was introduced in the early 1990s as a solution for the problems caused by the outdated ISA bus and its incompatible successors. By mid-1980s, the deficiencies of the ISA bus were becoming more and more obvious. Configuring ISA add-on cards was black magic dreaded by mere mortals, and 386 systems were requiring higher bandwidth than ISA could provide.
IBM's solution to this problem was the MicroChannel Architecture (MCA), introduced in the PS/2 line of computers. Technically, MCA was very advanced and did indeed solve the problems with ISA (configuration, bandwidth, interrupt sharing), but MCA systems were also completely incompatible with existing ISA adapters while MCA adapters were scarce and more expensive than their ISA equivalents. The "rest of the world" (that is, Compaq, HP, and others) came with a different solution called EISA which used a backwards compatible form factor and solved most of the ISA deficiencies, but suffered from the same problem with lack and high price of adapter cards. EISA bus was very rare outside of server PCs.
The PCI initiative was driven by Intel and coincided with the development of the Pentium processor, which promised to be too powerful for even MCA and EISA to handle. PCI was in some ways similar to MCA by using a new form factor not compatible with ISA and providing full software-driven discovery and configuration of hardware. Backwards compatibility was solved by a PCI-to-ISA bridge and separate ISA expansion slots.
Pentium systems were almost exclusively PCI based, and both EISA and MCA were replaced by PCI within a few years.
One of the design objectives for PCI based PC compatibles was a high degree of backwards compatibility. That, significantly, included the ability to run DOS and Windows 3.x, systems with no built-in PCI support. The BIOS had to configure all PCI devices required for booting and the systems kept using 8259A compatible interrupt controllers.
Handling PCI interrupts presented two significant challenges:
Few PCI Interrupts
The PCI specification defines four interrupt signals, INTA# to INTD#. That is not much and interrupt sharing is a must. Fortunately, that problem had already been solved by EISA by the introduction of the ELCR register. An 8259A compatible interrupt controller could selectively treat specific interrupt inputs as edge or level triggered.
This problem was compounded by PCI specification requirements. Only a multi-function PCI device could use all PCI interrupts. A single-function device - and most PCI devices are single-function - may only use the INTA# signal. In practice that meant most PCI devices in a system would use INTA# to signal interrupts.
Few available 8259A Interrupts
Because existing DOS/Windows software had to keep working, the existing 8259A compatible interrupt controller scheme had to be used. The cascaded master/slave model introduced in the IBM PC/AT provided 15 interrupts usable by devices. Unfortunately, most of those were already taken by common devices:
- IRQ0: System Timer (8254 PIT)
- IRQ1: System Keyboard (8042 controller)
- IRQ2: Cascade Interrupt (slave 8259A)
- IRQ3: Serial Port (COM 2)
- IRQ4: Serial Port (COM 1)
- IRQ5: Sound Card (Sound Blaster compatible or other)
- IRQ6: Floppy Drive
- IRQ7: Printer Port (LPT)
- IRQ8: Real-Time Clock (RTC)
- IRQ9: VGA/IRQ2
- IRQ10: Unassigned
- IRQ11: Unassigned
- IRQ12: PS/2 Mouse (8042 controller)
- IRQ13: Coprocessor Error
- IRQ14: IDE Controller (primary)
- IRQ15: IDE Controller (secondary)
In a typical PC without many additional ISA adapters, perhaps only 1-2 interrupt request lines were available for PCI devices. IRQ10 and IRQ11 were often available. IRQ9 was typically available unless the VGA interrupt was used. IRQ5 was available if no ISA sound card was installed.
The two major problems, excessive interrupt sharing and scarcity of interrupt lines, had naturally very different solutions.
INTs and PIRQs
The problem of most devices using the same PCI interrupt signal (INTA#) was solved by creative motherboard wiring. A new set of signals called PIRQA# to PIRQD# (PCI Interrupt Requests) was introduced and the motherboard would map INTx# to PIRQx# in one way or another. The mapping typically depends on the PCI slot number. Each slot often rotates the INTx# to PIRQx# mapping and the result might look like the following:
|Interrupt||PCI Slot 1||PCI Slot 2||PCI Slot 3|
In this way, three PCI devices in three different slots might each use the same PCI interrupt, yet signal three different interrupts to the motherboard.
This interrupt routing scheme is not standardized and is motherboard specific. Special tables exist which describe how PCI interrupts are mapped. The mapping is typically fixed and neither the BIOS nor an OS can change it.
Now that a motherboard chipset receives up to four different interrupt signals from PCI devices, it has to solve the opposite problem again: Routing these signals to the interrupt controller which may have very few unused interrupt lines available.
This routing is accomplished by the motherboard chipset. It is not standardized, but with the knowledge of the chipset it is possible to both query and modify the PIRQ routing. In the original PIIX/PIIX3 southbridges of the Pentium and Pentium Pro era, the PIRQ routing was quite straightforward and modern Intel chipsets have not changed much.
The PCI-to-ISA bridge, and later the LPC device, provided four 8-bit registers in its PCI configuration space. Each register would map one PIRQ signal to a specified 8259A interrupt. The mapping might look as follows:
In the above example, PIRQA and PIRQD are routed to the same 8259A interrupt, presumably because all other interrupt inputs were taken by ISA devices. A less sophisticated BIOS might map all PIRQs to a single 8259A input.
The PIRQ registers may also block the routing completely. In the case of Intel chipsets, that is accomplished by setting the high bit of a PIRQx_ROUT register.
Finding a Way
Device drivers are normally spared from having to fight with this complexity. The PCI specification provides a read/write register called Interrupt Line at offset 0x3C in a device's configuration space. This register does not affect the functionality of the device in any way. It exists solely as a way for the firmware or potentially an operating system (which has full knowledge of PCI interrupt routing) to provide interrupt routing information to device drivers (which don't).
Operating systems are in a trickier position, because they need to understand both how the motherboard maps PCI INTx# signals to PIRQx# signals, as well as how PIRQx# signals are routed to the interrupt controller. For use with Windows 95 OSR2 and later, Microsoft designed the PCI IRQ Routing Table. This table is provided by the BIOS and uses the $PIR signature; it describes the motherboard wiring (INTx# to PIRQx# mapping) and provides information about the chipset IRQ routing. The PCI BIOS services may also be used to query PCI interrupt routing information.
The ACPI specification provides a PCI Routing Table (_PRT) which provides the INTx# to PIRQx# mapping information. Additional LNKA-LNKD tables describe the chipset PCI IRQ routing, if applicable.
Systems with I/O APIC
In multi-processor and later single-processor systems, the legacy 8259A compatible interrupt controller is replaced by one or more I/O APICs (Advanced Programmable Interrupt Controller). Devices signal interrupts to an I/O APIC which forwards them to Local APICs on the CPUs.
With an I/O APIC, the PCI interrupt routing is somewhat different for at least two reasons. With PIIX3/PIIX4 southbridges, the 8259A compatible circuitry was built in while an I/O APIC was an optional external chip. The PIRQx_ROUT registers were internal to the chipset and had no effect on the I/O APIC. Furthermore, an I/O APIC hooked to these southbridges typically has 24 or more interrupt lines; that provides an opportunity to use four separate lines for PIRQx# signals while leaving all interrupts in the ISA legacy 0-15 range for use by ISA devices. On older SIO.A or PCEB/ESC southbridges, both the 8259A and I/O APIC was built-in, and the I/O APIC had only 16 interrupt lines, thus PCI interrupts were routed to both the 8259A and the I/O APIC using the same PIR that was configured by the BIOS.
Finding the APIC Way
For device drivers, there is no change; the PCI Interrupt Line register performs the same function as before.
For operating systems, things are quite different. The $PIR table is only suitable for use with an 8259A compatible interrupt controller. Instead, the interrupt routing tables specified by the Intel MultiProcessor Specification (MPS) or ACPI. Both MPS and ACPI can describe complex systems with multiple PCI buses and multiple I/O APIC chips.
More recent systems have changed the landscape in some small and some significant ways.
Some chipsets allow motherboard devices to use additional interrupt signals not specified by the PCI standard. The interrupts may be called INTE#-INTH# and map to PIRQE#-PIRQH#. These interrupts are never present on a PCI bus and only exist within a chipset. In a chipset with many integrated devices (disk, LAN, audio, USB, etc.) it is advantageous to use additional interrupt lines to reduce the need for interrupt sharing. The additional interrupt signals may be unused by default and only activated by chipset specific drivers.
Message Signaled Interrupts or MSIs are a completely different way of delivering interrupts from PCI devices to a CPU. MSIs can solve interrupt routing problems, as well as problems with latency and ordering that are typical with traditional INTx# PCI interrupts.
MSIs are expected to eventually replace traditional interrupt controllers. With MSIs, interrupts are signaled by writing a DWORD to a specified address in the host's memory instead of using a dedicated interrupt line. The DWORD is typically decoded by a CPU's Local APIC. MSIs solve several problems:
- The system is not limited to four interrupt signals
- A single-function device may use more than one interrupt request
- Memory transactions avoid problems with ordering, especially for devices behind PCI-to-PCI bridges
- Configuration does not require motherboard or chipset specific knowledge
Details of MSI operation are beyond the scope of this article.