Conference turris::digital_unix

8869.0. "Request for SMP related information" by FREE::TYREE () Tue Feb 18 1997 10:08

Here's a copy of a mail message I received from a vendor of EBSA and PCI
boards.  Digital is working with this company to port device drivers from
other environments to Digital Unix 4.x.      

They are naturally interested in maintaining a maximum amount of source code
compatability between the code that runs on Digital Unix and code that runs
on other platforms.

They are just now trying to support SMP systems - both Digital and others.

Any direct answers to their questions, or suggested additional source of
information would be welcome.

We've already provided the vendor with :
	Alpha AXP Architecture Reference Manual
	Writing Network Device Drivers
        Writing Device Drivers: Tutorial
	Writing Device Drivers: Reference
        Writing Device Drivers: Advanced Topics
	Writing EISA and ISA Bus Device Drivers
	Writing PCI Bus Device  Drivers
We've also ported one of their drivers to Digital Unix (with SMP support),
and provided a copy to them.

In the messages below, please read:
        "APAPI"         --> our Streams Device Driver


From the board vendor:

Anyway, I have a new question to add to this list.  There appears to be 
a case inside the APAPI where an SMP Lock would be held across a block.  
You had warned me that it was not recommended to use DEC UNIX Complex 
Locks within a Device Driver. 

The scenarios I am referring to are with the queueAPReadPacket() and 
queueAPWritePacket() functions.  These functions map and lock the user's 
Packet into its physical memory pages.  These "physical" pages are what 
is passed to the firmware Packet rings. 

Under AT&T UNIX, this mapping function is implemented with the 
physiock() DDI/DKI call.  This is an asynchronous function which may 
incur blocks as faulted pages are swapped into physical memory. 

The SMP conflict is that the physiock() function does not guarantee 
multiple mapping operations will be serialized.  For example, if one CPU 
is queueing a Read Packet which is in the middle of mapping, a second 
CPU mapping operation could theoretically intersperse its "physical" 
Packets.  Therefore, the APAPI code would like to hold an SMP Lock 
during which blocks could be incurred. 

George

----------
Subject: 	Alpha SMP Server Atomic Operations

Hello Jim,

I have a few questions about the SMP features of the Alpha server 
platform, running Digital UNIX. 

To what degree are atomic operations sanctioned?  It is certainly true 
that integrity could be insured using simple locks.  On this OS, 
acquiring a simple lock incurs a minimal performance penalty.  Our 
conflict is with portability to UNIX SVR4.2.  The SMP lock facilities in 
this DDI/DKI always adjusts the interrupt priority level.  Under some 
UNIX implementations, this may be time consuming.  It is desirable to 
structure the APAPI such that atomic operations are used to replace some 
cases where a lock could also be used. 

The two atomic operations desired are atomic OR and atomic AND.  These 
would essentially perform a Read/Modify/Write operation on a memory 
location which excludes intermediary memory accesses by other Alpha 
CPUs. 

So, if these operations are available, and are sanctioned, how are they 
referenced/utilized from the APAPI C source? 

Sometimes (maybe often), it is desirable to acquire a lock and 
additionally disable interrupts on the CPU acquiring the lock.  These 
are two distinct steps under Digital UNIX.  I have come across both 
recommendations for the order to do this.  Is it recommended to disable 
interrupts first, followed by acquiring the lock OR to acquire the lock 
and then disable interrupts on that CPU?  Or, does it really matter?  
(My guess would be the former sequence.) 

Another potential optimization to this is the degree to which interrupts 
are disabled.  In our APAPI Channel interface, we would like to use 
simple locks only, as opposed to a combination of simple locks and 
complex locks.  When a simple lock is obtained by one CPU, the APAPI is 
primarily concerned with not being preempted by an interrupt from the 
Channel adapter.  Therefore, it may be possible to change the SPL to 
only mask out the servicing of interrupts from the Channel adapter 
verses disabling all interrupts.  This optimization might be too complex 
to be warranted.  The question would be how to determine the minimum SPL 
which would disable Channel adapter interrupts. 

However, disabling all interrupts would also prevent Thread preemption 
from a timer interrupt.  Since it is intended to minimize the amount of 
time interrupts are disabled, this latter masking of timer interrupts 
may be desirable anyway. 

Thank you. 

George