| ***** Cross Posted DIGITAL_UNIX notesfile Primes::thomas *****
I am having a problem with lost packets while reading from the DLI at FDDI
saturation rates above 60%.
I am on a 2100 under OSF/1 3.2. The application is using the VADS Self Ada
compiler. The application is doing non-blocking I/O on the recvfrom socket I/O
call.
An Ada task attempting a read calls recvfrom. If recvfrom returns EWOULDBLOCK,
the reader Ada task blocks on a binary semaphore. A separate signal handler
intercepts sigio signals, does a select to determine if that LAN now has a
read buffer available, and releases the binary semaphore. The reader Ada task
now unblocks on the binary semaphore and retries the recvfrom call. The reader
Ada task increments counters for received packets, received bytes, and checks
the packet sequence number against the expected sequence number.
A sender task on another machine is sending full FDDI packets which include a
sequence number for the reader to check. The sender can be throttled to send
at a desired FDDI bandwidth saturation rate by specifying delays between sendto
calls.
When the sender is sending at rates below 60%, the reader gets all packets and
in the proper sequence. Above that rate, the reader looses packets every few
hundred reads. The missing packets are lost 1, 2 or 3 at a time.
The sb_max has been dbx patched to anywhere from 132K to 9M bytes with the only
effect being the length of time until the symptom occurs. Larger values of
sb_max allow the application to run longer before experiencing the problem.
But once it occurs, it then happens consistently.
Questions:
1. Is this loss of packets on the reading side to be expected?
2. What can be done to reduce/eliminate the problem?
2.1 Can the UNIX priority of the application be set to advantage?
2.2 Is there other UNIX level tuning that can be done?
2.3 Is another non-blocking approach to reading better?
(Note: the application is running on a single threaded compiler and
cannot do blocking I/O as background processing must be done when
a read is blocked.)
Related issue on send side:
Since the DLI does not support blocking write operations, the application has
been modified to throttle sending as follows:
a) attempt a sendto call
b) if ENOBUFS is returned, delay to allow DLI to clear some buffer space
and retry
Questions:
1. Is there good way to calculate the best backoff delay so that:
i) the DLI send buffers are not allowed to completely empty during the
delay and thus waste bandwidth, and
ii) the retry does not happen too soon so as to have lots of busy looping
Currently this delay time is arrived at empirically: it is increased until the
bandwidth utilization just drops below saturation. The backoff delay time is a
function of the sb_max size and the LAN maximum bandwidth.
How do existing applications perform this send throttling?
2. What impact will this approach have of receive operations on the same
machine?
2.1 Do reads and writes to the DLI sockets share the same set of buffers?
2.1.1 If so, will exhausting buffers on sending interfere with the DLI receive
processing?
2.1.1.1 If exhausting DLI buffers on sends hurts receiving, how to best deal
with this - what do existing applications do?
Our customer is very hot to get this problem resolved. It affects this
project and related follow-on projects as to vendor selection.
Any thoughts, ideas, or related experiences would be helpful. Also, is there
any test code that already exists that we could us to check out our LAN read
reliability as a function of FDDI bandwidth saturation rates?
Thanks for your help.
Mike Thomas
(410) 765-5357
If helpful, source code follows... (Copyrighted by Westinghouse - used with
permission.)
--
--FCNTL(2) UNIX Programmer's Manual FCNTL(2)
--
-- fcntl - file control
-- see /usr/include/fcntl.h
--
with Os_Files;
with Unix;
package Modified_Fcntl is
-- Flag values accessible to open(2) and fcntl(2)
-- (The first three can only be set by open)
-- These values are defined in os_files.a
-- O_RDONLY 0
-- O_WRONLY 1
-- O_RDWR 2
-- O_NDELAY FNDELAY -- Non-blocking I/O
-- O_APPEND FAPPEND -- append (writes guaranteed at the end)
type Fcntl_Commands is (F_Dupfd, -- Duplicate fildes
F_Getfd, -- Get fildes flags
F_Setfd, -- Set fildes flags
F_Getfl, -- Get file flags
F_Setfl, -- Set file flags
F_Getown, -- Get owner
F_Setown); -- Set owner
-- flags for F_GETFL, F_SETFL-- copied from <sys/file.h>
-- non-blocking reads
Fndelay : constant Os_Files.Open_Flags := Os_Files.O_Ndelay;
-- asychronous I/O (raise SIGIO when I/O transitions from blocked to ready)
Fasync : constant Os_Files.Open_Flags := 8#0100#;
Fnonblocking : constant Os_Files.Open_Flags := 8#0100000#;
-- append on each write
Fappend : constant Os_Files.Open_Flags := Os_Files.O_Append;
function Fcntl (Fd : Os_Files.File_Descriptor;
Cmd : Fcntl_Commands;
Arg : Os_Files.Open_Flags := 0) return Unix.Status_Code;
pragma Interface (C, Fcntl);
end Modified_Fcntl;
with V_Semaphores;
with Os_Files;
with Errno;
package Sigio_Distributor is
Signal_Count : Long_Integer := 0;
pragma Volital (Signal_Count);
------------------------------------------------------------------
-- Called to register a semaphore that will be signalled whenever
-- a SIGIO occurs and the select shows that a read is possible on
-- the indicated file descriptor.
-- Also sets fd so that SIGIO occurs when state changes from
-- I/O not possible to I/O possible.
------------------------------------------------------------------
procedure Register_Read_File (Fd : Os_Files.File_Descriptor;
Sema : V_Semaphores.Binary_Semaphore_Id);
------------------------------------------------------------------
-- Called to unregister a read semaphore.
------------------------------------------------------------------
procedure Un_Register_Read_File (Fd : Os_Files.File_Descriptor);
------------------------------------------------------------------
-- Called to register a semaphore that will be signalled whenever
-- a SIGIO occurs and the select shows that a write is possible on
-- the indicated file descriptor.
-- Also sets fd so that SIGIO occurs when state changes from
-- I/O not possible to I/O possible.
------------------------------------------------------------------
procedure Register_Write_File (Fd : Os_Files.File_Descriptor;
Sema : V_Semaphores.Binary_Semaphore_Id);
------------------------------------------------------------------
-- Called to unregister a write semaphore.
------------------------------------------------------------------
procedure Un_Register_Write_File (Fd : Os_Files.File_Descriptor);
------------------------------------------------------------------
-- Called to register a semaphore that will be signalled whenever
-- a SIGIO occurs and the select shows that an exceptional condition
-- is present for the indicated file descriptor.
-- Also sets fd so that SIGIO occurs when state changes from
-- I/O not possible to I/O possible.
------------------------------------------------------------------
procedure Register_Exec_File (Fd : Os_Files.File_Descriptor;
Sema : V_Semaphores.Binary_Semaphore_Id);
------------------------------------------------------------------
-- Called to unregister a semaphore.
------------------------------------------------------------------
procedure Un_Register_Exec_File (Fd : Os_Files.File_Descriptor);
------------------------------------------------------------------
-- Count of the number of times a SIGIO occurs and there are no
-- pending i/o's for registered file descriptors.
------------------------------------------------------------------
Zero_Select_Status_Count : Integer := 0;
------------------------------------------------------------------
-- Set to false if the SIGIO handler encounters a problem (usually
-- with select) that prevents subsequent correct operation. When
-- set false from true, all registered semaphores are signalled.
------------------------------------------------------------------
Sigio_Task_Ok : Boolean := True;
------------------------------------------------------------------
-- Error code associated with a SIGIO handler failure. Contains
-- valid value only when sigio_task_ok is false.
------------------------------------------------------------------
Sigio_Task_Failure : Errno.Error_Codes;
------------------------------------------------------------------
-- Raised if registering a file descriptor for which there is already
-- a registration, or unregistering a file descriptor for which
-- there is no current registration.
------------------------------------------------------------------
Bad_Registration : exception;
------------------------------------------------------------------
-- Raised if the fcntl call to set SIGIO on as part of registering
-- or off as part of unregistering fails.
------------------------------------------------------------------
Bad_Ioctl_For_Sigio_State : exception;
end Sigio_Distributor;
with V_Semaphores;
with Os_Files;
with Errno;
-- body only with's
with Unchecked_Conversion;
with Os_Signal;
with System;
with Ada_Krn_Defs;
--with V_Sema;
with Unix_Time;
with Unix;
with Language;
with Modified_Fcntl;
--with Ubb_Unix_Io;
--with C_Types;
--with Text_Io; -- DEBUG
--use Text_Io;
package body Sigio_Distributor is
Bits_Per_Integer : constant := 32;
Max_Opened_File_Descriptors : constant := 10;
-- last index in last word with any file descriptor
Max_Opened_File_Descriptors_Ceiling : constant :=
(((Max_Opened_File_Descriptors - 1) / Bits_Per_Integer) + 1) *
Bits_Per_Integer - 1;
function To_Integer is
new Unchecked_Conversion
(Source => Os_Files.File_Descriptor, Target => Integer);
function "=" (Left, Right : in Errno.Error_Codes) return Boolean
renames Errno."=";
function "+" (Left, Right : in Os_Files.Open_Flags)
return Os_Files.Open_Flags renames Os_Files."+";
function C_Select (Socket_Descriptors_Num : in Integer;
Read_Socket_Descriptor_Ptr : in System.Address;
Write_Socket_Descriptor_Ptr : in System.Address;
Exec_Socket_Descriptor_Ptr : in System.Address;
Timeout_Ptr : in Unix_Time.Timeval_Ptr)
return Unix.Status_Code;
pragma Interface (C, C_Select);
pragma Interface_Name (C_Select, Language.C_Subp_Prefix & "select");
task Sigio_Handler is
entry Signal_Occurred;
for Signal_Occurred use at Ada_Krn_Defs.Intr_Entry_Init
(Intr_Vector => Os_Signal.Sigio,
Prio => System.Priority'Last);
-- pragma Passive (Abort_Unsafe,
-- Ada_Krn_Defs.Intr_Attr_Init
-- (Disable_Status => Ada_Krn_Defs.Disable_Intr_Status));
-- Set priority of code outside accept body if PASSIVE commented out
pragma Priority (System.Priority'Last);
end Sigio_Handler;
type Fd_Index is new Integer range 0 .. Max_Opened_File_Descriptors_Ceiling;
type Semaphores is array (Fd_Index) of V_Semaphores.Binary_Semaphore_Id;
Read_Semaphores : Semaphores;
Write_Semaphores : Semaphores;
Exec_Semaphores : Semaphores;
type Descriptors is array (Fd_Index) of Boolean;
for Descriptors'Size use Max_Opened_File_Descriptors_Ceiling + 1;
Save_Read_Descriptors : Descriptors := (others => False);
Read_Descriptors : Descriptors;
Save_Write_Descriptors : Descriptors := (others => False);
Write_Descriptors : Descriptors;
Save_Exec_Descriptors : Descriptors := (others => False);
Exec_Descriptors : Descriptors;
Timeout : Unix_Time.Timeval_Ptr :=
new Unix_Time.Timeval_Rec'(0, 0); -- No delay
Status : Unix.Status_Code;
Last_Status : Unix.Status_Code;
Error : Errno.Error_Codes;
Last_Error : Errno.Error_Codes;
Fcntl_Arg : Os_Files.Open_Flags;
-- -- Indexing into arrays must reverse bits in each integer to be
-- -- consistent with the C select mechanism, which uses bits numbered by
-- -- powers of 2, while on an HP, bits are numbered "high_endian".
-- -- On input we have bits 0..31 from the right. On output we have
-- -- bits 0..31 from the left (per integer) ;
-- -- We convert once and use the reversed index for both the bits in
-- -- the select descriptor bit arrays and the semaphore arrays, so
-- -- we do not need an inverse function (though to_index should be its
-- -- own inverse!).
-- function To_Index (I : in Natural) return Fd_Index is
-- Bpi : constant := Bits_Per_Integer;
-- begin
-- return Fd_Index ((I / Bpi) * Bpi + (Bpi - (I mod Bpi) - 1));
-- end To_Index;
-- pragma Inline_Only (To_Index);
task body Sigio_Handler is
-- Select_Status : C_Types.Int;
begin
loop
accept Signal_Occurred do
Status := Unix.Error;
Error := Errno.Eintr;
--Put_Line ("Sigio_Handler: accepted Signal_Occurred");
while Status < Unix.Ok and
(Error /= Last_Error or Error = Errno.Eintr) loop
--Put_Line ("Sigio_Handler: Status < Unix.Ok");
Last_Status := Status;
Last_Error := Error;
Read_Descriptors := Save_Read_Descriptors;
Write_Descriptors := Save_Write_Descriptors;
Exec_Descriptors := Save_Exec_Descriptors;
Status := C_Select (Max_Opened_File_Descriptors,
Read_Descriptors'Address, System.No_Addr,
System.No_Addr, Timeout);
-- Ubb_Unix_Io.Unix_Select
-- (Select_Status, Max_Opened_File_Descriptors,
-- Read_Descriptors'Address, System.No_Addr,
-- System.No_Addr, Timeout'Address);
-- Status := Integer (Select_Status);
--Write_Descriptors'Address,
--Exec_Descriptors'Address, Timeout);
Error := Errno.Errno;
end loop;
if Status < Unix.Ok then
-- Select is dead with a repeatable error that is not EINTR
--Put_Line
-- ("Sigio_Handler: Select is dead with a repeatable error that is not EINTR");
if Sigio_Task_Ok then
Sigio_Task_Ok := False;
Sigio_Task_Failure := Error;
-- signal all registered semaphores once
for I in Descriptors'Range loop
if Save_Read_Descriptors (I) then
V_Semaphores.Signal_Semaphore (Read_Semaphores (I));
end if;
if Save_Write_Descriptors (I) then
V_Semaphores.Signal_Semaphore (Write_Semaphores (I));
end if;
if Save_Exec_Descriptors (I) then
V_Semaphores.Signal_Semaphore (Exec_Semaphores (I));
end if;
end loop;
end if;
elsif Status > Unix.Ok then
--Put_Line ("Sigio_Handler: Status > Unix.Ok");
-- There is at least one "ready" registed fd
-- This is the usual case so this could do well to
-- be optimized, perhaps doing a word at a time and
-- only hitting bits if the word is non-zero.
-- Could also look at keeping the limits of which
-- descriptors have been registered and only checking
-- that bit range.
for I in Descriptors'Range loop
if Read_Descriptors (I) then
--Put_Line ("Sigio_Handler: Read_Descriptors (" &
-- Fd_Index'Image (I) & " ) = true");
V_Semaphores.Signal_Semaphore (Read_Semaphores (I));
--Put_Line ("Sigio_Handler: after read Signal_Semaphore");
end if;
if Write_Descriptors (I) then
--Put_Line ("Sigio_Handler: Write_Descriptors (" &
-- Fd_Index'Image (I) & " ) = true");
V_Semaphores.Signal_Semaphore (Write_Semaphores (I));
--Put_Line ("Sigio_Handler: after write Signal_Semaphore");
end if;
if Exec_Descriptors (I) then
--Put_Line ("Sigio_Handler: Exec_Descriptors (" &
-- Fd_Index'Image (I) & " ) = true");
V_Semaphores.Signal_Semaphore (Exec_Semaphores (I));
--Put_Line ("Sigio_Handler: after exec Signal_Semaphore");
end if;
end loop;
else
--Put_Line ("Sigio_Handler: Zero_Select_Status_Count =" &
-- Integer'Image (Zero_Select_Status_Count));
if Zero_Select_Status_Count /= Integer'Last then
Zero_Select_Status_Count := Zero_Select_Status_Count + 1;
end if;
end if;
end Signal_Occurred;
end loop;
end Sigio_Handler;
procedure Register (Fd : Os_Files.File_Descriptor;
Sema : V_Semaphores.Binary_Semaphore_Id;
Desc_List : in out Descriptors;
Sema_List : in out Semaphores) is
Index : Fd_Index := Fd_Index (To_Integer (Fd));
-- Index : Fd_Index := to_index (To_Integer (Fd));
begin
if Desc_List (Index) then
raise Bad_Registration;
end if;
Desc_List (Index) := True;
Sema_List (Index) := Sema;
-- set fd so that SIGIO goes to the local process
Fcntl_Arg := Os_Files.Open_Flags (Unix.Getpid);
Status := Modified_Fcntl.Fcntl (Fd, Modified_Fcntl.F_Setown, Fcntl_Arg);
if Status < Unix.Ok then
raise Bad_Ioctl_For_Sigio_State;
end if;
-- set fd so that SIGIO is generated on state change
Fcntl_Arg := Modified_Fcntl.Fasync + Modified_Fcntl.Fnonblocking;
Status := Modified_Fcntl.Fcntl (Fd, Modified_Fcntl.F_Setfl, Fcntl_Arg);
if Status < Unix.Ok then
raise Bad_Ioctl_For_Sigio_State;
end if;
end Register;
procedure Un_Register (Fd : Os_Files.File_Descriptor;
Desc_List : in out Descriptors) is
-- Index : Fd_Index := To_Index (To_Integer (Fd));
Index : Fd_Index := Fd_Index (To_Integer (Fd));
begin
if not Desc_List (Index) then
raise Bad_Registration;
end if;
Desc_List (Index) := False;
if not Save_Read_Descriptors (Index) and
not Save_Write_Descriptors (Index) and
not Save_Exec_Descriptors (Index) then
-- set fd so that SIGIO is not generated on state change
Fcntl_Arg := 0;
Status := Modified_Fcntl.Fcntl
(Fd, Modified_Fcntl.F_Setown, Fcntl_Arg);
if Status < Unix.Ok then
raise Bad_Ioctl_For_Sigio_State;
end if;
end if;
end Un_Register;
procedure Register_Read_File (Fd : Os_Files.File_Descriptor;
Sema : V_Semaphores.Binary_Semaphore_Id) is
begin
Register (Fd, Sema, Save_Read_Descriptors, Read_Semaphores);
end Register_Read_File;
procedure Un_Register_Read_File (Fd : Os_Files.File_Descriptor) is
begin
Un_Register (Fd, Save_Read_Descriptors);
end Un_Register_Read_File;
procedure Register_Write_File (Fd : Os_Files.File_Descriptor;
Sema : V_Semaphores.Binary_Semaphore_Id) is
begin
Register (Fd, Sema, Save_Write_Descriptors, Write_Semaphores);
end Register_Write_File;
procedure Un_Register_Write_File (Fd : Os_Files.File_Descriptor) is
begin
Un_Register (Fd, Save_Write_Descriptors);
end Un_Register_Write_File;
procedure Register_Exec_File (Fd : Os_Files.File_Descriptor;
Sema : V_Semaphores.Binary_Semaphore_Id) is
begin
Register (Fd, Sema, Save_Exec_Descriptors, Exec_Semaphores);
end Register_Exec_File;
procedure Un_Register_Exec_File (Fd : Os_Files.File_Descriptor) is
begin
Un_Register (Fd, Save_Exec_Descriptors);
end Un_Register_Exec_File;
end Sigio_Distributor;
------------------------------------------------------------------------------
-- Copyright (C) 1994 Westinghouse Electric Corporation --
-- All Rights Reserved. --
------------------------------------------------------------------------------
------------------------------------------------------------------------------
--
-- CSCI: Operational Computer Program
-- CSC: System Management
-- subCSC: Network Manager
-- Component: OYN_RMF_NETWORK Specification
-- File Name: OYNRMFNW
--
-- NARRATIVE
--
-- This package provides the Reliable Mu
|
| >>under Digital UNIX V3.2c. So, I wondering whether FDDI suffers the same
>>problem?
Yes, although the FDDI implementation is helped in that the ATM
signaling and other such work is not needed. Remember that Windows NT
knows *nothing* about ATM today, so the NDIS 3 device driver for ATM
has to be very intelligent, and subsequently, must do a lot more in the
critical run-time code during receive and transmit. To offset that,
ATM offers a faster data-link, and as Windows NT becomes more
ATM-literate, expect that gap to shrink.
As for DEFPA, point-point between two Alpha stations should offer
better than 40Mbps throughput. We've run the infamous bricks demo at
about that rate. Typically, FDDI server implementations are placed in
bridged 10 Mbps Ethernet to FDDI backbone network configurations where
it's not vital that the full FDDI line rate is achieved. To offer
hope, however, Microsoft is well aware of the performance impact of
some of the decisions made while architecting the NDIS 3 specification
and is pursuing changes to that spec for an upcoming release of NT.
Some of the changes will allow Bus Master DMA device drivers to pass up
DMA'd buffers directly on receive (no extraneous buffer copy) and to
queue up multiple transmit packets on one transmit request which should
help performance for protocols like TCP/IP.
- Larry
|