Conference abbott::mailworks-unix

1348.0. "System Freezes" by TAMARA::NEUMAN::Neumann (Stan Neumann) Wed Feb 12 1997 11:04

On a heavily loaded system, we have seen problems in which the
mail system effectively freezes for a few minutes (around
three minutes in the case we studied most closely) - after
this short freeze, the mail system will again start responding.
During the freeze, most other functions on the system continue
to work although some UNIX monitoring functions (such as
the top command, and ps auxw) will also freeze temporarily.

It appears that the operating system's virtual memory
manager has chosen to swap one of the MailWorks processes
out to disk, which effectively halts MailWorks until that
process can be swapped back in.  Note that there is a
difference between paging, in which individual pages of
memory are moved between physical memory and disk, and
swapping, in which an entire process is written
out to disk.  Paging is a normal part of a healthy
system; swapping takes place when the system feels that
it is working too hard at paging.  Paging will not
stop the system (alhtough it can slow it down somewhat),
whereas swapping temporarily stops the application being
swapped.

When this problem occurs, it can be avoided by adjusting
the kernel tuning parameter vm-free-page-optimal.
In the file sysconfigtab, add the following entry
vm:
	vm-free-page-optimal = 5
	
(note: if your sysconfigtab already has a vm: entry, place
the vm-free-page-optimal entry below it.)
After making this change, the operating system must be
rebooted for the change to take effect.

How can you tell if you are seeing this problem?
If you run a background process that consists only 
of the command:
    vmstat 10 > vmstat_logfile
then in the vmstat_logfile around the time of the 
temporary freeze, you will see several things happening:
* Shortly before the freeze, free space will drop to a very
  low level; usually the system idle time will also drop to
  a low level.
* At the time of the freeze, the number of waiting threads
  (the number in column 2 under "w") will drop suddenly,
  then this number will rise suddenly when the freeze
  is over.
* During the freeze, system idle time will usually rise
  significantly.
* As a side effect of the freeze, you may see broken pipes,
  network timeouts, and protocol errors in the MailWorks
  log files.

If this problem occurs, it may indicate that the system needs
more physical memory.  The tuning change above will avoid
the hard freeze of the system, but the system may still
be paging enough to affect the performance of the system. 

-Stan
================================================================
Some background from the system tuning and performance manual:

   If the number of pages in the free page list fall below the value
   associated with the vm-free-page-optimal attribute for more than
   five seconds, the task swapper (an extension of the page reclamation
   code) is activated.  The task swapper thread suspends processes,
   writes to disk all of the dirtied pages associated with the
   suspended processes, and places those pages on the free page list.

   The task swapper first swaps out all swappable tasks that have been idle
   for 30 seconds or more.  If this does not satisfy the memory demand,
   it begins swapping out the lowest priority tasks with the largest
   resident set size, one at a time, until the memory demands are
   satisifed (that is until the number of pages on the free page list
   reaches the value associated with the vm-page-free-target
   attribute.)