[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference azur::mcc

Title:DECmcc user notes file. Does not replace IPMT.
Notice:Use IPMT for problems. Newsletter location in note 6187
Moderator:TAEC::BEROUD
Created:Mon Aug 21 1989
Last Modified:Wed Jun 04 1997
Last Successful Update:Fri Jun 06 1997
Number of topics:6497
Total number of notes:27359

6273.0. "DECmcc performance problem" by BUDDIE::KENWORTHY () Fri Apr 14 1995 00:42

I have a customer that is running DECmcc 1.3 (Polycenter 200) on an ULTRIX
system. The customer is complaining about very poor performance when they 
are polling every 90 seconds. The performance is so bad that all of the nodes 
are flashing red, then changing back one cycle behind. What I mean is, the 
software is saying the node is not responding but in actuality, the response 
is not being recorded in time.

The customer is seeing all of the nodes constantly changing state when they 
are not. When they change the polling time to 5 minutes the system is still 
very slow but they can live with the performance. What the problem is, that 
the 5 minute polling time is unacceptable.

Following is some information concerning the customers system:

The system is a 5000/240 with 128 meg of memory, running Ultrix 4.3a. The
    system is configured with 325 megabytes of swap space. The disk partitions 
    that are used for DECmcc are not overloaded. /usr/mcc is 66% full and 
    /netdata is 20% full.

The number of nodes being polled is 313, broken into 9 domains

I had the customer run 'uerf -S' and they have no hardware errors at all.
I had them look at the buffercache statistics and the read hit rate is
99.0%. I also had them run netstat -i every five seconds during the 90
second polling for one hour and they did not get any output or input errors.

I had the customer run vmstat every five seconds during the 90 second polling
time. The idle time went to zero and the user and system time was between
40-60%. There was some paging in but no paging out. The amount of free 
memory remaind the same. There was very little disk I/O.

The number of system calls jumped up considerably and context switches 
doubled.

I am going to ask the customer to get me some swap statistics.

Any ideas as to why the performance is so bad? Any suggestions as to whee
    to look for performance problems?
    
    regards
    jimk
T.RTitleUserPersonal
Name		DateLines
6273.1Swap informationBUDDIE::KENWORTHYMon Apr 17 1995 17:0823
    I wanted to add some swapping statistics that I had the customer get
    from the system while they were doing the 90 second polling and the
    system performance is so bad.
    
    381473k swap configured
    118113k reserved virtual address space
    	    272k used (160k text, 0k smem)
    	    381201k free, 0k wasted, 0k missing
    avail: 11910*32k 81*1k
    
    Not much that can be said from the above. System has plenty of swap
    space and in fact I had the customer run 'vmstat -S' during the polling
    time and there was no swapping taking place.
    
    As far as I can tell the 5000/240 and the Ultrix operating system is
    doing just fine and the problem is with DECmcc (POLYCENTER 200). Can
    anyone tell me where to look for possible problems with DECmcc?
    
    The customer is tring to get the system (Hardware and Software) flight
    certified by NASA and will be reaching a critical mass soon.
    
    regards
    jimk
6273.2AZUR::DURIFTue Apr 18 1995 10:4314
Hi Jim,

>>I have a customer that is running DECmcc 1.3 (Polycenter 200) on an ULTRIX
>>system. The customer is complaining about very poor performance when they 
>>are polling every 90 seconds. The performance is so bad that all of the nodes 
>>are flashing red, then changing back one cycle behind. What I mean is, the 
>>software is saying the node is not responding but in actuality, the response 
>>is not being recorded in time.

What are they polling ?? What type of entity ??

Regards,

Benoit
6273.3snmp entitiesBUDDIE::KENWORTHYTue Apr 18 1995 17:1427
    The customer is polling snmp entities on the network. All nodes in each
    domain are snmp entities except one domain. In this domain the nodes
    are just 'pinged'
    
    As it turn out there are 12 domains with the following layout:
    domain number		number of entities
    
    	1				190
    	2				48
    	3				4
    	4				15
    	5				32
    	6				8
    	7				3 (non snmp entities)
    	8				2
    	9				2
    	10				2
    	11				3
    	12				4
    
    The customer said that they are polling using an asterisk in each
    domain.
    
    I hope that this is of some help.
    
    regards
    jimk
6273.4AZUR::DURIFWed Apr 19 1995 11:0712
Hi Jim,

    
>>The customer said that they are polling using an asterisk in each
>>domain.

Are they using alarms rules or the IP Poller ?
In what command is the asterisk used ???

Benoit
6273.5Not really an asterixBUDDIE::KENWORTHYThu Apr 20 1995 17:2418
    Benoit,
    
    The customer explained to me that they are using the IP Poller in all
    of the domains. She said that they have two alarm rules that fire based
    on the results of the IP Poller.
    
    The alarm rules are IP_up and IP_down. They are ony looking for a state
    change. If a system is reported as down and the the IP Poller reports
    the system state has changed the alarm rule for IP_down fires to change
    the color of the icon for the respective node.
    
    I hope that this information is some help with solving my problem. If
    you haven't already guessed I am not a POLYCENTER 200 expert, so please
    bear with me.
    
    regards
    jimk