[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference azur::mcc

Title:	DECmcc user notes file. Does not replace IPMT.
Notice:	Use IPMT for problems. Newsletter location in note 6187
Moderator:	TAEC::BEROUD

Created:	Mon Aug 21 1989
Last Modified:	Wed Jun 04 1997
Last Successful Update:	Fri Jun 06 1997
Number of topics:	6497
Total number of notes:	27359

1531.0. "Event notification stopped, MCC_EVENT_LOG decode" by KAOT01::S_HYNDMAN () Thu Sep 19 1991 18:20


	I'm having some major problems with event notification.

First some background:

	We are being paid by a customer to monitor there phaseIV decnet
nodes in many plants across the country.  Their X25 routers are sinking the
decnet events to this digital facility where mcc alarms are used to notify our
operators.

The problem:
 
	At some point during our operation (usually less than 24 hours)
the alarm notification ceases to function.  A quick check shows that the 
enabling batch job is still running as well as the MCC_DNA4_EVL process
and EVL.  Although I can see from the console that events are arriving
and that they should be setting off alarms, nothing happens. (This is when
Digital starts paying the customer due to a penalty clause!)

	Note 807 contains the same problems we are seeing with MCC_DNA4_EVL
including 807.3's secondary interlock failure.  I'm using DCM to check for
the disappearence of mcc_DNA4_EVL and restart it.  I followed Matt's
recomendation to define MCC_EVENT_LOG 1 in my rule enabling command file to 
determine if the Event manager is getting notified.  This has provided some 
information, which sadly, is all greek to me.

	I will post as replys some examples of the log files.  Can someone
tell me what they mean or give some suggestions?  Unless DECmcc (or my use
of the product) becomes reliable, we will have to go to a different solution.
This would be unfortunate since the customer liked what they saw especially
the iconic map, but it has to be reliable.  

	For us at least:
		 working = money in
             not working = money out

 Being interested in a job continuation program I prefer the former.

T.R	Title	User	Personal Name	Date	Lines
1531.1	example log file with MCC_EVENT_LOG enabled	KAOT01::S_HYNDMAN		`Thu Sep 19 1991 18:23`	285
	Get Event was Alerted Maintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested EDQ_DEQW_SUBSCR_EVENT found alert termination request Maintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested EDQ_DEQW_SUBSCR_EVENT found alert termination request EDQ_DEQW_SUBSCR_EVENT found alert termination requestMaintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested EDQ_DEQW_SUBSCR_EVENT found alert termination request Maintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested EDQ_DEQW_SUBSCR_EVENT found alert termination request Maintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requestedEDQ_DEQW_SUBSCR_EVENT found alert termination request Maintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested EDQ_DEQW_SUBSCR_EVENT found alert termination request Get Event was Alerted Maintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested EDQ_DEQW_SUBSCR_EVENT found alert termination request Maintainer Thread was alerted during Timer Wait: Maintainer Thread was alerted during Timer Wait:EDQ_DEQW_SUBSCR_EVENT found alert termination request Get Event was Alerted Maintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested EDQ_DEQW_SUBSCR_EVENT found alert termination request Maintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested EDQ_DEQW_SUBSCR_EVENT found alert termination request Maintainer Thread was alerted during Timer Wait: Maintainer Thread was alerted during Timer Wait:EDQ_DEQW_SUBSCR_EVENT found alert termination request Maintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested EDQ_DEQW_SUBSCR_EVENT found alert termination request Maintainer Thread was alerted during Timer Wait: %MCC-E-ALERT_TERMREQ, thread termination requested EDQ_DEQW_SUBSCR_EVENT found alert termination request Maintainer Thread was alerted during Timer Wait: Maintainer Thread was alerted during Timer Wait:EDQ_DEQW_SUBSCR_EVENT found alert termination request %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested Get Event was Alerted %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested %MCC-E-ALERT_TERMREQ, thread termination requested Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queueGet Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: Failed to dequeue event:Get Event was Alerted Get Event was Alerted Get Event was Alerted %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queueGet Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: Failed to dequeue event:Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queueGet Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: Failed to dequeue event:Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queueGet Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queueGet Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: Failed to dequeue event:Get Event was Alerted Get Event was Alerted Get Event was Alerted %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queueGet Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: Failed to dequeue event:Get Event was Alerted Failed to dequeue event:Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: Failed to dequeue event:Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queueGet Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queueGet Event was Alerted Get Event was Alerted %MCC-E-NOMOREVT, no more events in event queue %MCC-E-NOMOREVT, no more events in event queueGet Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: %MCC-E-NOMOREVT, no more events in event queue Failed to dequeue event: Failed to dequeue event:Get Event was Alerted Get Event was Alerted Get Event was Alerted Get Event was Alerted %MCC-E-NOMOREVT, no more events in event queue Job MCC_ENABLE_ALL_ALARMS (queue SYS$BATCH, entry 988) started on SYS$BATCH SYSTEM job terminated at 19-SEP-1991 10:32:15.13
1531.2	example log file with MCC_EVENT_LOG enabled	KAOT01::S_HYNDMAN		`Thu Sep 19 1991 18:27`	153
	I noticed that the trace for the rule enabling changes as it gets to rules later in the procedure. This was something new. The job abort was due to me doing a del/entry inorder to get notification working again. $! SYLOGIN.COM $ set noverify DECmcc (V1.1.0) MCC 0 ALARMS RULE CALGARY_ADJACENCY_DOWN AT 19-SEP-1991 10:32:23 Tracing error paths in Event Manager Normal operation has begun. MCC 0 ALARMS RULE CALGARY_ADJACENCY_DOWN_OPERATOR_INITIATED AT 19-SEP-1991 10:32:24 Normal operation has begun. Waiting until -> 19-SEP-1991 10:32:27.19 Start_time = 19-SEP-1991 10:32:26.27 End_time = 6-APR-1992 10:32:26.27 FINAL wait time -> 6-APR-1992 10:32:26.27 MCC 0 ALARMS RULE CALGARY_ADJACENCY_UP AT 19-SEP-1991 10:32:27 Normal operation has begun. Start_time = 19-SEP-1991 10:32:27.60 End_time = 6-APR-1992 10:32:27.60 FINAL wait time -> 6-APR-1992 10:32:27.60 MCC 0 ALARMS RULE EDMONTON_ADJACENCY_DOWN AT 19-SEP-1991 10:32:28 Normal operation has begun. Start_time = 19-SEP-1991 10:32:28.75 End_time = 6-APR-1992 10:32:28.75 FINAL wait time -> 6-APR-1992 10:32:28.75 MCC 0 ALARMS RULE EDMONTON_ADJACENCY_DOWN_OPERATOR_INITIATED AT 19-SEP-1991 10:32:29 Normal operation has begun. Waiting until -> 19-SEP-1991 10:32:29.51 Start_time = 19-SEP-1991 10:32:29.44 End_time = 6-APR-1992 10:32:29.44 MCC 0 ALARMS RULE EDMONTON_ADJACENCY_UP AT 19-SEP-1991 10:32:29 Normal operation has begun. FINAL wait time -> 6-APR-1992 10:32:29.44 Waiting until -> 19-SEP-1991 10:32:30.14 Start_time = 19-SEP-1991 10:32:30.05 End_time = 6-APR-1992 10:32:30.05 FINAL wait time -> 6-APR-1992 10:32:30.05 MCC 0 ALARMS RULE HALIFAX_ADJACENCY_DOWN AT 19-SEP-1991 10:32:30 Normal operation has begun. MCC 0 ALARMS RULE HALIFAX_ADJACENCY_DOWN_OPERATOR_INITIATED AT 19-SEP-1991 10:32:31 Normal operation has begun. MCC 0 ALARMS RULE HALIFAX_ADJACENCY_UP AT 19-SEP-1991 10:32:31 Normal operation has begun. MCC 0 ALARMS RULE HAMILTON_ADJACENCY_DOWN_OPERATOR_INITIATED AT 19-SEP-1991 10:32:32 Normal operation has begun. MCC 0 ALARMS RULE HAMILTON_ADJACENCY_UP AT 19-SEP-1991 10:32:32 Normal operation has begun. MCC 0 ALARMS RULE HAMILTON_ADJACNECY_DOWN AT 19-SEP-1991 10:32:33 Waiting until -> 19-SEP-1991 10:32:30.86 Normal operation has begun. Waiting until -> 19-SEP-1991 10:32:31.46 MCC 0 ALARMS RULE DLM_X25-CCEXOUT_CIRCUIT_DOWN AT 19-SEP-1991 10:32:34 Normal operation has begun. Waiting until -> 19-SEP-1991 10:32:34.75 MCC 0 ALARMS RULE DLM_X25-CCEXOUT_CIRCUIT_DOWN_CIRCUIT_FAULT AT 19-SEP-1991 10:32:35 Normal operation has begun. Waiting until -> 19-SEP-1991 10:32:35.59 MCC 0 ALARMS RULE DLM_X25-CCEXOUT_CIRCUIT_UP AT 19-SEP-1991 10:32:36 Normal operation has begun. Waiting until -> 19-SEP-1991 10:32:36.42 MCC 0 ALARMS RULE DLM_X25-CCLXOUT_CIRCUIT_DOWN AT 19-SEP-1991 10:32:36 Normal operation has begun. Waiting until -> 19-SEP-1991 10:32:37.31 MCC 0 ALARMS RULE DLM_X25-CCLXOUT_CIRCUIT_DOWN_CIRCUIT_FAULT AT 19-SEP-1991 10:32:37 Normal operation has begun. Waiting until -> 19-SEP-1991 10:32:38.21 MCC 0 ALARMS RULE DLM_X25-CCLXOUT_CIRCUIT_DOWN_CIRCUIT_FAULT AT 19-SEP-1991 10:32:37 Normal operation has begun. Waiting until -> 19-SEP-1991 10:32:38.21 MCC 0 ALARMS RULE VANCOUVER_ADJACENCY_DOWN_OPERATOR_INITIATED AT 19-SEP-1991 10:33:36 Normal operation has begun. Waiting until -> 19-SEP-1991 10:33:37.21 MCC 0 ALARMS RULE VANCOUVER_ADJACENCY_UP AT 19-SEP-1991 10:33:37 Normal operation has begun. Waiting until -> 19-SEP-1991 10:33:38.38 %MCC-F-FATAL_FW, fatal framework condition: Failed to dequeue next ready thread %MCC-F-FATAL_FW, fatal framework condition: Failed to dequeue next ready thread %JBC-F-JOBABORT, job aborted during execution SYSTEM job terminated at 19-SEP-1991 12:22:08.42
1531.3	Yes, but what CAUSED the shutdown of the Event Requests?	TOOK::GUERTIN	Don't fight fire with flames	`Fri Sep 20 1991 10:10`	15
	Unfortunately, the information you posted is of very limited use. The information describes the "dead body" and the "smoking gun", but little else. It appears to me that something (or someone) initiated the shutdown of the Alarms FM (or maybe Notification?). What you have shown is that the Event Requests are being ask to abort execution (i.e., they are being "shot") by someone/something for some unknown reason. I would look higher up in the food chain. Did you say that you were getting Secondary Interlock Failures? I didn't see any such error messages. And which process is generating the Event trace messages? Iconic Map or Alarms? Having said that, what you describe rings a bell. Have you applied the patches (note 1267 -- I think)? -Matt.
1531.4	update	KAOT01::S_HYNDMAN		`Fri Sep 20 1991 14:49`	25
	I spoke with Matt off line but will update this note with my actions thus far. The MCC_EVENT_LOG data in the previous replies came from the logfile of the batch job which enables all my rules. The com file now set MCC_EVENT_TRACE 180 in addition to the MCC_EVENT_LOG 1. Also before exiting MCC and resubmitting the same job, I disable all rules. Perhaps this will be a cleaner exit. Also EVENT LOG and TRACE have been enabled for the mcc sessions the operators are using with the iconic map. This is being logged to a file. I had missed the patch for the Kernel from note 1267 so I installed that as well. All other patches were previously installed. Hopefully the traces will give an indication why the notification is going to sleep.(fingers crossed) I'll look for my MCC_DNA4_EVL logs with the Secondary Interlock Failure. Scott