[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference smurf::ase

Title:ase
Moderator:SMURF::GROSSO
Created:Thu Jul 29 1993
Last Modified:Fri Jun 06 1997
Last Successful Update:Fri Jun 06 1997
Number of topics:2114
Total number of notes:7347

1910.0. "Cluster partitioning happend.TCR140" by TKTVFS::NAGAI_K (katsuya nagai) Mon Mar 03 1997 03:10

Hello,
Please give  me any sagestion�� 

I have a serious problem at a customer site last week.
I did have a cluster partition in TruCluster V1.4 enviroment.
OPS raw data had been lost.

I wants to do escalation of a problem, but
there was not enough information, I wrote it here. 

We have flowing infomation.   daemon.log  and  messages

I want you to teach where is  a problem begins with 

The customer confguration is this.

         +-------------+         +------------+
         | red1 (tu0   |	 | red2(tu0)  |
         | green1(tu1) |	 | green2(tu1)|
         | inner1(mc0) |--HUB--- | inner2(mc1)|
         +-------------+         +------------+
                      |             |
		      +----DRD------+

	Digital UNIX V4.0A
		    TCR140
	All guru patches has been installed.

###########/  etc/hosts #################################
/etc/hosts
172.16.1.1	red1
172.16.2.1	green1
172.16.7.1	inner1

172.16.1.2	red2
172.16.2.2	green2
172.16.7.1	inner2

172.16.7.42       	cluster_cnx

########### log file ( Mixed ) #############################

RED2's daemon.log
V  RED1's daemon.log    ( ........ red1 vmunix .. is a messages in just time  )
   V
13:08:13 red2 prestoctl_svc: Ignoring invalid arguments.
13:08:13 red2 prestoctl_svc: Warning: cannot open /dev/pr0: No such file or directory

13:08:14 red2 bssd:[9150]: Another BSSD is already running, exiting.
13:08:14 red2 cnxagentd: starting
13:08:14 red2 cnxmond: registration reports already in use
13:08:14 red2 cnxmond: registration retry cleanup failed
........ red2 vmunix: cnxlock: releasing director lock
13:08:14 red2 cnxmond: changed alias with : /sbin/ifconfig mc0 -alias
........ red2 vmunix: arp: local IP address 172.16.7.42 in use by hardware 
						address 40:00:00:00:00:00  
   ........ red1 vmunix: cnxlock: acquired director lock: entering CNX_RUN state
   13:08:14 red1 cnxmond: changed alias with : /sbin/ifconfig mc0 alias 
					    172.16.7.42 netmask 255.255.255.0
   ........ red1 vmunix arp: local IP address 172.16.7.42 in use by hardware 
						address 40-00-00-00-00-02  
   13:08:14 red1 cnxmgrd: starting
   13:08:16 red1 cnxpingd: comm err with node monitor: : RPC: Can't decode result: elapsed 0
   13:08:16 red1 cnxmond: mark active inner1
   ........ red1 vmunix: cnxagent: suspending

   13:09:14 red1 cnxmond: recovery delay completed
   13:09:14 red1 cnxmond: config change: notify director
   13:09:14 red1 cnxmgrd: attempting cluster recovery/formation
   13:09:14 red1 cnxmgrd: recovery, considering inner1
   13:09:14 red1 cnxmgrd: node inner1, cluster incarn 33110fcd, update_seq 3
   13:09:14 red1 cnxmgrd: found member of cluster incarnation 33110fcd
   13:09:14 red1 cnxmgrd: node is inner1, update_seq is 3
   13:09:14 red1 cnxmgrd: same update_seq 3, node inner1
   13:09:14 red1 cnxmgrd: using inner1
   13:09:14 red1 cnxmond: director acknowleged node inner1
   ........ red1 vmunix: dlm_agent: suspending lock activity
   13:09:14 red1 cnxmgrd: recovering a cluster
   13:09:14 red1 cnxmgrd: starting recovery update
   ........ red1 vmunix: cnx_agent: removed node inner2
   ........ red1 vmunix: cnx_agent: disconnecting channel to inner2
   13:09:14 red1 cnxmgrd: update complete, summary follows
   13:09:14 red1 cnxmgrd:   members are:
   13:09:14 red1 cnxmgrd:    inner1
   13:09:14 red1 cnxmgrd:   timed out are:
   13:09:14 red1 cnxmgrd:    none
   13:09:17 red1 cnxmgrd: finished recovery update, update_seq 4
   13:09:17 red1 cnxmgrd: completed cluster recovery/formation
   ........ red1 vmunix: dlm_agent: resuming lock activity
   ........ red1 vmunix: cnxagent: resuming
13:10:22 red2 cnxmond: mark timeout inner1: remove delay: 2
13:10:22 red2 cnxmond: node inner1 timed out

13:10:42 red2 cnxmond: mark inactive inner1
13:10:42 red2 cnxmond: config change: notify director
13:10:42 red2 cnxmgrd: starting removal operation
........ red2 vmunix: dlm_agent: suspending lock activity
13:10:42 red2 cnxmgrd: removing node inner1
13:10:42 red2 cnxmond: removing node inner1 from watch list
........ red2 vmunix: cnx_agent: removed node inter1
........ red2 vmunix: cnx_agent: disconnectiong channel to inner1
13:10:42 red2 cnxmgrd: update complete, summary follows
13:10:42 red2 cnxmgrd:   members are:
13:10:42 red2 cnxmgrd:    inner2
13:10:42 red2 cnxmgrd:   timed out are:
13:10:42 red2 cnxmgrd:    none
13:10:43 red2 cnxmgrd: finished removal operation, update_seq 4
........ red2 vmunix: dlm_agent: resuming lock activity
T.RTitleUserPersonal
Name		DateLines
1910.1questions after questions ...VIRGIN::SUTTERWho are you ??? - I'm BATMAN !!!Mon Mar 03 1997 05:3523
What 'HUB' is involved? 


> /etc/hosts
> 172.16.1.1	red1
> 172.16.2.1	green1
> 172.16.7.1	inner1
> 
> 172.16.1.2	red2
> 172.16.2.2	green2
> 172.16.7.1	inner2
> 
> 172.16.7.42       	cluster_cnx

What are your netmasks? 

'inner1' and 'inner2' have the SAME IP-Adresses, make sure you make 'inner2' 
to be 172.16.7.2 !!

How are the clients connected to the Ethernets? 

Does 'clu_ivp' show any errors/problems?
1910.2What happened before these daemon.log entries?NETRIX::"[email protected]"Gregory P. MyrdalMon Mar 03 1997 11:368
From the daemon.log entries that was included in .0 it sure appears
that the system(s) already think that are running services.  This
does not appear to be a clean reboot, what was going on before these
entries that we were given?

-- Greg
[Posted by WWW Notes gateway]
1910.3TKTVFS::NAGAI_Kkatsuya nagaiTue Mar 04 1997 03:0642
Thank you for your response. 

> What 'HUB' is involved? 

We are using HUB( CCMHA-AA ). not a virtial HUB.

>> /etc/hosts
>> 172.16.1.1	red1
>> 172.16.2.1	green1
>> 172.16.7.1	inner1
> 
>> 172.16.1.2	red2
>> 172.16.2.2	green2
>> 172.16.7.1	inner2
> 
>> 172.16.7.42       	cluster_cnx

>What are your netmasks? 

The netmask is 255.255.255.0

>'inner1' and 'inner2' have the SAME IP-Adresses, make sure you make 'inner2' 
> to be 172.16.7.2 !!

sorry, I wrote it incorrectly. the custmore site is no mistake.

> How are the clients connected to the Ethernets? 

I don't understand it well in this respect.

> Does 'clu_ivp' show any errors/problems? 

There is not an error at all, and this system worked 4 months  without
problem  as OPS system.






 Katsuya Nagai
1910.4TKTVFS::NAGAI_Kkatsuya nagaiTue Mar 04 1997 03:3725
Thank you for your help

>From the daemon.log entries that was included in .0 it sure appears
>that the system(s) already think that are running services.  This
>does not appear to be a clean reboot, what was going on before these
>entries that we were given?
>-- Greg

This is a clean reboot. We checked a boot messages and shutdown messages in a
binary.errlog file.

I attach original daemon.log ( red1 and red2 ) next reply.
I suspect somebody may be activate these daemon intentionally on node red2.

>13:08:14 red2 bssd:[9150]: Another BSSD is already running, exiting.
>13:08:14 red2 cnxagentd: starting
>13:08:14 red2 cnxmond: registration reports already in use
>13:08:14 red2 cnxmond: registration retry cleanup failed

During normal operation,If we execute these daemons ( bssd,cnxagentd,cnxmond )
with a manual, How will it have happend? 
 
  Katsuya Nagai
1910.5red1 daemon.logTKTVFS::NAGAI_Kkatsuya nagaiTue Mar 04 1997 03:47196
This is a red1's daemon.log
This system booted at Feb 24 12:51:19  ( in a binary.log )

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feb 24 12:51:21 red1 cnxpingd: starting
Feb 24 12:51:21 red1 cnxpingd: waiting to register with kernel agent
Feb 24 12:51:21 red1 cnxagentd: starting
Feb 24 12:51:36 red1 mountd[1635]: startup
Feb 24 12:51:36 red1 mountd[1635]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:51:40 red1 automount[1878]: No file systems to serve
Feb 24 12:51:41 red1 ASE: local HSM Notice: Able to ping inner2 over the network
Feb 24 12:51:41 red1 ASE: local HSM Notice: Able to ping inner2 over the SCSI bus
Feb 24 12:51:42 red1 ASE: local HSM ***ALERT: HSM_PATH_STATUS:172.16.7.2:UP
Feb 24 12:51:42 red1 ASE: local HSM Notice: member inner2 is UP
Feb 24 12:51:42 red1 xntpd[1972]: xntpd version=3.4x
Feb 24 12:51:42 red1 xntpd[1972]: tickadj = 1, tick = 976, tvu_maxslew = 1023
Feb 24 12:51:42 red1 xntpd[1972]: precision = 976 usec
Feb 24 12:51:43 red1 ASE: inner1 Agent Notice: initializing agent... stopping all services
Feb 24 12:51:49 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/IGES: not currently mounted
Feb 24 12:51:49 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/IGES already unmounted
Feb 24 12:51:49 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMAP: not currently mounted
Feb 24 12:51:49 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMAP already unmounted
Feb 24 12:51:49 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMD: not currently mounted
Feb 24 12:51:49 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMD already unmounted
Feb 24 12:51:49 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/add: not currently mounted
Feb 24 12:51:49 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/add already unmounted
Feb 24 12:51:49 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/font: not currently mounted
Feb 24 12:51:49 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/font already unmounted
Feb 24 12:51:50 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/job/batch3: not currently mounted
Feb 24 12:51:50 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/job/batch3 already unmounted
Feb 24 12:51:50 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/log/batch3: not currently mounted
Feb 24 12:51:50 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/log/batch3 already unmounted
Feb 24 12:51:50 red1 ASE: inner1 Agent Notice: user script: batch service is not running.
Feb 24 12:51:50 red1 ASE: inner1 Agent Notice: user script: Can not operator batch server
Feb 24 12:51:51 red1 utxd[2790]: reading utxd configuration file /etc/utxd.conf
Feb 24 12:51:51 red1 utxd[2790]: line 1: activate service 'odl'
Feb 24 12:51:51 red1 utxd[2790]: line 2: activate service 'sim'
Feb 24 12:51:51 red1 utxd[2790]: line 3: activate service 'kkc'
Feb 24 12:51:51 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/info/batch: not currently mounted
Feb 24 12:51:51 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/info/batch already unmounted
Feb 24 12:51:52 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/job/batch: not currently mounted
Feb 24 12:51:52 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/job/batch already unmounted
Feb 24 12:51:52 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/log/batch: not currently mounted
Feb 24 12:51:52 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/log/batch already unmounted
Feb 24 12:51:53 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/bin: not currently mounted
Feb 24 12:51:53 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/bin already unmounted
Feb 24 12:51:53 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/csh: not currently mounted
Feb 24 12:51:53 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/csh already unmounted
Feb 24 12:51:53 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/include: not currently mounted
Feb 24 12:51:53 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/include already unmounted
Feb 24 12:51:54 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/lib: not currently mounted
Feb 24 12:51:54 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/lib already unmounted
Feb 24 12:51:54 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/msg: not currently mounted
Feb 24 12:51:54 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/msg already unmounted
Feb 24 12:51:54 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/sys: not currently mounted
Feb 24 12:51:54 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/sys already unmounted
Feb 24 12:51:54 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/tools: not currently mounted
Feb 24 12:51:54 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/tools already unmounted
Feb 24 12:51:54 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/users/sys: not currently mounted
Feb 24 12:51:54 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/users/sys already unmounted
Feb 24 12:51:55 red1 ASE: inner1 Agent Notice: user script: rpc service is not running.
Feb 24 12:51:55 red1 ASE: inner1 Agent Notice: user script: Can not operator rpc server
Feb 24 12:51:56 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/info/rpc: not currently mounted
Feb 24 12:51:56 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/info/rpc already unmounted
Feb 24 12:51:56 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/job/rpc: not currently mounted
Feb 24 12:51:56 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/job/rpc already unmounted
Feb 24 12:51:56 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/rpc: not currently mounted
Feb 24 12:51:56 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/rpc already unmounted
Feb 24 12:51:56 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/transfer: not currently mounted
Feb 24 12:51:56 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/transfer already unmounted
Feb 24 12:51:59 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/job/batch2: not currently mounted
Feb 24 12:51:59 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/job/batch2 already unmounted
Feb 24 12:51:59 red1 SWXCR[3285]: Monitor Daemon Started.
Feb 24 12:51:59 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/log/batch2: not currently mounted
Feb 24 12:51:59 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/log/batch2 already unmounted
Feb 24 12:51:59 red1 ASE: inner1 Agent Notice: user script: event service is not running.
Feb 24 12:51:59 red1 ASE: inner1 Agent Notice: user script: Can not operator event server
Feb 24 12:52:00 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/info/event: not currently mounted
Feb 24 12:52:00 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/info/event already unmounted
Feb 24 12:52:01 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/log/event: not currently mounted
Feb 24 12:52:01 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/log/event already unmounted
Feb 24 12:52:02 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/data/print: not currently mounted
Feb 24 12:52:02 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/data/print already unmounted
Feb 24 12:52:02 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/info/print: not currently mounted
Feb 24 12:52:02 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/info/print already unmounted
Feb 24 12:52:02 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/job/print: not currently mounted
Feb 24 12:52:02 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/job/print already unmounted
Feb 24 12:52:02 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/log/print: not currently mounted
Feb 24 12:52:02 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/log/print already unmounted
Feb 24 12:52:03 red1 ASE: inner1 Agent Notice: user script: ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address
Feb 24 12:52:03 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/log: not currently mounted
Feb 24 12:52:03 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/log already unmounted
Feb 24 12:52:03 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/perf: not currently mounted
Feb 24 12:52:03 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/perf already unmounted
Feb 24 12:52:03 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/sys: not currently mounted
Feb 24 12:52:03 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/sys already unmounted
Feb 24 12:52:04 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/syslog: not currently mounted
Feb 24 12:52:04 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/syslog already unmounted
Feb 24 12:52:04 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/tmp: not currently mounted
Feb 24 12:52:04 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/tmp already unmounted
Feb 24 12:52:05 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/md: not currently mounted
Feb 24 12:52:05 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/md already unmounted
Feb 24 12:52:05 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/od: not currently mounted
Feb 24 12:52:05 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/od already unmounted
Feb 24 12:52:05 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/release: not currently mounted
Feb 24 12:52:05 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/release already unmounted
Feb 24 12:52:05 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd1: not currently mounted
Feb 24 12:52:05 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd1 already unmounted
Feb 24 12:52:05 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd2: not currently mounted
Feb 24 12:52:05 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd2 already unmounted
Feb 24 12:52:06 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/stub: not currently mounted
Feb 24 12:52:06 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/stub already unmounted
Feb 24 12:52:06 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/users: not currently mounted
Feb 24 12:52:06 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/users already unmounted
Feb 24 12:52:07 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/data/app: not currently mounted
Feb 24 12:52:07 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/data/app already unmounted
Feb 24 12:52:07 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/info/app: not currently mounted
Feb 24 12:52:07 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/info/app already unmounted
Feb 24 12:52:07 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/app: not currently mounted
Feb 24 12:52:07 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/app already unmounted
Feb 24 12:52:07 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/users: not currently mounted
Feb 24 12:52:07 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/users already unmounted
Feb 24 12:52:07 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/log/app: not currently mounted
Feb 24 12:52:07 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/log/app already unmounted
Feb 24 12:52:08 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/log: not currently mounted
Feb 24 12:52:08 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/log already unmounted
Feb 24 12:52:08 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/perf: not currently mounted
Feb 24 12:52:08 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/perf already unmounted
Feb 24 12:52:08 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/sys: not currently mounted
Feb 24 12:52:08 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/sys already unmounted
Feb 24 12:52:09 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/syslog: not currently mounted
Feb 24 12:52:09 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/syslog already unmounted
Feb 24 12:52:09 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/tmp: not currently mounted
Feb 24 12:52:09 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/tmp already unmounted
Feb 24 12:52:24 red1 ASE: inner2 Agent Notice: stopping service sdata
Feb 24 12:52:24 red1 ASE: inner2 Agent Notice: stopping service batch
Feb 24 12:52:24 red1 ASE: inner2 Agent Notice: stopping service system
Feb 24 12:52:24 red1 ASE: inner2 Agent Notice: stopping service rpc
Feb 24 12:52:24 red1 ASE: inner2 Agent Notice: stopping service transfer
Feb 24 12:52:24 red1 ASE: inner2 Agent Notice: stopping service lred1
Feb 24 12:52:24 red1 ASE: inner2 Director Notice: stopped service transfer on inner2
Feb 24 12:52:24 red1 ASE: inner1 Agent Notice: starting service transfer
Feb 24 12:52:25 red1 ASE: inner2 Director Notice: started service transfer on inner1
Feb 24 12:52:26 red1 ASE: inner2 Director Notice: stopped service lred1 on inner2
Feb 24 12:52:26 red1 ASE: inner1 Agent Notice: starting service lred1
Feb 24 12:52:26 red1 ASE: inner2 Director Notice: stopped service batch on inner2
Feb 24 12:52:26 red1 ASE: inner1 Agent Notice: starting service batch
Feb 24 12:52:26 red1 ASE: inner2 Director Notice: stopped service sdata on inner2
Feb 24 12:52:26 red1 ASE: inner1 Agent Notice: starting service sdata
Feb 24 12:52:26 red1 ASE: inner2 Director Notice: stopped service rpc on inner2
Feb 24 12:52:27 red1 ASE: inner1 Agent Notice: starting service rpc
Feb 24 12:52:27 red1 ASE: inner2 Director Notice: stopped service system on inner2
Feb 24 12:52:27 red1 ASE: inner1 Agent Notice: starting service system
Feb 24 12:52:32 red1 mountd[1696]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:52:33 red1 ASE: inner2 Director Notice: started service lred1 on inner1
Feb 24 12:52:33 red1 mountd[1696]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:52:34 red1 mountd[1696]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:52:34 red1 ASE: inner2 Director Notice: started service sdata on inner1
Feb 24 12:52:35 red1 ASE: inner2 Director Notice: started service system on inner1
Feb 24 12:52:35 red1 mountd[1696]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:52:36 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_run_sh start: /autostart/S59_BatSvr.sh: /Map/bin/S59_BatSvr started. 6253
Feb 24 12:52:36 red1 ASE: inner2 Director Notice: started service batch on inner1
Feb 24 12:52:36 red1 ASE: inner1 Agent Notice: /var/ase/sbin/ase_run_sh rpc: /autostart/S54_DCSvr.sh: /Map/bin/S54_DCSvr started. 6291
Feb 24 12:52:36 red1 ASE: inner2 Director Notice: started service rpc on inner1
Feb 24 12:52:36 red1 ASE: inner2 Director Notice: agent on inner1 came ONLINE
Feb 24 12:55:59 red1 xntpd[1972]: synchronized to LOCAL(1), stratum=3
Feb 24 13:02:51 red1 mountd[1696]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 13:08:14 red1 cnxmond: changed alias with : /sbin/ifconfig mc0 alias 172.16.7.42 netmask 255.255.255.0
Feb 24 13:08:14 red1 cnxmgrd: starting
Feb 24 13:08:16 red1 cnxpingd: comm err with node monitor: : RPC: Can't decode result: elapsed 0
Feb 24 13:08:16 red1 cnxmond: mark active inner1
Feb 24 13:09:13 red1 mountd[1696]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 13:09:14 red1 cnxmond: recovery delay completed
Feb 24 13:09:14 red1 cnxmond: config change: notify director
Feb 24 13:09:14 red1 cnxmgrd: attempting cluster recovery/formation
Feb 24 13:09:14 red1 cnxmgrd: recovery, considering inner1
Feb 24 13:09:14 red1 cnxmgrd: node inner1, cluster incarn 33110fcd, update_seq 3
Feb 24 13:09:14 red1 cnxmgrd: found member of cluster incarnation 33110fcd
Feb 24 13:09:14 red1 cnxmgrd: node is inner1, update_seq is 3
Feb 24 13:09:14 red1 cnxmgrd: same update_seq 3, node inner1
Feb 24 13:09:14 red1 cnxmgrd: using inner1
Feb 24 13:09:14 red1 cnxmond: director acknowleged node inner1
Feb 24 13:09:14 red1 cnxmgrd: recovering a cluster
Feb 24 13:09:14 red1 cnxmgrd: starting recovery update
Feb 24 13:09:14 red1 cnxmgrd: update complete, summary follows
Feb 24 13:09:14 red1 cnxmgrd:   members are: 
Feb 24 13:09:14 red1 cnxmgrd:    inner1
Feb 24 13:09:14 red1 cnxmgrd:   timed out are: 
Feb 24 13:09:14 red1 cnxmgrd:    none
Feb 24 13:09:17 red1 cnxmgrd: finished recovery update, update_seq 4
Feb 24 13:09:17 red1 cnxmgrd: completed cluster recovery/formation
Feb 24 14:10:34 red1 mountd[1696]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 14:32:22 red1 mountd[1696]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 25 08:25:45 red1 ASE: inner2 Director Warning: Director exiting...
Feb 25 08:25:45 red1 ASE: inner2 Agent Warning: aseagent exiting on request...
1910.6red2 daemon.logTKTVFS::NAGAI_Kkatsuya nagaiTue Mar 04 1997 03:50413
This is a red2's daemon.log.
This system booted at 12:48:32 ( in a binary.log ).
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Feb 24 12:48:33 red2 cnxpingd: starting
Feb 24 12:48:33 red2 cnxmond: changed alias with : /sbin/ifconfig mc0 alias 172.16.7.42 netmask 255.255.255.0
Feb 24 12:48:33 red2 cnxpingd: waiting to register with kernel agent
Feb 24 12:48:34 red2 cnxagentd: starting
Feb 24 12:48:34 red2 cnxmgrd: starting
Feb 24 12:48:37 red2 cnxmond: mark active inner2
Feb 24 12:48:49 red2 mountd[1681]: startup
Feb 24 12:48:49 red2 mountd[1681]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:48:53 red2 ASE: local HSM Warning: Can't ping inner1 over the SCSI bus
Feb 24 12:48:58 red2 ASE: local HSM Warning: Can't ping inner1 over the network
Feb 24 12:48:59 red2 ASE: local HSM ***ALERT: HSM_PATH_STATUS:172.16.7.1:DOWN
Feb 24 12:48:59 red2 ASE: local HSM Warning: member inner1 is DOWN
Feb 24 12:49:00 red2 ASE: inner2 Agent Notice: initializing agent... stopping all services
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/IGES: not currently mounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/IGES already unmounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMAP: not currently mounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMAP already unmounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMD: not currently mounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMD already unmounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/add: not currently mounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/add already unmounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/font: not currently mounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/font already unmounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/job/batch3: not currently mounted
Feb 24 12:49:05 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/job/batch3 already unmounted
Feb 24 12:49:06 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/log/batch3: not currently mounted
Feb 24 12:49:06 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/log/batch3 already unmounted
Feb 24 12:49:06 red2 ASE: inner2 Agent Notice: user script: batch service is not running.
Feb 24 12:49:06 red2 ASE: inner2 Agent Notice: user script: Can not operator batch server
Feb 24 12:49:07 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/info/batch: not currently mounted
Feb 24 12:49:07 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/info/batch already unmounted
Feb 24 12:49:07 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/job/batch: not currently mounted
Feb 24 12:49:07 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/job/batch already unmounted
Feb 24 12:49:07 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/log/batch: not currently mounted
Feb 24 12:49:07 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/log/batch already unmounted
Feb 24 12:49:08 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/bin: not currently mounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/bin already unmounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/csh: not currently mounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/csh already unmounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/include: not currently mounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/include already unmounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/lib: not currently mounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/lib already unmounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/msg: not currently mounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/msg already unmounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/sys: not currently mounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/sys already unmounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/tools: not currently mounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/tools already unmounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/users/sys: not currently mounted
Feb 24 12:49:09 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/users/sys already unmounted
Feb 24 12:49:10 red2 ASE: inner2 Agent Notice: user script: rpc service is not running.
Feb 24 12:49:10 red2 ASE: inner2 Agent Notice: user script: Can not operator rpc server
Feb 24 12:49:11 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/info/rpc: not currently mounted
Feb 24 12:49:11 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/info/rpc already unmounted
Feb 24 12:49:11 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/job/rpc: not currently mounted
Feb 24 12:49:11 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/job/rpc already unmounted
Feb 24 12:49:11 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/rpc: not currently mounted
Feb 24 12:49:11 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/rpc already unmounted
Feb 24 12:49:11 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/transfer: not currently mounted
Feb 24 12:49:11 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/transfer already unmounted
Feb 24 12:49:12 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/job/batch2: not currently mounted
Feb 24 12:49:12 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/job/batch2 already unmounted
Feb 24 12:49:12 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/log/batch2: not currently mounted
Feb 24 12:49:12 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/log/batch2 already unmounted
Feb 24 12:49:12 red2 ASE: inner2 Agent Notice: user script: event service is not running.
Feb 24 12:49:12 red2 ASE: inner2 Agent Notice: user script: Can not operator event server
Feb 24 12:49:13 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/info/event: not currently mounted
Feb 24 12:49:13 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/info/event already unmounted
Feb 24 12:49:13 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/log/event: not currently mounted
Feb 24 12:49:13 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/log/event already unmounted
Feb 24 12:49:14 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/data/print: not currently mounted
Feb 24 12:49:14 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/data/print already unmounted
Feb 24 12:49:14 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/info/print: not currently mounted
Feb 24 12:49:15 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/info/print already unmounted
Feb 24 12:49:15 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/job/print: not currently mounted
Feb 24 12:49:15 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/job/print already unmounted
Feb 24 12:49:15 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/log/print: not currently mounted
Feb 24 12:49:15 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/log/print already unmounted
Feb 24 12:49:15 red2 ASE: inner2 Agent Notice: user script: ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address
Feb 24 12:49:16 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/log: not currently mounted
Feb 24 12:49:16 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/log already unmounted
Feb 24 12:49:16 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/perf: not currently mounted
Feb 24 12:49:16 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/perf already unmounted
Feb 24 12:49:16 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/sys: not currently mounted
Feb 24 12:49:16 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/sys already unmounted
Feb 24 12:49:16 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/syslog: not currently mounted
Feb 24 12:49:16 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/syslog already unmounted
Feb 24 12:49:16 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/tmp: not currently mounted
Feb 24 12:49:16 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/tmp already unmounted
Feb 24 12:49:17 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/md: not currently mounted
Feb 24 12:49:17 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/md already unmounted
Feb 24 12:49:17 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/od: not currently mounted
Feb 24 12:49:17 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/od already unmounted
Feb 24 12:49:17 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/release: not currently mounted
Feb 24 12:49:17 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/release already unmounted
Feb 24 12:49:17 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd1: not currently mounted
Feb 24 12:49:17 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd1 already unmounted
Feb 24 12:49:18 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd2: not currently mounted
Feb 24 12:49:18 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd2 already unmounted
Feb 24 12:49:18 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/stub: not currently mounted
Feb 24 12:49:18 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/stub already unmounted
Feb 24 12:49:18 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/users: not currently mounted
Feb 24 12:49:18 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/users already unmounted
Feb 24 12:49:19 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/data/app: not currently mounted
Feb 24 12:49:19 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/data/app already unmounted
Feb 24 12:49:19 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/info/app: not currently mounted
Feb 24 12:49:19 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/info/app already unmounted
Feb 24 12:49:19 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/app: not currently mounted
Feb 24 12:49:19 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/app already unmounted
Feb 24 12:49:19 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/users: not currently mounted
Feb 24 12:49:19 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/users already unmounted
Feb 24 12:49:19 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/log/app: not currently mounted
Feb 24 12:49:19 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/log/app already unmounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/log: not currently mounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/log already unmounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/perf: not currently mounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/perf already unmounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/sys: not currently mounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/sys already unmounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/syslog: not currently mounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/syslog already unmounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/tmp: not currently mounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/tmp already unmounted
Feb 24 12:49:20 red2 ASE: inner2 Agent Notice: starting a new director
Feb 24 12:49:21 red2 ASE: local Director ***ALERT: Member inner1 is not available
Feb 24 12:49:33 red2 cnxmond: recovery delay completed
Feb 24 12:49:33 red2 cnxmond: config change: notify director
Feb 24 12:49:33 red2 cnxmgrd: attempting cluster recovery/formation
Feb 24 12:49:33 red2 cnxmgrd: recovery, considering inner2
Feb 24 12:49:33 red2 cnxmgrd: node inner2, cluster incarn 0, update_seq 0
Feb 24 12:49:33 red2 cnxmgrd: node inner2 not a member
Feb 24 12:49:33 red2 cnxmond: disabling node inner2 in watch list
Feb 24 12:49:33 red2 cnxmond: mark timeout inner2: remove delay: 3
Feb 24 12:49:33 red2 cnxmond: removing node inner2 from watch list
Feb 24 12:49:33 red2 cnxmgrd: forming a cluster
Feb 24 12:49:33 red2 cnxmgrd: completed cluster recovery/formation
Feb 24 12:49:36 red2 ASE: inner2 Agent Notice: starting service sdata
Feb 24 12:49:36 red2 ASE: inner2 Agent Notice: starting service batch
Feb 24 12:49:36 red2 ASE: inner2 Agent Notice: starting service system
Feb 24 12:49:36 red2 ASE: inner2 Agent Notice: starting service rpc
Feb 24 12:49:36 red2 ASE: inner2 Agent Notice: starting service batch2
Feb 24 12:49:37 red2 ASE: inner2 Agent Notice: starting service event
Feb 24 12:49:37 red2 ASE: inner2 Agent Notice: starting service print
Feb 24 12:49:37 red2 ASE: inner2 Agent Notice: starting service transfer
Feb 24 12:49:37 red2 ASE: inner2 Agent Notice: starting service lred2
Feb 24 12:49:37 red2 ASE: inner2 Agent Notice: starting service develop
Feb 24 12:49:37 red2 ASE: inner2 Agent Notice: starting service users
Feb 24 12:49:37 red2 ASE: inner2 Agent Notice: starting service lred1
Feb 24 12:49:37 red2 ASE: inner2 Agent Notice: starting service drd
Feb 24 12:49:38 red2 ASE: inner2 Director Notice: started service transfer on inner2
Feb 24 12:49:38 red2 cnxmond: mark active inner2
Feb 24 12:49:38 red2 cnxmond: config change: notify director
Feb 24 12:49:38 red2 cnxmond: config change: notify director
Feb 24 12:49:38 red2 cnxmgrd: starting join operation for inner2
Feb 24 12:49:38 red2 cnxmgrd: join, getting status from inner2
Feb 24 12:49:38 red2 cnxmgrd: node inner2, cluster incarn 0, update_seq 0
Feb 24 12:49:38 red2 cnxmgrd: adding node inner2
Feb 24 12:49:38 red2 cnxmond: director acknowleged node inner2
Feb 24 12:49:38 red2 cnxmgrd: update complete, summary follows
Feb 24 12:49:38 red2 cnxmgrd:   members are: 
Feb 24 12:49:38 red2 cnxmgrd:    inner2
Feb 24 12:49:38 red2 cnxmgrd:   timed out are: 
Feb 24 12:49:38 red2 cnxmgrd:    none
Feb 24 12:49:41 red2 mountd[1745]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:49:42 red2 ASE: inner2 Director Notice: started service batch2 on inner2
Feb 24 12:49:44 red2 mountd[1745]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:49:45 red2 cnxmgrd: finished join operation, update_seq 2
Feb 24 12:49:48 red2 mountd[1745]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:49:48 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_run_sh event: /autostart/S65_EvtSvr.sh: /Map/bin/S65_EvtSvr does not exist.
Feb 24 12:49:48 red2 ASE: inner2 Director Notice: started service event on inner2
Feb 24 12:49:49 red2 mountd[1745]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:49:50 red2 ASE: inner2 Director Notice: started service lred1 on inner2
Feb 24 12:49:50 red2 mountd[1745]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:49:50 red2 ASE: inner2 Director Notice: started service lred2 on inner2
Feb 24 12:49:51 red2 ASE: inner2 Director Notice: started service users on inner2
Feb 24 12:49:52 red2 ASE: inner2 Director Notice: started service sdata on inner2
Feb 24 12:49:52 red2 ASE: inner2 Director Notice: started service drd on inner2
Feb 24 12:49:52 red2 ASE: inner2 Director Notice: started service print on inner2
Feb 24 12:49:52 red2 mountd[1745]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:49:53 red2 mountd[1745]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:49:53 red2 ASE: inner2 Agent Notice: /var/ase/sbin/nfs_mountd: cat: cannot open /etc/exports.ase_ASELOCK
Feb 24 12:49:54 red2 ASE: inner2 Director Notice: started service develop on inner2
Feb 24 12:49:54 red2 ASE: inner2 Director Notice: started service system on inner2
Feb 24 12:49:54 red2 mountd[1745]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:49:55 red2 ASE: inner2 Agent Notice: /var/ase/sbin/nfs_mountd: cat: cannot open /etc/exports.ase_ASELOCK
Feb 24 12:49:55 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_run_sh rpc: /autostart/S54_DCSvr.sh: /Map/bin/S54_DCSvr does not exist.
Feb 24 12:49:55 red2 ASE: inner2 Director Notice: started service rpc on inner2
Feb 24 12:49:55 red2 ASE: inner2 Agent Notice: /var/ase/sbin/ase_run_sh start: /autostart/S59_BatSvr.sh: /Map/bin/S59_BatSvr does not exist.
Feb 24 12:49:55 red2 ASE: inner2 Director Notice: started service batch on inner2
Feb 24 12:50:09 red2 xntpd[7799]: xntpd version=3.4x
Feb 24 12:50:09 red2 xntpd[7799]: tickadj = 1, tick = 976, tvu_maxslew = 1023
Feb 24 12:50:09 red2 xntpd[7799]: precision = 976 usec
Feb 24 12:50:18 red2 utxd[7933]: reading utxd configuration file /etc/utxd.conf
Feb 24 12:50:18 red2 utxd[7933]: line 1: activate service 'odl'
Feb 24 12:50:18 red2 utxd[7933]: line 2: activate service 'sim'
Feb 24 12:50:18 red2 utxd[7933]: line 3: activate service 'kkc'
Feb 24 12:50:25 red2 SWXCR[7996]: Monitor Daemon Started.
Feb 24 12:51:19 red2 ASE: local HSM Notice: Able to ping inner1 over the network
Feb 24 12:51:19 red2 ASE: local HSM Notice: Able to ping inner1 over the SCSI bus
Feb 24 12:51:20 red2 ASE: local HSM ***ALERT: HSM_PATH_STATUS:172.16.7.1:UP
Feb 24 12:51:20 red2 ASE: local HSM Notice: member inner1 is UP
Feb 24 12:51:27 red2 cnxmond: mark active inner1
Feb 24 12:51:27 red2 cnxmond: config change: notify director
Feb 24 12:51:27 red2 cnxmond: config change: notify director
Feb 24 12:51:27 red2 cnxmgrd: starting join operation for inner1
Feb 24 12:51:27 red2 cnxmgrd: join, getting status from inner1
Feb 24 12:51:27 red2 cnxmgrd: node inner1, cluster incarn 0, update_seq 0
Feb 24 12:51:27 red2 cnxmgrd: adding node inner1
Feb 24 12:51:27 red2 cnxmond: director acknowleged node inner1
Feb 24 12:51:27 red2 cnxmgrd: update complete, summary follows
Feb 24 12:51:27 red2 cnxmgrd:   members are: 
Feb 24 12:51:27 red2 cnxmgrd:    inner1
Feb 24 12:51:27 red2 cnxmgrd:    inner2
Feb 24 12:51:27 red2 cnxmgrd:   timed out are: 
Feb 24 12:51:27 red2 cnxmgrd:    none
Feb 24 12:51:28 red2 cnxmgrd: finished join operation, update_seq 3
Feb 24 12:51:39 red2 mountd[1745]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 12:51:44 red2 ASE: inner1 Agent Notice: initializing agent... stopping all services
Feb 24 12:51:50 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/IGES: not currently mounted
Feb 24 12:51:50 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/IGES already unmounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMAP: not currently mounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMAP already unmounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMD: not currently mounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/ZMD already unmounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/add: not currently mounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/add already unmounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/font: not currently mounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/data/font already unmounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/job/batch3: not currently mounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/job/batch3 already unmounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/log/batch3: not currently mounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/sdata/Map/log/batch3 already unmounted
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: user script: batch service is not running.
Feb 24 12:51:51 red2 ASE: inner1 Agent Notice: user script: Can not operator batch server
Feb 24 12:51:53 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/info/batch: not currently mounted
Feb 24 12:51:53 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/info/batch already unmounted
Feb 24 12:51:53 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/job/batch: not currently mounted
Feb 24 12:51:53 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/job/batch already unmounted
Feb 24 12:51:53 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/log/batch: not currently mounted
Feb 24 12:51:53 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch/Map/log/batch already unmounted
Feb 24 12:51:54 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/bin: not currently mounted
Feb 24 12:51:54 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/bin already unmounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/csh: not currently mounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/csh already unmounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/include: not currently mounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/include already unmounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/lib: not currently mounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/lib already unmounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/msg: not currently mounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/msg already unmounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/sys: not currently mounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/sys already unmounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/tools: not currently mounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/tools already unmounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/users/sys: not currently mounted
Feb 24 12:51:55 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/system/Map/users/sys already unmounted
Feb 24 12:51:56 red2 ASE: inner1 Agent Notice: user script: rpc service is not running.
Feb 24 12:51:56 red2 ASE: inner1 Agent Notice: user script: Can not operator rpc server
Feb 24 12:51:57 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/info/rpc: not currently mounted
Feb 24 12:51:57 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/info/rpc already unmounted
Feb 24 12:51:57 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/job/rpc: not currently mounted
Feb 24 12:51:57 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/job/rpc already unmounted
Feb 24 12:51:57 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/rpc: not currently mounted
Feb 24 12:51:57 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/rpc already unmounted
Feb 24 12:51:58 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/transfer: not currently mounted
Feb 24 12:51:58 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/rpc/Map/log/transfer already unmounted
Feb 24 12:52:00 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/job/batch2: not currently mounted
Feb 24 12:52:00 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/job/batch2 already unmounted
Feb 24 12:52:00 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/log/batch2: not currently mounted
Feb 24 12:52:00 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/batch2/Map/log/batch2 already unmounted
Feb 24 12:52:00 red2 ASE: inner1 Agent Notice: user script: event service is not running.
Feb 24 12:52:00 red2 ASE: inner1 Agent Notice: user script: Can not operator event server
Feb 24 12:52:02 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/info/event: not currently mounted
Feb 24 12:52:02 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/info/event already unmounted
Feb 24 12:52:02 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/log/event: not currently mounted
Feb 24 12:52:02 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/event/Map/log/event already unmounted
Feb 24 12:52:03 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/data/print: not currently mounted
Feb 24 12:52:03 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/data/print already unmounted
Feb 24 12:52:03 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/info/print: not currently mounted
Feb 24 12:52:03 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/info/print already unmounted
Feb 24 12:52:03 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/job/print: not currently mounted
Feb 24 12:52:03 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/job/print already unmounted
Feb 24 12:52:04 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/log/print: not currently mounted
Feb 24 12:52:04 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/print/Map/log/print already unmounted
Feb 24 12:52:04 red2 ASE: inner1 Agent Notice: user script: ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address
Feb 24 12:52:04 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/log: not currently mounted
Feb 24 12:52:04 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/log already unmounted
Feb 24 12:52:05 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/perf: not currently mounted
Feb 24 12:52:05 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/perf already unmounted
Feb 24 12:52:05 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/sys: not currently mounted
Feb 24 12:52:05 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/sys already unmounted
Feb 24 12:52:05 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/syslog: not currently mounted
Feb 24 12:52:05 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/syslog already unmounted
Feb 24 12:52:05 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/tmp: not currently mounted
Feb 24 12:52:05 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred2/tmp already unmounted
Feb 24 12:52:06 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/md: not currently mounted
Feb 24 12:52:06 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/md already unmounted
Feb 24 12:52:06 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/od: not currently mounted
Feb 24 12:52:06 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/od already unmounted
Feb 24 12:52:06 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/release: not currently mounted
Feb 24 12:52:06 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/release already unmounted
Feb 24 12:52:07 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd1: not currently mounted
Feb 24 12:52:07 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd1 already unmounted
Feb 24 12:52:07 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd2: not currently mounted
Feb 24 12:52:07 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/sd2 already unmounted
Feb 24 12:52:07 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/stub: not currently mounted
Feb 24 12:52:07 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/stub already unmounted
Feb 24 12:52:07 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/users: not currently mounted
Feb 24 12:52:07 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/develop/Develop/users already unmounted
Feb 24 12:52:08 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/data/app: not currently mounted
Feb 24 12:52:08 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/data/app already unmounted
Feb 24 12:52:08 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/info/app: not currently mounted
Feb 24 12:52:08 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/info/app already unmounted
Feb 24 12:52:08 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/app: not currently mounted
Feb 24 12:52:08 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/app already unmounted
Feb 24 12:52:08 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/users: not currently mounted
Feb 24 12:52:08 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/job/users already unmounted
Feb 24 12:52:09 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/log/app: not currently mounted
Feb 24 12:52:09 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /var/ase/mnt/users/Map/log/app already unmounted
Feb 24 12:52:09 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/log: not currently mounted
Feb 24 12:52:09 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/log already unmounted
Feb 24 12:52:09 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/perf: not currently mounted
Feb 24 12:52:09 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/perf already unmounted
Feb 24 12:52:09 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/sys: not currently mounted
Feb 24 12:52:09 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/sys already unmounted
Feb 24 12:52:10 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/syslog: not currently mounted
Feb 24 12:52:10 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/syslog already unmounted
Feb 24 12:52:10 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/tmp: not currently mounted
Feb 24 12:52:10 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_mount_action: /Lred1/tmp already unmounted
Feb 24 12:52:25 red2 ASE: inner2 Agent Notice: stopping service sdata
Feb 24 12:52:25 red2 ASE: inner2 Agent Notice: stopping service batch
Feb 24 12:52:25 red2 ASE: inner2 Agent Notice: stopping service system
Feb 24 12:52:25 red2 ASE: inner2 Agent Notice: stopping service rpc
Feb 24 12:52:25 red2 ASE: inner2 Agent Notice: stopping service transfer
Feb 24 12:52:25 red2 ASE: inner2 Agent Notice: stopping service lred1
Feb 24 12:52:26 red2 ASE: inner2 Director Notice: stopped service transfer on inner2
Feb 24 12:52:26 red2 ASE: inner1 Agent Notice: starting service transfer
Feb 24 12:52:26 red2 ASE: inner2 Director Notice: started service transfer on inner1
Feb 24 12:52:27 red2 ASE: inner2 Director Notice: stopped service lred1 on inner2
Feb 24 12:52:27 red2 ASE: inner1 Agent Notice: starting service lred1
Feb 24 12:52:27 red2 ASE: inner2 Director Notice: stopped service batch on inner2
Feb 24 12:52:27 red2 ASE: inner1 Agent Notice: starting service batch
Feb 24 12:52:27 red2 ASE: inner2 Director Notice: stopped service sdata on inner2
Feb 24 12:52:28 red2 ASE: inner1 Agent Notice: starting service sdata
Feb 24 12:52:28 red2 ASE: inner2 Director Notice: stopped service rpc on inner2
Feb 24 12:52:28 red2 ASE: inner1 Agent Notice: starting service rpc
Feb 24 12:52:28 red2 ASE: inner2 Director Notice: stopped service system on inner2
Feb 24 12:52:28 red2 ASE: inner1 Agent Notice: starting service system
Feb 24 12:52:34 red2 ASE: inner2 Director Notice: started service lred1 on inner1
Feb 24 12:52:35 red2 ASE: inner2 Director Notice: started service sdata on inner1
Feb 24 12:52:36 red2 ASE: inner2 Director Notice: started service system on inner1
Feb 24 12:52:37 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_run_sh start: /autostart/S59_BatSvr.sh: /Map/bin/S59_BatSvr started. 6253
Feb 24 12:52:37 red2 ASE: inner2 Director Notice: started service batch on inner1
Feb 24 12:52:38 red2 ASE: inner1 Agent Notice: /var/ase/sbin/ase_run_sh rpc: /autostart/S54_DCSvr.sh: /Map/bin/S54_DCSvr started. 6291
Feb 24 12:52:38 red2 ASE: inner2 Director Notice: started service rpc on inner1
Feb 24 12:52:38 red2 ASE: inner2 Director Notice: agent on inner1 came ONLINE
Feb 24 13:00:19 red2 xntpd[7799]: synchronized to 172.16.1.1, stratum=4
Feb 24 13:00:18 red2 xntpd[7799]: time reset (step) -1.232541 s
Feb 24 13:00:18 red2 xntpd[7799]: synchronisation lost
Feb 24 13:04:37 red2 mountd[1745]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 13:05:39 red2 xntpd[7799]: synchronized to 172.16.1.1, stratum=4
Feb 24 13:08:13 red2 mountd[9143]: startup
Feb 24 13:08:13 red2 mountd[9143]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 13:08:13 red2 prestoctl_svc: Ignoring invalid arguments.
Feb 24 13:08:13 red2 prestoctl_svc: Warning: cannot open /dev/pr0: No such file or directory
Feb 24 13:08:14 red2 bssd:[9150]: Another BSSD is already running, exiting.
Feb 24 13:08:14 red2 cnxagentd: starting
Feb 24 13:08:14 red2 cnxmond: registration reports already in use
Feb 24 13:08:14 red2 cnxmond: registration retry cleanup failed
Feb 24 13:08:14 red2 cnxmond: changed alias with : /sbin/ifconfig mc0 -alias 
Feb 24 13:09:13 red2 mountd[9142]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 13:09:14 red2 last message repeated 17 times
Feb 24 13:10:22 red2 cnxmond: mark timeout inner1: remove delay: 2
Feb 24 13:10:22 red2 cnxmond: node inner1 timed out
Feb 24 13:10:42 red2 cnxmond: mark inactive inner1
Feb 24 13:10:42 red2 cnxmond: config change: notify director
Feb 24 13:10:42 red2 cnxmgrd: starting removal operation
Feb 24 13:10:42 red2 cnxmgrd: removing node inner1
Feb 24 13:10:42 red2 cnxmond: removing node inner1 from watch list
Feb 24 13:10:42 red2 cnxmgrd: update complete, summary follows
Feb 24 13:10:42 red2 cnxmgrd:   members are: 
Feb 24 13:10:42 red2 cnxmgrd:    inner2
Feb 24 13:10:42 red2 cnxmgrd:   timed out are: 
Feb 24 13:10:42 red2 cnxmgrd:    none
Feb 24 13:10:43 red2 cnxmgrd: finished removal operation, update_seq 4
Feb 24 13:11:08 red2 mountd[9142]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 13:29:56 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 13:35:33 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 13:39:49 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 14:24:58 red2 mountd[9142]: fopen failure /etc/exports.ase.green3: No such file or directory
Feb 24 14:26:50 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 14:40:19 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 15:52:10 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 16:04:58 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 16:09:14 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 16:26:18 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 16:26:19 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 16:56:10 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 17:00:26 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 17:08:58 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 17:30:23 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 17:55:59 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 18:04:33 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 18:08:47 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 18:38:39 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 18:55:43 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 18:55:44 red2 xntpd[7799]: Previous time adjustment didn't complete
Feb 24 18:55:44 red2 xntpd[7799]: *** No more 'Prev time adj didn't complete'
1910.7File to CLDTKTVFS::NAGAI_Kkatsuya nagaiFri Mar 21 1997 01:5811
As a result of this test ( cluster partitioning )

We can reproducing very easy.
On the Directore node,  If /sbin/cnxmond  re-execute. 
Only this command, we were able to reproduce " cluster partitioning ".

We were already escalating to CLD. 
(CLD 49595 )   

Thank you.