Conference spezko::cluster

5293.0. "crash in exit_callback code" by CSC32::REIGELMAN () Tue Apr 22 1997 18:45

I am investigating a System Services Exception that has the same footprint as
an article in the Canasta database "267-{CAN} ALP EXIT_CALLBACK_C+0015C
LOGINOUT current image" R4 contains the count of the Termination Handler
Control Block (R4=2) and R5 contains the pointer to the Termination Control
Block List.
R4  = 00000000 00000002
R5  = 00000000 01ACD8E0

But when EXIT_CALLBACK attempts to get them and call the handler              
that was at address 01ACD8E0 it does not exist any longer. The page has been  
returned back to free memory.

So the problem is occuring because the application is not cleaning up all the 
EXEC mode handlers before the image exits. When the image exits VMS still     
thinks that there are handlers but the memory where they existed has been     
returned back to the free list by image rundown.

Customer had a similiar crash footprint back in March. I am unable to determine
what image/application is causing the system service exception. The only
"common" denominator I see in the recall buffers is Oracle. Has anyone seen
this before and/or aware of an application that will cause this problem?

Below is both recall buffers from both crashes:

	**First crash**
Process DCL Recall Buffer:
--------------------------
Index  Command
  1    LOGO
  2    set term/width=80
  3    T
  4    SQL
  5    DIR TTMP:AQ199702_FAS_SERVICES_SUM.*;
  6    DIR TTMP:AQ199702_FAS_SERVICES_SUM.DAT
  7    SH ENTRY
  8    set term/width=80
  9    T
 10    IBM
 11    SUBM REGR_AQ_fS0100P.COM/LOG=[]/PARAM=(199702, FAS)
 12    T
 13    EDT REGR_AQ_FS0100P.COM
 14    COP [.DBG]AQ_FS0100P.COM REGR_AQ_FS0100P.COM
 15    @LINK_FS0100P.COM
 16    @LINK_DS0100P.COM
 17    COB AQ_fS0100P
 18    SH DEF

		**looking for symbols**
FSCDEV$ search login.com "sql"
$ sqlcomp :== "@aqdsk_d2:[spu1745.com]sqlcompile.com"
$ sls   :==show log sql$database
$ EIS_DB    :== "DEFINE SQL$DATABASE aqdsk_p21:[aq_regr.rdb]aqfl_eis.rdb"
$ FL_REGR   :== "DEFINE SQL$DATABASE fscdev$dkb500:[aq_regr.data.fl]aqfl.rdb"
$ FL_PROD   :== "DEFINE SQL$DATABASE $6$dkb201:[aq_prod.data.fl]aqfl.rdb"
$ ANDY_LIAB :== "DEFINE SQL$DATABASE aqdsk_p43:[aq_regr.data.fl]aqfl_liab.rdb"
$ HIST      :== "DEFINE SQL$DATABASE aqdsk_p23:[aq_regr.data.co]aqco.rdb"
$ rmu_dump :== "rmu/dump/user sql$database"
$ rmu_show :== "rmu/show user sql$database"

$ t     :== show time

$ ibm   :== "set host/sna/acc=tso albibm/wait

		**Second Crash**
Process DCL Recall Buffer:
--------------------------
Index  Command
  1    LO
  2    SET HOST FSCPA2
  3    RMU/DUMP/USERS SQL$DATABASE
  4    CLS
  5    TY/TAIL AQPRODLOG:SYSBAK_AQ_OFFSITE_AQFLLIAB_DAILY_RMU.LOG
  6    TY AQPRODLOG:SYSBAK_AQ_OFFSITE_AQFLLIAB_DAILY_RMU.LOG
  7    SCHED SHOW JOB AQ_PERFORM_BACKUP_RMU_AQFLLIAB=TAPESYS
  8    RMU/DUMP/USERS SQL$DATABASE
  9    CLS
 10    RMU/DUMP/USERS SQL$DATABASE
 11    7
 12    END
 13    AQ_FSBACKUPP30
 14    COM
 15    Y
 16    7358
 17    FIX BACKUP TO BE WEEKLY
 18    00000
 19    3
 20    2
 21    AQ
 22    17
 23    6
 24    LIB_MAINT
 25    DEL TCOM:AQ_FSBACKUPP30.COM;*
 26    COPY TCOM:AQ_FSBACKUPP30.COM RCOM
 27    E TCOM:AQ_FSBACKUPP30.COM
 28    7
 29    AQ_FSBACKUPP30
 30    COM
 31    AQ
 32    6
 33    2
 34    LIB_MAINT
 35    NEWPRF
 36    RMU/DUMP/USERS SQL$DATABASE
 37    SCHED SHOW JOB AQ_PERFORM_BACKUP_RMU_AQFLLIAB=TAPESYS
 38    TY AQDSK_P1:[AQ_PROD.SPL]AQ_START_BACKUP_RMU.LOG
 39    SCHED SHOW JOB 135
 40    MAIL
 41    RMU/DUMP/USERS SQL$DATABASE
 42    CLS
 43    RMU/DUMP/USERS SQL$DATABASE
 44    @[SPEC771]DEFINE_P30
 45    @PSRC:AQ_SETUP PROD 30 LIAB
 46    PRINT TEMP.LIS
 47    E TEMP.LIS
 48    HOME
 49    E AQ_FS0020L.CPY
 50    NEWPRF
 51    DIR BUILD_MASTER.EXE
 52    DIR BKUILD_MASTER
 53    DIR BUIKLD_MASTER
 54    HOME
 55    COPY BUILD_MASTER.EXE [-]
 56    LINK BUILD_MASTER
 57    BAS BUILD_MASTER
 58    TEST
 59    E AQ_FS0020L.CPY
 60    @lINK_DSM020P
 61    @lINK_FS0020P
 62    COB AQ_DSM020P
 63    COB AQ_FS0020P
 64    NEWPRF
 65    PURGE [...]
 66    HOME
 67    SQL
 68    FOCUS
 69    E GREG.SQL
 70    SQL
 71    @[SPEC771]DEFINE_P22
 72    E GREG.SQL
 73    SEARCH *.COM;* 199601
 74    E RUN_UPDATE.COM;-1
 75    E RUN_UPDATE.COM
 76    DIRT *.COM
 77    SEARCH *.COM RUN_FS0020
 78    E RUN_FS0020.COM
 79    @LINK_DSM020P
 80    @LINK_FS0020P
 81    COB AQ_DSM020P
 82    COB AQ_FS0020P
 83    @PSRC:AQ_SETUP TEST FL
 84    COB AQ_FS0020P
 85    E AQ_FS0020P.COB
 86    E AQ_FS0020L.CPY
 87    E GREG.SQL
 88    STOP
 89    FOCUS
 90    E GREG.FEX
 91    SQL
 92    DIRT *.MAS
 93    FOCUS
 94    SQL
 95    @[SPEC771]DEFINE_P22
 96    E TOPX1.SQL
 97    DIR *.SQL
 98    NEWPRF
 99    DIRT *.SQL
100    SHOW DEV AQDSK_D2
101    HOME
102    DIR/TOTAL/SIZE [...]
103    SET DEF [SPE0800]
104    SHOW DEV AQDSK_D2
105    DIR/SIZE/TOTAL [...]*BUG*
106    DIR/SIZE/TOTAL *.DMP
107    DIR/SIZE/TOTAL [...]*.DMP
108    DIR/SIZE/TOTAL [...]
109    SET DEF [UCQ7221]
110    SHOW DEV AQDSK_D2/FILES/OUTPUT
111    DIRS/TOTAL/SIZE [UCQ7221]
112    DIRS [UCQ7221]
113    SHOW DEV AQDSK_D2/FILES/OUTPUT
114    HELP SHOW DEV
115    SHOW DEV AQDSK_D2/FILES/FULL
116    SHOW DEV AQDSK_D2/FILES
117    HELP SHOW DEV
118    SHOW DEV AQDSK_D2/FULL
119    SHOW DEV AQDSK_D2
120    TY TOPX1.SQL
121    E TOPX1.SQL
122    E TOPX2.SQL
123    NEWPRF
124    E CREATE_VIEW.SQL
125    DIR *VIEW*
126    SET DEF [.DSM]
127    DIR [...]*VIEW*.SQL
128    DIR [...]*CREATE*.SQL
129    HOME
130    E VIEW.SQL
131    DIR *VIEW*
132    SET DEF [.SHAWMUT]
133    DIR [...]*VIEW*.SQL
134    DIR [...]*.SQL VIEW
135    HOME
136    MAIL
137    E TEMP.LIS
138    SCHED SHOW JOB 1606/FULL
139    SCHED SHOW JOB 1598
140    SCHED SHOW JOB 1558/FULL
141    SCHED SHOW JOB 1599/FULL
142    SCHED SHOW JOB 1599
143    SCHED SHOW JOB 1580
144    SHCED SHOW JOB 1603
145    SCHED SHOW JOB 1580
146    DIR AQDSK_PO:[AQ_PROD.COM]AQ_FSCOHAM820.COM
147    DIR AQDSK_P0:[AQ_PROD.COM]AQ_FSCOHAM820.COM
148    SCHED SHOW JOB 1606/FULL
149    SCHED  SHOW JOB 1603/FULL
150    SHCED SHOW JOB 1603/FULL
151    E TEMP.LIS
152    HOME
153    E CALL_TREND.FEX
154    SEARCH *.FEX CALL
155    DIR *.FEX;0
156    DIR *.COB;0
157    TY LINKFUSE.COM
158    SET DEF [.HISTORY]
159    DIR [...]*FUS*
160    HOME
161    DIR *FUSLIB*
162    DIR *.FEX;0
163    SEARHC *.FEX HISCONVM
164    E HISCONVM.COB
165    DIR *M.COB
166    SET DEF [.HISTORY]
167    DIR *.DIR
168    HOME
169    DIR *.COB;0
170    SET DEF [.SHAWMUT]
171    DIR *.DIR
172    HOME
173    COPY RUN_UPDATE.COM  RUN_BALANCE.COM
174    E RUN_UPDATE.COM
175    @LINK_FS0200P
176    COB AQ_FS0200P
177    E AQ_FS0200P.COB;-1
178    @LINK_FS0200P
179    @LINK_FS0200
180    COB AQ_FS0200P
181    E AQ_FS0200P.COB
182    PURGE RUN_FS0200.COM
183    E RUN_FS0200.COM
184    E RUN_FS0020.COM
185    NEWPRF
186    MAIL
187    TY TEMP.LIS
188    HOME
189    TY $6$DKB1:[AQ_PROD.DATA.FL]AQFAS_$DATEW013_LIAB.DAT
190    TY AQDSK_P1:[AQ_PROD.LOG.FL]AQ_FS123W013.LOG
191    SCHED SHOW JOB 1603
192    SCHED SHOW 1603
193    TY TEMP.LIS
194    HOME
195    E RUN_FS0200.COM
196    NEWPRF
197    E RUN_FS0200.COM
198    MAIL
199    E/READ DSA8:[AQ_PROD.SCR]AQ_FSLOAD123.COM;5
200    SEARCH PSRC:AQ_FS*.COM MAIL
201    E TEMP.LIS
202    MAIL
203    E TEMP.LIS
204    HOME
205    TY TEMP.LIS
206    TY RUN_FS0200.LOG
207    E AQ_FS0200P.COB
208    DIR AQ*.COB;0
209    DIR *.COB;0
210    NEWPRF
211    SCHED SHOW JOB 1580
212    MAIL
213    LOOP
214    SCHED SHOW JOB 1580
215    LOOP
216    SCHED SHOW JOB 1580
217    RMU/DUMP/USER SQL$DATABASE
218    @DEFINE_P30
219    HOME
220    @[SPEC771]DEFINE_P30
221    E/READ PCOM:AQ_FSBACKUPP30.COM
222    DIR RCOM:AQ_FSBACKUPP30.COM
223    SCHED SHOW JOB 1580
224    SCHED SHOW JOB 1606/FULL
225    SCHED SHOW JOB 16068/FULL

Crashdump Summary Information:
------------------------------
Crash Time:        22-APR-1997 14:22:44.64
Bugcheck Type:     SSRVEXCEPT, Unexpected system service exception
Node:              FSCDEV  (Clustered)
CPU Type:          AlphaServer 2100 4/200
VMS Version:       V6.2-1H3
Current Process:   SPEC771
Current Image:     FSCDEV$DKA0:[SYS0.SYSCOMMON.][SYSEXE]LOGINOUT.EXE
Failing PC:        FFFFFFFF A084403C
Failing PS:        38000000 00000009
Module:            IMAGE_MANAGEMENT
Offset:            0001203C

Boot Time:         21-APR-1997 14:44:58.00
System Uptime:               0 23:37:46.64
Crash/Primary CPU: 00/00
Saved Processes:   57
Pagesize:          8 KByte (8192 bytes)
Physical Memory:   576 MByte (73728 PFNs)
Dumpfile Pagelets: 293158 blocks

    Press RETURN for more.
SDA>

Crashdump Summary Information:
------------------------------
Dump Flags:        olddump,writecomp,errlogcomp,dump_style
EXE$GL_FLAGS:      poolpging,init,bugdump

Stack Pointers:
KSP = 00000000 7FF92000   ESP = 00000000 7FF95C48   SSP = 00000000 7FF9E000
USP = 00000000 000EB8D0

General Registers:
R0  = 00000000 0000000C   R1  = 00000000 7FF95E68   R2  = FFFFFFFF 80A4E5C0
R3  = FFFFFFFF 845B64E0   R4  = 00000000 00000002   R5  = 00000000 01ACD8E0
R6  = 00000000 00000001   R7  = 00000000 7FF91FC0   R8  = 00000000 000EBA48
R9  = 00000000 00000000   R10 = 00000000 7FF9D228   R11 = 00000000 7FFBE3E0
R12 = 00000000 00010001   R13 = FFFFFFFF A0659020   R14 = 00000000 00010540
R15 = 00000000 7F9E9DA0   R16 = 00000000 000003C0   R17 = 00000000 7FF95D00
R18 = 00000000 00000020   R19 = FFFFFFFF 84584200   R20 = 00000000 7FF91FB8
R21 = FFFFFFFF 80A4E4B6   R22 = FFFFFFFF A0658000   R23 = 00000000 7FF96000
R24 = 00000000 00000000   AI  = 00000000 00000003   RA  = 00000000 00000001
PV  = FFFFFFFF 84590848   R28 = 00000000 000005AC   FP  = 00000000 7FF95C50
PC  = FFFFFFFF A072AC00   PS  = 00000000 00000009
  

   Have you checked with Oracle to see if they've also seen this?