| Title: | DECthreads Conference |
| Moderator: | PTHRED::MARYS TE ON |
| Created: | Mon May 14 1990 |
| Last Modified: | Fri Jun 06 1997 |
| Last Successful Update: | Fri Jun 06 1997 |
| Number of topics: | 1553 |
| Total number of notes: | 9541 |
This may turn into an IPMT later on, but John Reguerra
suggested I post it here first.
We have a UNIX 3.2g customer (Bankers Trust) that sees
their C program hang, waiting forever on pthread_join,
when they compile the code with c++ (5.5 or 5.6). The
same exact code, using the same switches runs fine when
compiled with cc or cc -newc.
I dialed in and ran the debugger. The output is included
below.
Also the source to the program (minus lots of header
files and required libraries is also included).
I havent been able to explain why its hanging or solve
this. Any insights?
Pat
-----------------
NYCSEX0000.btco.com> ladebug -pid 3041 ./fsdxSubThreads
Welcome to the Ladebug Debugger Version 4.0-37
------------------
object file name: ./fsdxSubThreads
Reading symbolic information ...done
Attached to process id 3041 ....
Thread received signal INT
stopped at [msg_receive_trap 0x3ff8052e574]
(ladebug) where
>0 0x3ff8052e574 in msg_receive_trap(0x20202041445f4f42, 0x20204f4220202020, 0x2020202020202020, 0x39, 0x3ff804c55ac, 0x1) DebugInf
ormationStrippedFromFile19
#1 0x3ff80524dd0 in msg_receive(0x3ff80577fdc, 0x0, 0x0, 0x0, 0x1, 0x11ffff1a8) DebugInformationStrippedFromFile6
#2 0x3ff805885dc in cma__vp_sleep(0x2801fd4558, 0x0, 0x6, 0x0, 0x3ff00000000, 0x3ffc0093530) DebugInformationStrippedFromFile106
#3 0x3ff8056e958 in cma__dispatch(0x6, 0x0, 0x3ff00000000, 0x3ffc0093530, 0x3ff80567e34, 0x3ffc00802b8) DebugInformationStrippedFro
mFile85
#4 0x3ff80567e30 in cma__int_wait(0x11ffff358, 0x2d218, 0x3ffc01de9f8, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile3
#5 0x3ff805811e0 in cma_thread_join(0x11ffff778, 0x11ffffb20, 0x11ffffb18, 0x3ff80581bf0, 0x0, 0x3ffc01e3de0) DebugInformationStrip
pedFromFile101
#6 0x3ff80577508 in pthread_join(0x32c18, 0x400000000000008, 0x11ffffc28, 0x3ffc01de9f8, 0x1200156b0, 0x2d418) DebugInformationStri
ppedFromFile95
#7 0x120015724 in main() fsdxSubThreads.cxx:293
---------------------
NYCSEX0000.btco.com> ladebug fsdxSubThreads
Welcome to the Ladebug Debugger Version 4.0-37
------------------
object file name: fsdxSubThreads
Reading symbolic information ...done
(ladebug) stop at 292
[#1: stop at "fsdxSubThreads.cxx":293 ]
(ladebug) run
opened map file
Sucessfully parsed map file
calling open
opened dsv
regis map file 1
0
1
[1] stopped at [int main(void):293 0x120015700]
293 trdStatus = pthread_join (threads[workerNum], (pthread_addr_t *)&exitValue);
(ladebug) s
2
trd0trd1trd2trd0Thread received signal INT
stopped at [msg_receive_trap 0x3ff8100e574]
(ladebug) where
>0 0x3ff8100e574 in msg_receive_trap(0x20202041445f4f42, 0x20204f4220202020, 0x2020202020202020, 0x39, 0x3ff820355ac, 0x1) DebugInf
ormationStrippedFromFile19
#1 0x3ff81004dd0 in msg_receive(0x3ff80837fdc, 0x0, 0x0, 0x0, 0x1, 0x11ffff1a8) DebugInformationStrippedFromFile6
#2 0x3ff808485dc in cma__vp_sleep(0x2801fd4558, 0x0, 0x7, 0x0, 0x3ff00000000, 0x3ffc2813530) DebugInformationStrippedFromFile106
#3 0x3ff8082e958 in cma__dispatch(0x7, 0x0, 0x3ff00000000, 0x3ffc2813530, 0x3ff80827e34, 0x3ffc28002b8) DebugInformationStrippedFro
mFile85
#4 0x3ff80827e30 in cma__int_wait(0x11ffff358, 0x2d218, 0x3ffc049e9f8, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile3
#5 0x3ff808411e0 in cma_thread_join(0x11ffff778, 0x11ffffb20, 0x11ffffb18, 0x3ff80841bf0, 0x0, 0x3ffc01e3de0) DebugInformationStrip
pedFromFile101
#6 0x3ff80837508 in pthread_join(0x32c18, 0x400000000000008, 0x11ffffc28, 0x3ffc049e9f8, 0x1200156b0, 0x2d418) DebugInformationStri
ppedFromFile95
#7 0x120015724 in main() fsdxSubThreads.cxx:293
(ladebug) call cma_debug()
DECthreads debug> mutex
Mutex 81 (recursive) "<pthread user@0x43f88>" is not locked
Mutex 82 (recursive) "<pthread user@0x3fffffba400>" is locked by 6 (depth 1), 1
thread waiting
Mutex 84 (fast) "<pthread user@0x3fffffee1f0>" is not locked
Mutex 85 (fast) "<pthread user@0x3fffffba490>" is not locked
Mutex 99 (fast) "<pthread user@0x140000568>" is locked, 2 threads waiting
Mutex 100 (fast) "<pthread user@0x140000578>" is not locked
Mutex 108 (fast) "<pthread user@0x3fffffba290>" is not locked
DECthreads debug> condition
Condition variable 3, "<pthread user@0x4a080>" has no waiters
Condition variable 4, "<pthread user@0x39e80>" has no waiters
Condition variable 7, "<pthread user@0x49ae8>" has no waiters
Condition variable 10, "<pthread user@0x41ac8>" has no waiters
Condition variable 11, "<pthread user@0x41aa8>" has 1 waiter using mutex 84
Condition variable 16, "<pthread user@0x418c8>" has 1 waiter using mutex 84
Condition variable 17, "<pthread user@0x418a8>" has 1 waiter using mutex 84
Condition variable 18, "<pthread user@0x140000588>" has no waiters
Condition variable 29, "<pthread user@0x49550>" has no waiters
DECthreads debug> thread -i -f
Thread 1 (blocked, cond wait) "default thread" (0x3ffc049e9f8)
Waiting on condition variable 19 using mutex 101
Scheduling: throughput policy at priority 19
No thread specific data
(*)Stack: 0x11fffecc8 (default stack)
General cancelability enabled, asynch cancelability disabled
Current vp is 6, synch port is 7
VP state is run: Mach policy throughput, priority 19 (RT kernel), state
running; synch port has 0 messages out of 1
Join uses mutex 21 and condition variable 1; wait uses mutex 22 and
condition variable 2
The thread's start function and argument are unknown
The thread's latest errno is 0
Thread 3 (running) "<pthread user@0x3ffc04a7aa8>" (0x33818)
Scheduling: throughput policy at priority 19
personal pending wake is set
No thread specific data
Stack: 0x3ffc04b0000; base is 0x3ffc04b0000, guard area at 0x3ffc04a9fff
Detached
General cancelability enabled, asynch cancelability disabled
Current vp is 12, synch port is 13
VP state is <unknown-wait> (hold): Mach policy throughput, priority 19 (RT
kernel), state waiting; synch port has 0 messages out of 1
Join uses mutex 91 and condition variable 8; wait uses mutex 92 and
condition variable 9
The thread's start function and argument are 0x3ffbffd2630 (0x41a08)
The thread's latest errno is 0
Thread 4 (blocked, cond wait) "<pthread user@0x3ffc04afa70>" (0x33c18)
Waiting on condition variable 11 using mutex 84
Scheduling: throughput policy at priority 19
No thread specific data
Stack: 0x3ffc04a8000; base is 0x3ffc04a8000, guard area at 0x3ffc04a1fff
Detached
General cancelability enabled, asynch cancelability disabled
Current vp is 9, synch port is 10
VP state is susp (hold): Mach policy throughput, priority 19 (RT kernel),
state waiting; synch port has 0 messages out of 1
Join uses mutex 86 and condition variable 5; wait uses mutex 87 and
condition variable 6
The thread's start function and argument are 0x3ffbffd3150 (0x41a08)
The thread's latest errno is 0
Thread 6 (blocked, cond wait) "<pthread user@0x3ffc04b7aa8>" (0x33018)
Waiting on condition variable 16 using mutex 84
Scheduling: throughput policy at priority 19
No thread specific data
Stack: 0x3ffc04c0000; base is 0x3ffc04c0000, guard area at 0x3ffc04b9fff
Detached
General cancelability enabled, asynch cancelability disabled
Current vp is 18, synch port is 19
VP state is susp (hold): Mach policy throughput, priority 19 (RT kernel),
state waiting; synch port has 0 messages out of 1
Join uses mutex 97 and condition variable 14; wait uses mutex 98 and
condition variable 15
The thread's start function and argument are 0x3ffbffd2630 (0x41808)
The thread's latest errno is 0
The thread has mutexes locked: 82
Thread 7 (blocked, cond wait) "<pthread user@0x3ffc04bfa70>" (0x33418)
Waiting on condition variable 17 using mutex 84
Scheduling: throughput policy at priority 19
No thread specific data
Stack: 0x3ffc04b8000; base is 0x3ffc04b8000, guard area at 0x3ffc04b1fff
Detached
General cancelability enabled, asynch cancelability disabled
Current vp is 15, synch port is 16
VP state is susp (hold): Mach policy throughput, priority 19 (RT kernel),
state waiting; synch port has 0 messages out of 1
Join uses mutex 93 and condition variable 12; wait uses mutex 94 and
condition variable 13
The thread's start function and argument are 0x3ffbffd3150 (0x41808)
The thread's latest errno is 0
Thread 0 (blocked, mutex wait) "<pthread user@0x140000598>" (0x32c18)
Waiting to lock mutex 99
Scheduling: throughput policy at priority 19
There is 1 thread waiting to join
No thread specific data
Stack: 0x3ffc04c8000; base is 0x3ffc04c8000, guard area at 0x3ffc04c1fff
General cancelability disabled, asynch cancelability disabled
Current vp is 21, synch port is 22
VP state is susp (hold): Mach policy throughput, priority 19 (RT kernel),
state waiting; synch port has 0 messages out of 1
Join uses mutex 101 and condition variable 19; wait uses mutex 102 and
condition variable 20
The thread's start function and argument are 0x120014dd0 (0x0)
The thread's latest errno is 1
The thread has mutexes locked: 99
Thread 0 (blocked, mutex wait) "<pthread user@0x1400005a8>" (0x32818)
Waiting to lock mutex 82
Scheduling: throughput policy at priority 19
personal pending wake is set
No thread specific data
Stack: 0x3ffc04d0000; base is 0x3ffc04d0000, guard area at 0x3ffc04c9fff
General cancelability disabled, asynch cancelability disabled
Current vp is 24, synch port is 25
VP state is susp (hold): Mach policy throughput, priority 19 (RT kernel),
state waiting; synch port has 0 messages out of 1
Join uses mutex 103 and condition variable 21; wait uses mutex 104 and
condition variable 22
The thread's start function and argument are 0x120014dd0 (0x1)
The thread's latest errno is 0
The thread has mutexes locked: 99
Thread 0 (blocked, mutex wait) "<pthread user@0x1400005b8>" (0x32418)
Waiting to lock mutex 99
Scheduling: throughput policy at priority 19
No thread specific data
Stack: 0x3ffc04d8000; base is 0x3ffc04d8000, guard area at 0x3ffc04d1fff
General cancelability disabled, asynch cancelability disabled
Current vp is 27, synch port is 28
VP state is susp (hold): Mach policy throughput, priority 19 (RT kernel),
state waiting; synch port has 0 messages out of 1
Join uses mutex 105 and condition variable 23; wait uses mutex 106 and
condition variable 24
The thread's start function and argument are 0x120014dd0 (0x2)
The thread's latest errno is 0
The thread has mutexes locked: 99
DECthreads debug>
DECthreads debug> stack
Thread 1 stack: 0x4000000 to 0x11fffffff (469762047 bytes)
Thread 3 stack: 0x3ffc04a8000 to 0x3ffc04b0000 (32768 bytes)
Thread 4 stack: 0x3ffc04a0000 to 0x3ffc04a8000 (32768 bytes)
Thread 6 stack: 0x3ffc04b8000 to 0x3ffc04c0000 (32768 bytes)
Thread 7 stack: 0x3ffc04b0000 to 0x3ffc04b8000 (32768 bytes)
Thread 0 stack: 0x3ffc04c0000 to 0x3ffc04c8000 (32768 bytes)
Thread 0 stack: 0x3ffc04c8000 to 0x3ffc04d0000 (32768 bytes)
Thread 0 stack: 0x3ffc04d0000 to 0x3ffc04d8000 (32768 bytes)
DECthreads debug>DECthreads debug> attributes
Attributes object 1 "default attr"
Scheduling: policy throughput, priority 19; inherit scheduling
Threads created joinable
Stack size 21120, guard size 2048
Mutex type fast
Attributes object 4 "<pthread user(thread)@0x11ffffba0>"
Scheduling: policy throughput, priority 19; inherit scheduling
Threads created joinable
Stack size 1024000, guard size 2048
Mutex type fast
DECthreads debug>
==================================================================
//
// Here is the source code to the program which hangs.
// The sources to libfsdx.so are not available.
//
/***********************************************************************
*
* Facility : FSDX
* Financial Services Data Exchange Message
*
* Module : FSDX Subscriber
*
* Description : This module will sequentially read all records from a
safestore
*
* Notes :
*
* Modification History :
*
* # date by reason
* --- -------- --- ---------------------------------------------
* 000 8/27/96 mm created
*
*******************************************************************/
#include <fsdxSubThreads.h>
#undef MODULE_NAME
#define MODULE_NAME "FsdxSubThreads"
FSDX_CTXHANDLE channel;
FSDX_STATUS fsdxStatus;
/*******************************************************************************
// Worker Thread Routine -
// Worker routine begins with a cond wait to synchronize the workers and
// the parent. Each worker thread then takes a turn taking a number which
// it will test for primeness
/*******************************************************************************
void
fsdxReads (pthread_addr_t arg)
{
long myNumber; // Worker thread identifier
int trdStatus; // Hold status from pthread calls
int recsRead=0;
int errBuffLen=128;
char errBuff[128];
char messageBuffer[10000];
char *messageMapReturned,
*userBuffer = messageBuffer,
*expirationTime;
FSDX_SEQNO seqNo;
myNumber = (long)arg;
fprintf(stderr, "%d\n", myNumber);
fflush(stderr);
// Synchronize threads and parent using a condition variable, of which
// predicate (thread_hold) will be set by the parent.
trdStatus = pthread_mutex_lock(&condMutex);
check(trdStatus,"1:Mutex Lock Bad Status\n");
while (threadHold)
{
trdStatus = pthread_cond_wait(&condVar, &condMutex);
check(trdStatus,"2:Ccond_wait, bad status\n");
}
trdStatus = pthread_mutex_unlock(&condMutex);
check(trdStatus,"3:Mutex unlock bad status");
messageMapReturned=(char *)malloc(sizeof(char) * 100);
// Perform reads until bad status is found
while (1)
{
// Cancelation point
pthread_testcancel();
// Explicitly turn off all cancels
pthread_setcancel(CANCEL_OFF);
trdStatus = pthread_mutex_lock(&fsdxMutex);
check(trdStatus,"4:fsdxMutex Lock Bad Status\n");
if (!((fsdxStatus == FSDX_S_SUCCESS) ||
(fsdxStatus == FSDX_S_HARD_READ)))
{
trdStatus = pthread_mutex_unlock(&fsdxMutex);
check(trdStatus,"4:fsdxMutex unlock bad status");
break;
}
fsdxStatus = FsdxSubReadNextSync(
channel,
messageMapReturned,
userBuffer,
&seqNo,
expirationTime);
if ((fsdxStatus == FSDX_S_SUCCESS) ||
(fsdxStatus == FSDX_S_HARD_READ))
{
trdStatus = pthread_mutex_unlock(&fsdxMutex);
check(trdStatus,"4:fsdxMutex unlock bad status");
recsRead++;
totalRecsRead++;
/* if (recsRead%10==0)
{
fprintf(stderr,".");
fflush(stderr);
fprintf(stderr,"trd%d",myNumber);
fflush(stderr);
}*/
fprintf(stderr,"trd%d",myNumber);
fflush(stderr);
sleep(1);
}
else
{
FsdxStatusToString(fsdxStatus,errBuff,errBuffLen);
trdStatus = pthread_mutex_unlock(&fsdxMutex);
check(trdStatus,"4:fsdxMutex unlock bad status");
printf("\n%s\n",errBuff);
break;
}
/* Reenable Cancels */
pthread_setcancel(CANCEL_ON);
pthread_testcancel();
}
pthread_exit ((pthread_addr_t *)myNumber);
}
main()
{
long workerNum; // Counter used to index workers
int exitValue; // Individual workers return status
int trdStatus; // Hold status from pthread calls
PARAMS *subParams;
char *pSessMarker;
char *userMap;
int lineCount;
FSDX_U2 astQuota = 100,
queueDepth = 3;
FSDX_U4 serviceDataStatus,
msgLength=0,
*msgCount,
*byteCount;
FSDX_ACCESSMODE ReadMode = ACCESSMODE_SQNTL_CNTNS;
int errBuffLen=128;
char errBuff[128];
time_t ts0, ts1;
int elapsedSecs,
recsPerSecs,
recsPerMin,
recsPerHr;
// Call a routine that reads program driving params from a simple file
if (!getParams(&subParams))
{
printf("Couldn't open params file for Insert Subscriber, Sorry !\n");
return(0);
}
// Call a routine that will parse an FSDX map from an external file
if (!processMap(&userMap,&msgLength,&lineCount))
{
printf("Failed to process map file properly\n");
return(0);
}
if (strncmp(subParams->sessionMarker,"spaces",6)==0)
memset(subParams->sessionMarker,' ',8);
if (strncmp(subParams->sessionMarker,"NULL",4)==0)
pSessMarker=NULL;
else
pSessMarker = subParams->sessionMarker;
printf("calling open\n");
fsdxStatus = FsdxSubOpenChannel (
&channel,
(FSDX_CSTR)subParams->subscriberName,
(FSDX_CSTR)subParams->dataServiceName,
ReadMode,
&serviceDataStatus,
pSessMarker,
astQuota,
queueDepth,
NULL,
FSDX_M_STREAM,
NULL);
if (fsdxStatus != FSDX_S_SUCCESS)
{
printf("Failed open %d", fsdxStatus);
FsdxStatusToString(fsdxStatus,errBuff,errBuffLen);
printf("\n%s\n",errBuff);
return(0);
}
printf("opened dsv \n");
fsdxStatus = FsdxSubRegisterMap(
channel,
(FSDX_CSTR) subParams->messageMapName,
userMap,
&msgLength);
if (fsdxStatus != FSDX_S_SUCCESS)
{
printf("Failed regis %d", fsdxStatus);
FsdxStatusToString(fsdxStatus,errBuff,errBuffLen);
printf("\n%s\n",errBuff);
for (fsdxStatus=0;fsdxStatus<lineCount;fsdxStatus++)
{
printf("%-41.41s",userMap);
userMap+=41;
}
return(0);
}
printf("regis map file %d\n", fsdxStatus);
time(&ts0);
// Beginning of thread nonsense
totalRecsRead=0;
// Create Mutexes
trdStatus = pthread_mutex_init (&fsdxMutex, pthread_mutexattr_default);
check(trdStatus,"8:fsdxMutex_init bad status\n");
trdStatus = pthread_mutex_init (&condMutex, pthread_mutexattr_default);
check(trdStatus,"8:Mutex_init bad status\n");
// Create Condition Variable "CV"
trdStatus = pthread_cond_init (&condVar, pthread_condattr_default);
check(trdStatus,"10:Cond_init bad status\n");
pthread_attr_t attr;
pthread_attr_create(&attr);
pthread_attr_setstacksize(&attr ,1024000);
// Create Worker Threads
for (workerNum = 0; workerNum < WORKERS; workerNum++)
{
trdStatus = pthread_create(&threads[workerNum],
pthread_attr_default,
// attr,
fsdxReads,
(pthread_addr_t)workerNum);
// Maxie:1 workerNum should be deref'ed.
check(trdStatus,"11:Pthread_create bad status\n");
}
// Set the predicate thread_hold to 0, and broadcast on condition varibale
// that worker threads may proceed
trdStatus = pthread_mutex_lock (&condMutex);
check(trdStatus,"12:Mutex_lock bad status\n");
threadHold = 0;
trdStatus = pthread_cond_broadcast (&condVar);
trdStatus = pthread_mutex_unlock (&condMutex);
check(trdStatus,"13:Mutex_unlock bad status\n");
// Join each worker inorder to obtain summation totals, and
// to ensure successful completion.
// Mark thread storage as free to be reclaimed upon termination
// by detaching it
for (workerNum = 0; workerNum < WORKERS; workerNum++)
{
trdStatus = pthread_join (threads[workerNum], (pthread_addr_t *)&exitVal
check(trdStatus,"14:pthread_join bad status\n");
if (exitValue == workerNum)
printf("\nThread %d terminated normally\n",workerNum);
// Upon normal termination the exit_value = the workerNum
trdStatus = pthread_detach (&threads[workerNum]);
check(trdStatus,"15:pthread_detach bad status\n");
}
// End of thread nonsense
time(&ts1);
elapsedSecs=(time_t)(ts1-ts0);
recsPerSecs=totalRecsRead/elapsedSecs;
recsPerMin=recsPerSecs*60;
recsPerHr=recsPerMin*60;
printf("\nTotal recs read = %d\nTime to read was %d\nRecords inserted at rat
of %d per second",
totalRecsRead,elapsedSecs,recsPerSecs);
printf("\nRecords inserted at rate of %d per minute\nRecords inserted at rat
of %d per Hour",
recsPerMin,recsPerHr);
printf("\n");
fsdxStatus = FsdxSubCloseChannelSync (
channel,
msgCount,
byteCount );
}
/***************************End Of
Main*******************************************************/
/* This is the routine that parses the params file*/
int getParams(PARAMS **pParams)
{
FILE *paramsFile;
*pParams=(PARAMS *)malloc(sizeof(PARAMS));
if ((paramsFile = fopen("fsdxSub.params","r")) == NULL)
return(0);
fscanf(paramsFile, "%s %s %s %s %s %s %s %s %s %s %s %s %s %s %s",
(*pParams)->publisherName,
(*pParams)->subscriberName,
(*pParams)->dataServiceName,
(*pParams)->sessionMarker,
(*pParams)->messageMapName,
(*pParams)->sequenceNumber,
(*pParams)->dbServer,
(*pParams)->dbName,
(*pParams)->dbUserId,
(*pParams)->dbPasswd,
(*pParams)->dbRcvStoredProc,
(*pParams)->dbSndStoredProc,
(*pParams)->param1,
(*pParams)->param2,
(*pParams)->tableName);
fclose (paramsFile);
return (1);
}
/* this is the routine that parses the map file */
int processMap(char **userMap,
FSDX_U4 *msgLength,
int *lineCount)
{
FILE *mapFile;
int mapStatus;
char map[10000];
char tmpMapBuffer[80];
char *inputStatus;
MAPELEMENT *pMap = (MAPELEMENT *)map;
if ((mapFile = fopen("fsdxSub.map","r")) == NULL)
{
return(0);
}
printf("opened map file \n");
*msgLength=0;
*lineCount=0;
*userMap=(char *)malloc(sizeof(char)*41);
while (1)
{
memset(tmpMapBuffer, 32, 80);
memset(pMap->type, 32, 4);
memset(pMap->name, 32, 31);
memset(pMap->length, 32, 4);
memset(pMap->status, 32, 2);
inputStatus = fgets (tmpMapBuffer, 80, mapFile);
*userMap=(char *)realloc(*userMap, strlen(*userMap)+41);
sprintf(*userMap+(*lineCount*41),"%-41.41s", tmpMapBuffer);
if (inputStatus == '\0')
break;
mapStatus = sscanf(tmpMapBuffer,"%s %s %s %s",
pMap->type,
pMap->name,
pMap->length,
pMap->status);
if(mapStatus > 0)
pMap->type[strlen(pMap->type)] = ' ';
if(mapStatus > 1)
pMap->name[strlen(pMap->name)] = ' ';
if(mapStatus > 2)
{
pMap->length[strlen(pMap->length)] = ' ';
*msgLength+=atoi(pMap->length);
}
if(mapStatus > 3)
pMap->status[strlen(pMap->status)] = ' ';
*lineCount += 1;
pMap += 1;
}
fclose(mapFile);
printf("Sucessfully parsed map file\n");
return(1);
}
--------------------------------
#ifndef FPCA_PUB_H
#define FPCA_PUB_H
/*******************************************************************
*
* Facility : FSDX
* Financial Services Data Exchange Message
*
* Module : FPCA Publisher Header file
*
* Notes :
*
* Modification History :
*
* # date by reason
* --- -------- --- ---------------------------------------------
* 000 10/17/96 mm created
*
*******************************************************************/
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <malloc.h>
#include <pthread.h>
extern "C" {
#include <fsdx.h> /* FSDX Header */
#include <fsdx_sts.h> /* FSDX Return Codes */
}
#define WORKERS 50
/* Macros */
#define check(status,string) \
if (status == -1) \
perror(string)
/* Global Data */
pthread_mutex_t fsdxMutex; /* Mutex used to lock fsdxStatus */
pthread_mutex_t condMutex; /* Mutex used to ensure CV integrity */
pthread_cond_t condVar; /* Condition Variable for thread start */
int threadHold = 1; /* Number assosciated with condition state
*/
pthread_t threads[WORKERS]; /* Array of worker threads */
int totalRecsRead;
typedef struct {
char publisherName[FSDX_K_NAMELEN+1];
char subscriberName[FSDX_K_NAMELEN+1];
char dataServiceName[FSDX_K_NAMELEN+1];
char sessionMarker[FSDX_K_SESSMARKLEN+1];
char messageMapName[FSDX_K_NAMELEN+1];
char sequenceNumber[FSDX_K_SESSMARKLEN+1];
char dbServer[FSDX_K_NAMELEN+1];
char dbName[FSDX_K_NAMELEN+1];
char dbUserId[FSDX_K_NAMELEN+1];
char dbPasswd[FSDX_K_NAMELEN+1];
char dbRcvStoredProc[FSDX_K_NAMELEN+1];
char dbSndStoredProc[FSDX_K_NAMELEN+1];
char param1[FSDX_K_NAMELEN+1];
char param2[FSDX_K_NAMELEN+1];
char tableName[FSDX_K_NAMELEN+1];
} PARAMS;
typedef struct {
char type[4];
char name[31];
char length[4];
char status[2];
} MAPELEMENT;
typedef struct {
char key[32];
char fld0[234];
char fld1[234];
char fld2[234];
char fld3[234];
} BOB_DATA;
/* struct for Msg List */
typedef struct _MSG {
char *buff;
int mode;
struct _MSG *next;
} MSG;
/* Struct to hold CB info */
typedef struct {
char *userBuffer;
int iteration;
char *tableName;
int flag;
int commit;
int connNum;
MSG *msgLst;
} CBDATA;
/* Forward declarations for error and message handlers and fcn calls */
int getParams(PARAMS **);
int processMap(char **, FSDX_U4 *,int *);
void readNextCb(void*,
char*,
char*,
int*,
FSDX_SEQNO*,
char*,
CBDATA*,
int* );
void addMsg(void **, void *);
#endif
| T.R | Title | User | Personal Name | Date | Lines |
|---|---|---|---|---|---|
| 1551.1 | It is scary to think that this code is supporting a large financial institution | WTFN::SCALES | Despair is appropriate and inevitable. | Fri May 30 1997 19:03 | 38 |
The threads list includes three "Thread 0"'s, but zero is not a legitimate sequence number, so it would seem that something has corrupted portions of the data structures which belong to the threads library. We took a look at the source code provided. We are scared. Of the various "interesting" features we found in the code, the single most disturbing is the fact that one of the routines called by the threads allocates 10K off the stack. Since the threads are being created with the default guard areas of only 2K, there's a real possibility that these threads could overflow their stacks without anybody realizing it. Threads should be created with a guard area at least as large as the largest stack allocation. On the other hand, since stack space in a thread is a scarce and limited resource, a good rule of thumb is that any allocation larger than 512 bytes (or perhaps up to half a page) should be allocated from the heap (i.e., via malloc()) and not from the stack. (In addition to easing stack size problems, following this rule of thumb also frees you from having to enlarge the guard size.) However, there is one other item, which could be the source of the corruption (and therefore, possibly, the hang). In the routine fsdxReads(), there is a call to FsdxSubReadNextSync() which specifies "expirationTime" as one of the parameters. This appears to be an uninitialized pointer. If this is an output parameter, then this would result in a random memory corruption. Since the dirty part of the stack frequently contains addresses of other parts of the stack or of data structures accessed in previous routine calls, writing to uninitialized local pointer variables tends to corrupt other parts of the stack or the afore mentioned data structures... As to why it hangs when compiled by C++ and not by C, there could be lots of reasons. However, in light of the above, I'd suggest that it's because the code compiled with the C compiler leaves different bit patterns on the stack, which result in the corruption occuring at a (relatively) benign location instead of nailing the threads data structures. Webb (and Peter) | |||||
| 1551.2 | Compilers work to obey stack guards | WIBBIN::NOYCE | Pulling weeds, pickin' stones | Mon Jun 02 1997 12:01 | 10 |
> Since the threads are being created with the default guard > areas of only 2K, there's a real possibility that these threads could overflow > their stacks without anybody realizing it. > Threads should be created with a guard area at least as large as the largest > stack allocation. No, this is not necessary. The Digital UNIX calling standard specifies a protocol for stack checking that guarantees that the guard size it specifies is adequate. As far as I know, all the Digital UNIX compilers follow the protocol. Have you seen otherwise? (PS- how do you get a 2K guard area on Alpha??) | |||||
| 1551.3 | DCETHD::BUTENHOF | Dave Butenhof, DECthreads | Mon Jun 02 1997 13:09 | 18 | |
Unfortunately, I believe we found out that DEC C had NOT been generating stack probes until relatively recently. Given that this is c++ "5.5 or 5.6", my guess would that it's more than "recent enough". On the other hand, DEC C and DEC C++ do not always maintain parity where I would expect parity to be an obvious basic requirement. (For example, c++ still apparently does not support long pointers on EVMS). So can we get confirmation of the exact actual version of compiler they're using... and then from the DEC C++ group that this version really does stack probes? We recently ran into a problem where our code was broken by bad stack probing during our context switch. We worked around the problem by changing the compiler options for the affected module to prevent the compiler from generating stack probes. I don't recall now exactly which option it was (I think it was optimization level, but I'm not sure). In any case, a full report on which compiler OPTIONS the affected program is using might also be useful. /dave | |||||
| 1551.4 | Which version of "the present" are we all in? ;-) | WTFN::SCALES | Despair is appropriate and inevitable. | Mon Jun 02 1997 14:17 | 8 |
.2> how do you get a 2K guard area on Alpha? Obviously you can't; we round up to the next protection granularity unit. Nevertheless, that's not enough in the absence of stack probing, and I don't remember us getting much of that when last we were on V3.2. Webb | |||||
| 1551.5 | Thanks. I will see what I can find out about the compiler options. | CREME::LAMPERT | Pat Lampert, UNIX Applications Support, 343-1050 | Tue Jun 03 1997 20:19 | 8 |
Sorry. I let a whole day go by without checking bac back on this note. Thanks for the suggestions. One of the things I tried (not in the sources I sent you) was setting up a really big stack. This didnt seem to matter. The program still failed in the same way. Pat | |||||
| 1551.6 | One other thing.... | CREME::LAMPERT | Pat Lampert, UNIX Applications Support, 343-1050 | Tue Jun 03 1997 20:25 | 126 |
Before entering note -.1 I did try using CMA_METER to get more information on which threads owns the mutexes and.... From my CHAMP w/u: ---------------------------------------------------------- Now that the problem with the sources are fixed. I dialed into the customer's system again. Unfortunately setting up CMA_METER is causing the program to fail in a different way... NYCSEX0000.btco.com> export CMA_METER NYCSEX0000.btco.com> ladebug fsdxSubThreads Welcome to the Ladebug Debugger Version 4.0-37 ------------------ object file name: fsdxSubThreads Reading symbolic information ...done (ladebug) run opened map file Sucessfully parsed map file calling open opened dsv regis map file 1 0 1 2 Thread received signal SEGV stopped at [_doprnt 0x3ff8102d7a8] (ladebug) where >0 0x3ff8102d7a8 in _doprnt(0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xa aaaaaaaaaaaaaaa) DebugInformationStrippedFromFile38 #1 0x3ff81056d14 in sprintf(0x3ffc24a6998, 0x3ffc1c9b7d8, 0x3ffc1c9b7b0, 0x3ffc24a69a8, 0x3ffc24a6c58, 0x8) DebugInformationStrippe dFromFile150 #2 0x3ff82030f34 in cma__error_inq_text(0x177db037, 0x3ffc24a6b4b, 0x3ffc24a6c58, 0x104c18, 0x0, 0x6) DebugInformationStrippedFromF ile91 #3 0x3ff82030390 in exc_get_message(0x3ffc24a6d50, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #4 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #5 0x3ff8202ffcc in exc_raise(0x3ffc24a6d50, 0x0, 0x78018, 0x0, 0x0, 0x104c18) DebugInformationStrippedFromFile87 #6 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #7 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #8 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaa aaaaaa) DebugInformationStrippedFromFile108 #9 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xa aaaaaaa0000000a) DebugInformationStrippedFromFile25 #10 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaa aaaaaaaaaaa) DebugInformationStrippedFromFile2 #11 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc24a6f5a, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStr ippedFromFile77 #12 0x3ff8102a060 in __catopen(0x3ffc24a6f50, 0x3ffc1c9b7c0, 0x3ffc24a6fd8, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrip pedFromFile28 #13 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc24a718b, 0x3ffc24a7298, 0x104c18, 0x0, 0x6) DebugInformationStrippedFromF ile91 #14 0x3ff82030390 in exc_get_message(0x3ffc24a7390, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #15 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #16 0x3ff8202ffcc in exc_raise(0x3ffc24a7390, 0x0, 0x78018, 0x0, 0x0, 0x104c18) DebugInformationStrippedFromFile87 #17 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #18 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #19 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaa aaaaaa) DebugInformationStrippedFromFile108 #20 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xa aaaaaaa0000000a) DebugInformationStrippedFromFile25 #21 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaa aaaaaaaaaaa) DebugInformationStrippedFromFile2 #22 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc24a759a, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStr ippedFromFile77 #23 0x3ff8102a060 in __catopen(0x3ffc24a7590, 0x3ffc1c9b7c0, 0x3ffc24a7618, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrip pedFromFile28 #24 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc24a77cb, 0x3ffc24a78d8, 0x104c18, 0x0, 0x6) DebugInformationStrippedFromF ile91 #25 0x3ff82030390 in exc_get_message(0x3ffc24a79d0, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #26 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #27 0x3ff8202ffcc in exc_raise(0x3ffc24a79d0, 0x0, 0x78018, 0x0, 0x0, 0x104c18) DebugInformationStrippedFromFile87 #28 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #29 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #30 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaa aaaaaa) DebugInformationStrippedFromFile108 #31 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xa aaaaaaa0000000a) DebugInformationStrippedFromFile25 #32 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaa aaaaaaaaaaa) DebugInformationStrippedFromFile2 #33 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc24a7bda, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStr . . Forever and ever. I finally just control c'd the thing. How do I tell which thread creates which mutex without cma_meter? Unless there are other suggestions from engineering I will forward my original trace to the developer who knows the software and ask him if he can narrow this down. Pat | |||||
| 1551.7 | Same corruption, different place, I presume. | WTFN::SCALES | Despair is appropriate and inevitable. | Tue Jun 03 1997 23:45 | 23 |
.5> One of the things I tried (not in the sources I sent you) was setting up a .5> really big stack. This didnt seem to matter. Well, it's probably not a stack overflow problem then. (That wasn't the likely explanation.) If it's the unitialized pointer issue that Peter and I raised, then it's quite possible that changing the size of the stack would have no effect. .6> Unfortunately setting up CMA_METER is causing the program to fail in a .6> different way... This is not particularly surprising. Turning on metering forces the code down different paths, changing the execution timing, data allocation patterns, and dirtying of the stack. .6> Forever and ever. I finally just control c'd the thing. What we would want to see would be the frames just below where the pattern stops repeating... It looks like maybe something has corrupted one of the libc mutexes resulting in the error on each call to cma_tis_mutex_unlock(). Webb | |||||
| 1551.8 | Same result even with expirationTime initialized. | CREME::LAMPERT | Pat Lampert, UNIX Applications Support, 343-1050 | Wed Jun 04 1997 21:26 | 346 |
Note customer is using c++ 5.6. UNIX 3.2d-2. Sorry, I dont have information on how libfsdx.so was built yet. Taking the advice from an earlier note I dialed back into the customer's system and from a less stripped down version of the program I was able to determine that expirationTime was supposed to be... char *expirationTime= (char *) malloc(sizeof(char) * 8); I fixed the test program and re-ran. Unfortunately it still hangs in the same manner. After lowering the size of the stack I found I had more manageable results from ladebug. Here is what I found.... Sorry its a script log so its a bit messy... I defined expirationTime... . . main() . . char *expirationTime= (char *) malloc(sizeof(char) * 8); . . Recompiled... . . recompiling cxx -g -o fsdxSubThreads fsdxSubThreads.cxx -I. -I/usr/include/cxx -I/usr/users/marzomi/hpFsdx/fsdx/include/ -L/usr/users/marzomi/hpFsdx/fsdx/lib/ -lfsdx -L/usr/lib -ldnet_stub -lc_r -threads NYCSEX0000.btco.com> fsdxSubThreads ksh: fsdxSubThreads: not found NYCSEX0000.btco.com> ./fsdxSubThreads opened map file Sucessfully parsed map file calling open opened dsv regis map file 1 0 1 2 trd0trd1 It hangs so I control-c... NYCSEX0000.btco.com> NYCSEX0000.btco.com> NYCSEX0000.btco.com> NYCSEX0000.btco.com> CMA_METER="1" NYCSEX0000.btco.com> export CMA_METER NYCSEX0000.btco.com> ladebug fsdxSubThreads Welcome to the Ladebug Debugger Version 4.0-37 ------------------ object file name: fsdxSubThreads Reading symbolic information ...done (ladebug) run opened map file Sucessfully parsed map file calling open opened dsv regis map file 1 0 1 2 Thread received signal SEGV stopped at [_doprnt 0x3ff8102d7a8] (ladebug) where >0 0x3ff8102d7a8 in _doprnt(0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile38 #1 0x3ff81056d14 in sprintf(0x3ffc1cd2800, 0x3ffc1c943a0, 0x0, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile150 #2 0x3ff820302a8 in exc_get_message(0x3ffc1cd2a10, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #3 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #4 0x3ff8202ffcc in exc_raise(0x3ffc1cd2a10, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #5 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #6 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #7 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile108 #8 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xaaaaaaaa0000000a) DebugInformationStrippedFromFile25 #9 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile2 #10 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd2c1a, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStrippedFromFile77 #11 0x3ff8102a060 in __catopen(0x3ffc1cd2c10, 0x3ffc1c9b7c0, 0x3ffc1cd2c98, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #12 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd2e4b, 0x3ffc1cd2f58, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #13 0x3ff82030390 in exc_get_message(0x3ffc1cd3050, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #14 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #15 0x3ff8202ffcc in exc_raise(0x3ffc1cd3050, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #16 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #17 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #18 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile108 #19 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xaaaaaaaa0000000a) DebugInformationStrippedFromFile25 #20 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile2 #21 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd325a, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStrippedFromFile77 #22 0x3ff8102a060 in __catopen(0x3ffc1cd3250, 0x3ffc1c9b7c0, 0x3ffc1cd32d8, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #23 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd348b, 0x3ffc1cd3598, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #24 0x3ff82030390 in exc_get_message(0x3ffc1cd3690, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #25 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #26 0x3ff8202ffcc in exc_raise(0x3ffc1cd3690, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #27 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #28 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #29 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile108 #30 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xaaaaaaaa0000000a) DebugInformationStrippedFromFile25 #31 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile2 #32 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd389a, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStrippedFromFile77 #33 0x3ff8102a060 in __catopen(0x3ffc1cd3890, 0x3ffc1c9b7c0, 0x3ffc1cd3918, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #34 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd3acb, 0x3ffc1cd3bd8, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #35 0x3ff82030390 in exc_get_message(0x3ffc1cd3cd0, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #36 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #37 0x3ff8202ffcc in exc_raise(0x3ffc1cd3cd0, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #38 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #39 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #40 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile108 #41 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xaaaaaaaa0000000a) DebugInformationStrippedFromFile25 #42 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile2 #43 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd3eda, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStrippedFromFile77 #44 0x3ff8102a060 in __catopen(0x3ffc1cd3ed0, 0x3ffc1c9b7c0, 0x3ffc1cd3f58, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #45 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd410b, 0x3ffc1cd4218, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #46 0x3ff82030390 in exc_get_message(0x3ffc1cd4310, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #47 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #48 0x3ff8202ffcc in exc_raise(0x3ffc1cd4310, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #49 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #50 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #51 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile108 #52 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xaaaaaaaa0000000a) DebugInformationStrippedFromFile25 #53 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile2 #54 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd451a, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStrippedFromFile77 #55 0x3ff8102a060 in __catopen(0x3ffc1cd4510, 0x3ffc1c9b7c0, 0x3ffc1cd4598, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #56 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd474b, 0x3ffc1cd4858, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #57 0x3ff82030390 in exc_get_message(0x3ffc1cd4950, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #58 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #59 0x3ff8202ffcc in exc_raise(0x3ffc1cd4950, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #60 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #61 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0x73) DebugInformationStrippedFromFile86 #62 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xa9808, 0x3ff8102937c, 0x0) DebugInformationStrippedFromFile108 #63 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xa9808, 0x3ff8102937c, 0x0, 0x3ff810277f0, 0xa) DebugInformationStrippedFromFile25 #64 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0x104418, 0x3ff8102a024, 0x0, 0x3ff82030f74, 0x3a9e000000000) DebugInformationStrippedFromFile2 #65 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd4b5a, 0x3ff81029fe4, 0x4200000000) DebugInformationStrippedFromFile77 #66 0x3ff8102a060 in __catopen(0x3ffc1cd4b50, 0x3ffc1c9b7c0, 0x3ffc1cd4bd8, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #67 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd4d8b, 0x3ffc1cd4e98, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #68 0x3ff82030390 in exc_get_message(0x3ffc1cd4f90, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #69 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #70 0x3ff8202ffcc in exc_raise(0x3ffc1cd4f90, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #71 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #72 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0x73) DebugInformationStrippedFromFile86 #73 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0x0, 0x3ff8102937c, 0x0) DebugInformationStrippedFromFile108 #74 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0x0, 0x3ff8102937c, 0x0, 0x3ff810277f0, 0xa) DebugInformationStrippedFromFile25 #75 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0x0, 0x3ff8102a024, 0x0, 0x3ff82030f74, 0x0) DebugInformationStrippedFromFile2 #76 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd519a, 0x3ff81029fe4, 0x4200000000) DebugInformationStrippedFromFile77 #77 0x3ff8102a060 in __catopen(0x3ffc1cd5190, 0x3ffc1c9b7c0, 0x3ffc1cd5218, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #78 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd53cb, 0x3ffc1cd54d8, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #79 0x3ff82030390 in exc_get_message(0x3ffc1cd55d0, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #80 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #81 0x3ff8202ffcc in exc_raise(0x3ffc1cd55d0, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #82 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #83 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0x2020202000000073) DebugInformationStrippedFromFile86 #84 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0x2020202020202020, 0x3ff8102937c, 0x2020202020202020) DebugInformationStrippedFromFile108 #85 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0x2020202020202020, 0x3ff8102937c, 0x2020202020202020, 0x3ff810277f0, 0x202020200000000a) DebugInformationStrippedFromFile25 #86 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0x2020202020202020, 0x3ff8102a024, 0x2020202020202020, 0x3ff82030f74, 0x2020202020202020) DebugInformationStrippedFromFile2 #87 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd57da, 0x3ff81029fe4, 0x2020004220202020) DebugInformationStrippedFromFile77 #88 0x3ff8102a060 in __catopen(0x3ffc1cd57d0, 0x3ffc1c9b7c0, 0x3ffc1cd5858, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #89 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd5a0b, 0x3ffc1cd5b18, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #90 0x3ff82030390 in exc_get_message(0x3ffc1cd5c10, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #91 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #92 0x3ff8202ffcc in exc_raise(0x3ffc1cd5c10, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #93 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #94 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #95 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile108 #96 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xaaaaaaaa0000000a) DebugInformationStrippedFromFile25 #97 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile2 #98 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd5e1a, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStrippedFromFile77 #99 0x3ff8102a060 in __catopen(0x3ffc1cd5e10, 0x3ffc1c9b7c0, 0x3ffc1cd5e98, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #100 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd604b, 0x3ffc1cd6158, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #101 0x3ff82030390 in exc_get_message(0x3ffc1cd6250, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #102 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #103 0x3ff8202ffcc in exc_raise(0x3ffc1cd6250, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #104 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #105 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #106 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile108 #107 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xaaaaaaaa0000000a) DebugInformationStrippedFromFile25 #108 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile2 #109 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd645a, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStrippedFromFile77 #110 0x3ff8102a060 in __catopen(0x3ffc1cd6450, 0x3ffc1c9b7c0, 0x3ffc1cd64d8, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #111 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd668b, 0x3ffc1cd6798, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #112 0x3ff82030390 in exc_get_message(0x3ffc1cd6890, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #113 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #114 0x3ff8202ffcc in exc_raise(0x3ffc1cd6890, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #115 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #116 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #117 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile108 #118 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xaaaaaaaa0000000a) DebugInformationStrippedFromFile25 #119 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile2 #120 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd6a9a, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStrippedFromFile77 #121 0x3ff8102a060 in __catopen(0x3ffc1cd6a90, 0x3ffc1c9b7c0, 0x3ffc1cd6b18, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #122 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd6ccb, 0x3ffc1cd6dd8, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #123 0x3ff82030390 in exc_get_message(0x3ffc1cd6ed0, 0x0, 0x0, 0x3e, 0x44, 0x0) DebugInformationStrippedFromFile87 #124 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x44, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #125 0x3ff8202ffcc in exc_raise(0x3ffc1cd6ed0, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #126 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #127 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0xaaaaaaaa00000073) DebugInformationStrippedFromFile86 #128 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile108 #129 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xaaaaaaaa0000000a) DebugInformationStrippedFromFile25 #130 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile2 #131 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x3ff81019828, 0x3ffc1cd70da, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStrippedFromFile77 #132 0x3ff8102a060 in __catopen(0x3ffc1cd70d0, 0x3ffc1c9b7c0, 0x3ffc1cd7158, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #133 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd730b, 0x3ffc1cd7418, 0x104418, 0x0, 0x6) DebugInformationStrippedFromFile91 #134 0x3ff82030390 in exc_get_message(0x3ffc1cd7510, 0x0, 0x0, 0x3e, 0x43, 0x0) DebugInformationStrippedFromFile87 #135 0x3ff8202fecc in exc_report(0x0, 0x3e, 0x43, 0x0, 0x3ff8202ffd0, 0x11ffffd58) DebugInformationStrippedFromFile87 #136 0x3ff8202ffcc in exc_raise(0x3ffc1cd7510, 0x0, 0x78018, 0x0, 0x0, 0x104418) DebugInformationStrippedFromFile87 #137 0x3ff82030198 in exc_raise_status(0x45586732, 0x2, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile87 #138 0x3ff8202fe54 in cma__error(0x0, 0x0, 0x0, 0x0, 0x3ff82049380, 0x0) DebugInformationStrippedFromFile86 #139 0x3ff8204937c in UnknownProcedure9FromFile108(0x3ffc0d18650, 0x0, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile108 #140 0x3ff81029228 in __cma_tis_mutex_unlock(0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff8102937c, 0xaaaaaaaaaaaaaaaa, 0x3ff810277f0, 0xaaaaaaaa0000000a) DebugInformationStrippedFromFile25 #141 0x3ff810277ec in __rec_mutex_unlock(0x3ffc0d18640, 0xaaaaaaaaaaaaaaaa, 0x3ff8102a024, 0xaaaaaaaaaaaaaaaa, 0x3ff82030f74, 0x104418) DebugInformationStrippedFromFile2 #142 0x3ff81045f14 in getenv(0x3ff8102a064, 0x0, 0x0, 0x3ffc1cd771a, 0x3ff81029fe4, 0xaaaa0042aaaaaaaa) DebugInformationStrippedFromFile77 #143 0x3ff8102a060 in __catopen(0x3ffc1cd7710, 0x3ffc1c9b7c0, 0x3ffc1cd7798, 0x3ffc1c94cf0, 0x3ffc1c9b7d8, 0x2) DebugInformationStrippedFromFile28 #144 0x3ff82030f8c in cma__error_inq_text(0x177db037, 0x3ffc1cd794b, 0x3ffc1cd7a58, 0x104418, 0x0, 0x0) DebugInformationStrippedFromFile91 #145 0x3ff82030390 in exc_get_message(0x3ffc1cd7e78, 0x0, 0x0, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile87 #146 0x3ff8202fecc in exc_report(0x0, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0x3ff8202ffd0, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile87 #147 0x3ff8202ffcc in exc_raise(0x3ffc1cd7e78, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile87 #148 0x3ff82030240 in exc_pop_ctx(0x3ffc1cd7bd0, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0x3ff8203b620, 0x0, 0xaaaaaaaa00000000) DebugInformationStrippedFromFile87 #149 0x3ff82041e50 in cma__thread_base(0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0x2, 0x120014db0) DebugInformationStrippedFromFile101 (ladebug) show thread Id State 0xfffffc0007b14b80 running 0xfffffc0005184000 unstarted 0xfffffc0005184800 unstarted 0xfffffc000195d400 unstarted 0xfffffc000195c800 unstarted 0xfffffc0006933800 unstarted 0xfffffc0006932c00 unstarted >* 0xfffffc0006933000 stopped (ladebug) thread 0xfffffc0007b14b80 Id State > 0xfffffc0007b14b80 running (ladebug) where >0 0x3ff8280e574 in msg_receive_trap(0xe7218, 0x3ffc1c9e9f8, 0x0, 0x39e7000000000, 0x768, 0x3d) DebugInformationStrippedFromFile19 #1 0x3ff82804dd0 in msg_receive(0x4e, 0x0, 0xd8018, 0x0, 0x1, 0x3ffc1c9e9f8) DebugInformationStrippedFromFile6 #2 0x3ff820485dc in cma__vp_sleep(0x2801000000, 0x1, 0x7, 0xe7218, 0x3ff00000000, 0x3ffc3013530) DebugInformationStrippedFromFile106 #3 0x3ff8202e958 in cma__dispatch(0x7, 0xe7218, 0x3ff00000000, 0x3ffc3013530, 0x3ff82027e34, 0x3ffc30002b8) DebugInformationStrippedFromFile85 #4 0x3ff82027e30 in cma__int_wait(0x11ffff328, 0xe7218, 0x3ffc1c9e9f8, 0x1000, 0x141fd000, 0x141cc300) DebugInformationStrippedFromFile3 #5 0x3ff820411e0 in cma_thread_join(0x11ffff748, 0x11ffffaf0, 0x11ffffae8, 0x3ff82041bf0, 0x0, 0x3ffc01e3de0) DebugInformationStrippedFromFile101 #6 0x3ff82037508 in pthread_join(0x104c18, 0x400000000000008, 0x11ffffc00, 0x3ffc1c9e9f8, 0x120015678, 0xe7418) DebugInformationStrippedFromFile95 #7 0x1200156ec in main() fsdxSubThreads.cxx:292 (ladebug) thread 0xfffffc0005184000 Id State > 0xfffffc0005184000 unstarted (ladebug) where >0 0x3ff8280e574 in msg_receive_trap(0x50218, 0x105c18, 0x0, 0x359d000000000, 0x6b8, 0x39) DebugInformationStrippedFromFile19 #1 0x3ff82804dd0 in msg_receive(0x47, 0x0, 0x90018, 0x0, 0x1, 0x105c18) DebugInformationStrippedFromFile6 #2 0x3ff820485dc in cma__vp_sleep(0x2801000000, 0x1, 0xa, 0x50218, 0x0, 0x2) DebugInformationStrippedFromFile106 #3 0x3ff8202e958 in cma__dispatch(0xa, 0x50218, 0x0, 0x2, 0x3ff82027e34, 0x10c848) DebugInformationStrippedFromFile85 #4 0x3ff82027e30 in cma__int_wait(0x3ff82037eb4, 0x50218, 0x105c18, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile3 #5 0x3ff82036a30 in pthread_cond_wait(0x3ffbffd31d8, 0x0, 0x3ffbffd32c0, 0x50118, 0x10c818, 0x50218) DebugInformationStrippedFromFile95 #6 0x3ffbffd31d4 in send_thread(parm=0xa3a08) btcu_s.c:129 #7 0x3ff82041ccc in cma__thread_base(0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xa3a08, 0x3ffbffd3150) DebugInformationStrippedFromFile101 (ladebug) thread 0xfffffc0005184800 Id State > 0xfffffc0005184800 unstarted (ladebug) where >0 0x3ff8108c648 in __recv(0x1, 0x105818, 0x0, 0x1, 0x0, 0x1) DebugInformationStrippedFromFile401 #1 0x3ffbffd2d5c in xrecv(s=3, buf=0x3ffc1cafb38, len=4, flag=0) btcu_r.c:409 #2 0x3ffbffd2678 in receive_thread(parm=0xa3a08) btcu_r.c:218 #3 0x3ff82041ccc in cma__thread_base(0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xa3a08, 0x3ffbffd2630) DebugInformationStrippedFromFile101 (ladebug) thread 0xfffffc000195d400 Id State > 0xfffffc000195d400 unstarted (ladebug) where >0 0x3ff8280e574 in msg_receive_trap(0x50218, 0x105418, 0x0, 0x3bcf000000000, 0x778, 0x3e) DebugInformationStrippedFromFile19 #1 0x3ff82804dd0 in msg_receive(0x59, 0x0, 0x90018, 0x0, 0x1, 0x105418) DebugInformationStrippedFromFile6 #2 0x3ff820485dc in cma__vp_sleep(0x2801000000, 0x1, 0x10, 0x50218, 0x0, 0x2) DebugInformationStrippedFromFile106 #3 0x3ff8202e958 in cma__dispatch(0x10, 0x50218, 0x0, 0x2, 0x3ff82027e34, 0xe7748) DebugInformationStrippedFromFile85 #4 0x3ff82027e30 in cma__int_wait(0x3ff82037eb4, 0x50218, 0x105418, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile3 #5 0x3ff82036a30 in pthread_cond_wait(0x3ffbffd31d8, 0x0, 0x3ffbffd32c0, 0x50118, 0xe7718, 0x50218) DebugInformationStrippedFromFile95 #6 0x3ffbffd31d4 in send_thread(parm=0xa3808) btcu_s.c:129 #7 0x3ff82041ccc in cma__thread_base(0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xa3808, 0x3ffbffd3150) DebugInformationStrippedFromFile101 (ladebug) thread 0xfffffc000195c800 Id State > 0xfffffc000195c800 unstarted (ladebug) where >0 0x3ff8280e574 in msg_receive_trap(0x50218, 0x105018, 0x0, 0x3c49000000000, 0x780, 0x3e) DebugInformationStrippedFromFile19 #1 0x3ff82804dd0 in msg_receive(0x5b, 0x0, 0x90018, 0x0, 0x1, 0x105018) DebugInformationStrippedFromFile6 #2 0x3ff820485dc in cma__vp_sleep(0x2801000000, 0x1, 0x13, 0x50218, 0x0, 0xa3428) DebugInformationStrippedFromFile106 #3 0x3ff8202e958 in cma__dispatch(0x13, 0x50218, 0x0, 0xa3428, 0x3ff82027e34, 0x50218) DebugInformationStrippedFromFile85 #4 0x3ff82027e30 in cma__int_wait(0x3ff82037eb4, 0x50218, 0x105018, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile3 #5 0x3ff82036a30 in pthread_cond_wait(0x3ffbffd27f4, 0x0, 0x3ffbffd27ac, 0x50118, 0xe7818, 0x50218) DebugInformationStrippedFromFile95 #6 0x3ffbffd27f0 in receive_thread(parm=0xa3808) btcu_r.c:254 #7 0x3ff82041ccc in cma__thread_base(0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xa3808, 0x3ffbffd2630) DebugInformationStrippedFromFile101 (ladebug) thread 0xfffffc0006933800 Id State > 0xfffffc0006933800 unstarted (ladebug) where >0 0x3ff8280e574 in msg_receive_trap(0x16100, 0x0, 0x4f1c000000000, 0x3a24000000000, 0x768, 0x3d) DebugInformationStrippedFromFile19 #1 0x3ff82804dd0 in msg_receive(0x4f, 0x14, 0xde100, 0x2, 0x3ffc0d184e8, 0x3ffc1cc5370) DebugInformationStrippedFromFile6 #2 0x3ff820485dc in cma__vp_sleep(0x28010de100, 0x0, 0x16, 0x104c18, 0x0, 0x3ffc1cc5370) DebugInformationStrippedFromFile106 #3 0x3ff8202e958 in cma__dispatch(0x16, 0x104c18, 0x0, 0x3ffc1cc5370, 0x3ff820341d4, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile85 #4 0x3ff820341d0 in cma__int_mutex_block(0x3ffc1cc5470, 0xa9888, 0x0, 0x0, 0x0, 0x0) DebugInformationStrippedFromFile93 #5 0x3ff82037eb0 in pthread_mutex_lock(0x0, 0x0, 0x120014f34, 0x104c18, 0x120014f0c, 0xe7518) DebugInformationStrippedFromFile95 #6 0x120014f30 in fsdxReads(arg=0x0) fsdxSubThreads.cxx:80 #7 0x3ff82041ccc in cma__thread_base(0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0x0, 0x120014db0) DebugInformationStrippedFromFile101 (ladebug) thread 0xfffffc0006932c00 Id State > 0xfffffc0006932c00 unstarted (ladebug) where >0 0x3ff8280e574 in msg_receive_trap(0x16200, 0x0, 0x4f1c000000000, 0x3a24000000000, 0x768, 0x3d) DebugInformationStrippedFromFile19 #1 0x3ff82804dd0 in msg_receive(0x50, 0x14, 0xde180, 0x2, 0x3ffc0d184e8, 0x3ffc1ccd370) DebugInformationStrippedFromFile6 #2 0x3ff820485dc in cma__vp_sleep(0x28010de180, 0x0, 0x19, 0x104818, 0x0, 0x3ffc1ccd370) DebugInformationStrippedFromFile106 #3 0x3ff8202e958 in cma__dispatch(0x19, 0x104818, 0x0, 0x3ffc1ccd370, 0x3ff820341d4, 0xaaaaaaaaaaaaaaaa) DebugInformationStrippedFromFile85 #4 0x3ff820341d0 in cma__int_mutex_block(0x3ffc1ccd470, 0xa9908, 0x0, 0x0, 0x1, 0x0) DebugInformationStrippedFromFile93 #5 0x3ff82037eb0 in pthread_mutex_lock(0x1, 0x0, 0x120014f34, 0x104818, 0x120014f0c, 0xe7518) DebugInformationStrippedFromFile95 #6 0x120014f30 in fsdxReads(arg=0x1) fsdxSubThreads.cxx:80 #7 0x3ff82041ccc in cma__thread_base(0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0xaaaaaaaaaaaaaaaa, 0x1, 0x120014db0) DebugInformationStrippedFromFile101 (ladebug) call cma_debug() Thread received signal SEGV Called routine terminated by longjmp or fault stopped at [_doprnt 0x3ff8102d7a8] (ladebug) quit NYCSEX0000.btco.com> logout ksh: logout: not found NYCSEX0000.btco.com> Connection closed by foreign host. NYGTSX0002.btco.com> ^Dlogout Local -011- Session 1 disconnected from NYGTSX0002 *logout Local -020- Logged out port 8 on server RED_B39_073_D2 NO CARRIER ~ [EOT] csh> ^D script done on Wed Jun 4 19:59:56 1997 | |||||
| 1551.9 | DCETHD::BUTENHOF | Dave Butenhof, DECthreads | Thu Jun 05 1997 08:45 | 59 | |
Sequence: some exception was raised in the thread, and was not handled. The stack was unwound to cma__thread_base (a DECthreads frame at the base of every created thread). It handles all exceptions, mostly because it needs to CATCH alert and exit. Because this was NOT one of those exceptions, it tried to reraise the exception to result in process termination. Because there were no more handlers on the DECthreads exception stack, DECthreads attempted to format and print a message reporting the nature of the exception prior to terminating the process. However, in the process of opening the message catalog file, libc called something that called getenv, which attempted to unlock a mutex (presumably after having locked it successfully, though we can't know that for sure from the stack trace). The attempt to unlock, however, suffered from an error of some sort, resulting in a call to cma_error to raise an exception. Which, because there were no exception handlers, attempted to terminate the process, after writing an informative message by opening a message catalog... And so forth. Because you were running under the debugger, it appears that none of these exceptions were signals (or, at least, not the signals for which the default action is "catch"... which should include all of the hardware errors like SEGV, unless you have a .dbxinit that "ignore"s some or all?) There are a few odd things here. First, the error reported by __cma_tis_mutex_unlock must be occurring AFTER the mutex has been unlocked, because the subsequent cycles are also able to reach the unlock call. I don't see how this could make any sense. Furthermore, although the cycle repeats identically 13 times, on the 14th cycle it appears that catopen manages to complete, and we proceed to format a text string (sprintf) using the message we've acquired -- leading to a SEGV in the depths of the printf subsystem. (This likely implies that we have acquired garbage from the catalog file, implying that the catalog open & get operations only APPEARED to succeed.) If you can reproduce this again under the debugger with a breakpoint set on cma__error and/or exc_raise_status, and report the actual value of $r16, we would be able to determine the exception. (I don't trust the value of argument registers shown in a stack trace, and, if the values shown in your output are correct, the argument is garbage.) Possible trivia: I'm a little uncomfortable with your build command. Offhand, I don't see any harm, but you're putting libc_r in the wrong place in the search order. The library is automatically included (in the proper place) by "-threads", so I would like you to remove the explicit -lc_r. Are you sure expirationTime is initialized correctly? What is the actual TYPE of the data (returned in?) the buffer to which expirationTime points? Is it a character string of 8 bytes? (Seven characters plus a null terminator?) If not, then I would strongly suggest that you utilize the appropriate TYPE rather than making assumptions. For example, I can't help wondering if this is a "struct timeval" or a "struct timespec". If so, you should declare it (and, if it must be allocated dynamically, specify the allocation size) using the proper type. For example, (and I don't know if this is what you intend), if the type was the "struct timespec" commonly used in POSIX realtime and threads interfaces, you should declare it as "struct timespec *expirationTime;" and allocate it using "= malloc(sizeof(struct timespec));" (The size of a struct timespec is 16 bytes (although the older BSD struct timeval is 8 bytes, that's NOT a good [or necessary] assumption to make, and at best it obscures the meaning of the code.) | |||||
| 1551.10 | Thanks. I will try out your suggestions. | CREME::LAMPERT | Pat Lampert, UNIX Applications Support, 343-1050 | Thu Jun 05 1997 11:13 | 23 |
>>I'm a little uncomfortable with your build command. Thanks for pointing this out. I had not intended to leave lc_r in the make file. The customer was manually including all the re-entrant libraries and specifying -lpthreads. I edited the make file intending to replace all this with "-threads", and accidentally left the lc_r in place. >>Are you sure expirationTime is initialized correctly? Now that its daytime, I will check with the customer on this. I am not familiar with the code so my guess was based on looking at another similar module. Thanks again. Pat | |||||