Conference turris::digital_unix

                      <<< Note 9248.0 by TAEC::GALLERI >>>
         -< Title: Troubles with DECthreads under Digital UNIX V3.2G >-

Hi guys,

    We have a customer that has developped applications using our DECss7
    APIs and numerous threads for their own usage. From time to time, they
    bump into the following problem: whilst applications are normally using
    only less than 5% of CPU, they come to a full 90+% CPU usage (and even
    up to 250% on a 3 CPUs machine !!!), but with no actual concrete
    operations performed (I mean, it seems that some threads come to loop
    abnormally with no apparent reason :-( [N.B.: machines are Alpha 2100
    running Digital UNIX V3.2G]

    By experience, we have seen this kind of behaviour being caused by
    thread stack overflow [we would have prefered a more explicit
    behaviour, such a somehow signalling "Thread Stack Overflow" somewhere,
    but ...] and the fact is that we achieved to go a bit further by
    increasing the size of some of the application threads (namely, those
    related to DECss7) from 32kb to more than 200kb, but the same
    misbehaviour still occurs at a different point.

Now, the questions:

    1) Did you already hear of such kind of troubles with thread stack size
    ? Are there means to get a more usable error message related to thread
    stack overflow ?

    2) Is there a mean by which we can configure either the OS or the
    DECthreads library to automatically allocate a stack with a
    user-defined size (much greater than 32kb) ? What do you think would be
    a more reasonable default stack size value ?

    3) When the problem occurs, the customer attached to the process either
    through dbx or ladebug to try to find out what thoses threads were
    trying to do. We also call cma_debug(), but this appears to block at
    some point too. In the end, they tried to kill the process, but this
    appears to be difficult, and even impossible !?!? Indeed, the process
    revealed itself to be in a non-interruptible state ... I reproduce
    hereafter the output of a "dbx -k /vmunix" session I made to try to
    find out where those threads were in the kernel. It would be very
    interesting for us if someone can understand and deduce something
    useful from this mess ;-)

    Generally speaking, my impression is that we need a bit (not to say
    more :-) of DECthreads consultancy to be able to understand what's
    going on. That said, I (and our customer too, of course !!!) cannot be
    satisfied by such behaviour of their applications. We definitely need a
    more general and usable mean to troubleshoot such situation.

    No use to tell you that this *VERY* urgent matter and that quick and
    efficient answers are expected ;-)

Thank you very much for your attention,

Xav


[2] record output /tmp/vmunix.dbx (0 lines)
(dbx) 
Thread 0xfffffc000ab34b80:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x3ff8056f4dc, 0x3ffc01e3500, 0xffffffffffffff35, 0x100)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]

Thread 0xfffffc001fe13c00:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 syscall(0x3, 0x3ffbfcc97f8, 0xa70c8, 0x41, 0x9d)
["../../../../src/kernel/arch/alpha/syscall_trap.c":658, 0xfffffc000047cb24]
   3 _Xsyscall(0x8, 0x3ff8051c758, 0x3ffc01e3500, 0x80c020, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]
warning: thread 0xfffffc0010c25c00 appears to be swapped out

Thread 0xfffffc0010c25c00:
warning: thread 0xfffffc0010c25c00 appears to be swapped out
warning: thread 0xfffffc0010c25c00 appears to be swapped out
More (n if no)?More (n if no)?warning: thread 0xfffffc0010c25c00 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc0010c25c00 appears to be swapped out
warning: thread 0xfffffc0010c25c00 appears to be swapped out
   1 msg_dequeue(0xfffffc0000200400, 0x28, 0x0, 0xfffffc001fe16000, 0xffffffffa2d178f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0xb0fa0, 0x0, 0x28, 0xc, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0xb0fa0, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]
warning: thread 0xfffffc0001813800 appears to be swapped out

Thread 0xfffffc0001813800:
warning: thread 0xfffffc0001813800 appears to be swapped out
warning: thread 0xfffffc0001813800 appears to be swapped out
warning: thread 0xfffffc0001813800 appears to be swapped out
warning: thread 0xfffffc0001813800 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc0001813800 appears to be swapped out
warning: thread 0xfffffc0001813800 appears to be swapped out
   1 msg_dequeue(0xfffffc000046ba08, 0x28, 0x0, 0xc0000, 0xffffffffa23bf8f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]More (n if no)?More (n if
no)?   2 msg_receive_trap(0xbefa0, 0x0, 0x28, 0xf, 0x0) ["../../../../src/kernel/ker
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0xbefa0, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]
warning: thread 0xfffffc0010c25000 appears to be swapped out

Thread 0xfffffc0010c25000:
warning: thread 0xfffffc0010c25000 appears to be swapped out
warning: thread 0xfffffc0010c25000 appears to be swapped out
warning: thread 0xfffffc0010c25000 appears to be swapped out
warning: thread 0xfffffc0010c25000 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc0010c25000 appears to be swapped out
warning: thread 0xfffffc0010c25000 appears to be swapped out
   1 mpsleep(0xfffffc00134a4500, 0x11a, 0xfffffc00002564f8, 0x0, 0x0)
["../../../../src/kernel/bsd/kern_synch.c":446, 0xfffffc000042c0c4]
   2 sosleep(0xfffffc00134a4500, 0x1, 0xfffffc000000011a, 0x0, 0xfffffc001ab06a00)
["../../../../src/kernel/bsd/uipc_socket2.c":692, 0xfffffc0000259668]
   3 sosbwait(0xfffffc00134a4560, 0xfffffc00134a4500, 0x0, 0xfffffc00134a4500, 0x3ffffff0168)
["../../../../src/kernel/bsd/uipc_socket2.c":668, 0xfffffc00002595d4]
   4 soreceive(0xffffffffa23cf6a0, 0x0, 0xffffffffa23cf768, 0xffffffffa23cc000, 0x0)
["../../../../src/kernel/bsd/uipc_socket.c":1390, 0xfffffc00002579a8]More (n if no)?More (n if
no)?   5 soo_read(0xffffffffa23cf708, 0x16, 0xfffffc001c0b9300, 0xfffffc000
   6 rwuio(0xfffffc000ab34210, 0xcf806, 0xffffffffa23cf8b8, 0xffffffffa23cf768, 0x0)
["../../../../src/kernel/bsd/sys_generic.c":1069, 0xfffffc00002523dc]
   7 read(0xfffffc0000002178, 0xffffffffa23cf758, 0x40c4789c60000000, 0x217800000001, 0x0)
["../../../../src/kernel/bsd/sys_generic.c":1021, 0xfffffc00002522e8]
   8 syscall(0x3, 0x3ffbfc9b2f0, 0xcd323, 0x41, 0x3)
["../../../../src/kernel/arch/alpha/syscall_trap.c":519, 0xfffffc000047c584]
   9 _Xsyscall(0x8, 0x3ff8051c018, 0x40000000ae0, 0x16, 0xcf5c4)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]

Thread 0xfffffc0001813000:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x3ff80574d44, 0x3ffc01e3500, 0xdd1c48, 0x1)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]

Thread 0xfffffc0001812400:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]More
(n if no)?More (n if no)?   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262,
0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e5e0, 0x3ffc01e3500, 0x0, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]
warning: thread 0xfffffc0001812c00 appears to be swapped out

Thread 0xfffffc0001812c00:
warning: thread 0xfffffc0001812c00 appears to be swapped out
warning: thread 0xfffffc0001812c00 appears to be swapped out
warning: thread 0xfffffc0001812c00 appears to be swapped out
warning: thread 0xfffffc0001812c00 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc0001812c00 appears to be swapped out
warning: thread 0xfffffc0001812c00 appears to be swapped out
   1 msg_dequeue(0xfffffc000065a121, 0x28, 0x0, 0x1e000, 0xffffffffa14278f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0xdf1580, 0x0, 0x28, 0x1b, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0xdf1580, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]

Thread 0xfffffc001ed60000:
More (n if no)?>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919,
0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x3ff805895d8, 0x3ffc01e3500, 0xdd1c48, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]

Thread 0xfffffc001647f400:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e5e0, 0x3ffc01e3500, 0x0, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]

Thread 0xfffffc001647ec00:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x3ff805895d8, 0x3ffc01e3500, 0xdd1c48, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]More (n if no)?More (n if
no)?
Thread 0xfffffc001647e800:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e5e0, 0x3ffc01e3500, 0x0, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]

Thread 0xfffffc001647f800:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e5e0, 0x3ffc01e3500, 0x0, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]

Thread 0xfffffc001647fc00:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 msg_dequeue(0x64, 0x28, 0x0, 0xdae000, 0xffffffffa164f8f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]More (n if no)?More (n if
no)?   2 msg_receive_trap(0xe208a0, 0x0, 0x28, 0x2d, 0x0)
["../../../../src/kernel/kern/ipc_basic
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0xe208a0, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]

Thread 0xfffffc001647e000:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e5e0, 0x3ffc01e3500, 0x0, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]

Thread 0xfffffc001647f000:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x3ff80589608, 0x3ffc01e3500, 0xdd1c48, 0xdd1c18)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]

Thread 0xfffffc001aff7800:
More (n if no)?>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919,
0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 syscall(0x3ff8051ad30, 0x1, 0xffffffffffffffff, 0x21, 0x59)
["../../../../src/kernel/arch/alpha/syscall_trap.c":658, 0xfffffc000047cb24]
   3 _Xsyscall(0x8, 0x3ff8051ad38, 0x14003ae70, 0x26, 0x5)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]

Thread 0xfffffc001e08f800:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 syscall(0x140035388, 0x1, 0x0, 0x21, 0x66)
["../../../../src/kernel/arch/alpha/syscall_trap.c":658, 0xfffffc000047cb24]
   3 _Xsyscall(0x8, 0x3ff8051c0e8, 0x14003ae70, 0x9, 0xe41790)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]

Thread 0xfffffc0009961400:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 syscall(0x140035388, 0x1, 0x0, 0x21, 0x66)
["../../../../src/kernel/arch/alpha/syscall_trap.c":658, 0xfffffc000047cb24]
   3 _Xsyscall(0x8, 0x3ff8051c0e8, 0x14003ae70, 0xb, 0xe49790)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]More (n if no)?More (n
if no)?
Thread 0xfffffc0009961c00:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 syscall(0x140035388, 0x1, 0x0, 0x21, 0x66)
["../../../../src/kernel/arch/alpha/syscall_trap.c":658, 0xfffffc000047cb24]
   3 _Xsyscall(0x8, 0x3ff8051c0e8, 0x14003ae70, 0xc, 0xe51790)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]

Thread 0xfffffc0009961000:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 syscall(0x140035388, 0x1, 0x0, 0x21, 0x66)
["../../../../src/kernel/arch/alpha/syscall_trap.c":658, 0xfffffc000047cb24]
   3 _Xsyscall(0x8, 0x3ff8051c0e8, 0x14003ae70, 0xf, 0xe59790)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]

Thread 0xfffffc001e08fc00:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 syscall(0x140035388, 0x1, 0x0, 0x21, 0x66)
["../../../../src/kernel/arch/alpha/syscall_trap.c":658, 0xfffffc000047cb24]More (n if no)?More
(n if no)?   3 _Xsyscall(0x8, 0x3ff8051c0e8, 0x14003ae70, 0x11, 0xe61790)
["../../../../src/kernel/arch/alpha/l

Thread 0xfffffc0009960800:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 syscall(0x140035388, 0x1, 0x0, 0x21, 0x66)
["../../../../src/kernel/arch/alpha/syscall_trap.c":658, 0xfffffc000047cb24]
   3 _Xsyscall(0x8, 0x3ff8051c0e8, 0x14003ae70, 0x10, 0xe69790)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]

Thread 0xfffffc0009960000:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x3ff8051ad30, 0x14003ae70, 0x30, 0x5)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]

Thread 0xfffffc0009960400:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]More
(n if no)?More (n if no)?   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262,
0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e5e0, 0x3ffc01e3500, 0x0, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]

Thread 0xfffffc0014023800:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e5e0, 0x3ffc01e3500, 0x0, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]

Thread 0xfffffc000b0bf800:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e5e0, 0x3ffc01e3500, 0x0, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]More (n if no)?More (n
if no)?
Thread 0xfffffc000dde5800:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x3ff8058915c, 0x3ffc01e3500, 0xe91580, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]

Thread 0xfffffc001fe16c00:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x3ff80574c00, 0x3ffc01e3500, 0xdd1c48, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]

Thread 0xfffffc000775f800:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]More (n if
no)?More (n if no)?   3 exception_exit(0x8, 0x3ff805895d4, 0x3ffc01e3500, 0xdd1c48,
0x1414e93f8) ["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b

Thread 0xfffffc000775e400:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x1200610b0, 0x14003ae70, 0x33, 0x5)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]

Thread 0xfffffc000775f400:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e5e0, 0x3ffc01e3500, 0x0, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]

Thread 0xfffffc000775ec00:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]More
(n if no)?More (n if no)?   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262,
0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e5e0, 0x3ffc01e3500, 0x0, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]

Thread 0xfffffc000335b800:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 syscall(0x3ff8051ad30, 0x1, 0xffffffffffffffff, 0x21, 0x59)
["../../../../src/kernel/arch/alpha/syscall_trap.c":658, 0xfffffc000047cb24]
   3 _Xsyscall(0x8, 0x3ff8051ad38, 0x14003ae70, 0x25, 0x5)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]

Thread 0xfffffc00197e1800:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x3ff80574c00, 0x3ffc01e3500, 0xdd1c48, 0x1414e93f8)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]More (n if no)?More (n if
no)?warning: thread 0xfffffc001d956400 appears to be swapped out

Thread 0xfffffc001d956400:
warning: thread 0xfffffc001d956400 appears to be swapped out
warning: thread 0xfffffc001d956400 appears to be swapped out
warning: thread 0xfffffc001d956400 appears to be swapped out
warning: thread 0xfffffc001d956400 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
warning: thread 0xfffffc001d956400 appears to be swapped out
warning: thread 0xfffffc001d956400 appears to be swapped out
   1 mpsleep(0xfffffc000b5c4e00, 0x11a, 0xfffffc0000625f90, 0x0, 0x0)
["../../../../src/kernel/bsd/kern_synch.c":446, 0xfffffc000042c0c4]
   2 sosleep(0xfffffc000b5c4e00, 0xffffffffa1e97720, 0xfffffc000000011a, 0x0,
0xfffffc000025a85c) ["../../../../src/kernel/bsd/uipc_socket2.c":692, 0xfffffc0000259668]
   3 accept1(0xfffffc000ab34210, 0xffffffffa1e978c8, 0xffffffffa1e978b8, 0x1,
0xffffffffa1e978b8) ["../../../../src/kernel/bsd/uipc_syscalls.c":374, 0xfffffc000025a8d8]
   4 oaccept(0xffffffffa1e978b8, 0x1, 0xffffffffa1e978b8, 0xfffffc0013203720,
0xfffffc000047c588) ["../../../../src/kernel/bsd/uipc_syscalls.c":319, 0xfffffc000025a7ac]
   5 syscall(0x13a1320, 0x0, 0x1, 0x30, 0x63)
["../../../../src/kernel/arch/alpha/syscall_trap.c":519, 0xfffffc000047c584]More (n if no)?More
(n if no)?   6 _Xsyscall(0x8, 0x3ff8051a5c8, 0x40000000ae0, 0x8, 0x13a1798)
["../../../../src/kernel/arch/alpha/l
warning: thread 0xfffffc001d957000 appears to be swapped out

Thread 0xfffffc001d957000:
warning: thread 0xfffffc001d957000 appears to be swapped out
warning: thread 0xfffffc001d957000 appears to be swapped out
warning: thread 0xfffffc001d957000 appears to be swapped out
warning: thread 0xfffffc001d957000 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc001d957000 appears to be swapped out
warning: thread 0xfffffc001d957000 appears to be swapped out
   1 msg_dequeue(0xfffffc000046ba08, 0x28, 0x0, 0xfffffc0013203720, 0xffffffffa1e878f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0x13b8c60, 0x0, 0x28, 0x72, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x13b8c60, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]
warning: thread 0xfffffc001d957400 appears to be swapped out

Thread 0xfffffc001d957400:
warning: thread 0xfffffc001d957400 appears to be swapped out
More (n if no)?More (n if no)?warning: thread 0xfffffc001d957400 appears to be swapped out
warning: thread 0xfffffc001d957400 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc001d957400 appears to be swapped out
warning: thread 0xfffffc001d957400 appears to be swapped out
   1 msg_dequeue(0xfffffc000046ba08, 0x28, 0x0, 0x140030000, 0xffffffffa1e378f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0x1a69330, 0x0, 0x28, 0x75, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1a69330, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]
warning: thread 0xfffffc001d956000 appears to be swapped out

Thread 0xfffffc001d956000:
warning: thread 0xfffffc001d956000 appears to be swapped out
warning: thread 0xfffffc001d956000 appears to be swapped out
warning: thread 0xfffffc001d956000 appears to be swapped out
warning: thread 0xfffffc001d956000 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc001d956000 appears to be swapped out
warning: thread 0xfffffc001d956000 appears to be swapped out
   1 msg_dequeue(0x0, 0x28, 0x0, 0x80000, 0xffffffffa1e7f8f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]More (n if no)?More (n if
no)?   2 msg_receive_trap(0x1a81330, 0x0, 0x28, 0x78, 0x0)
["../../../../src/kernel/kern/ipc_basics
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1a81330, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]
warning: thread 0xfffffc000d326400 appears to be swapped out

Thread 0xfffffc000d326400:
warning: thread 0xfffffc000d326400 appears to be swapped out
warning: thread 0xfffffc000d326400 appears to be swapped out
warning: thread 0xfffffc000d326400 appears to be swapped out
warning: thread 0xfffffc000d326400 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc000d326400 appears to be swapped out
warning: thread 0xfffffc000d326400 appears to be swapped out
   1 msg_dequeue(0x0, 0x28, 0x0, 0x72000, 0xffffffffa1e778f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0x1a99330, 0x0, 0x28, 0x7b, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1a99330, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]
warning: thread 0xfffffc0017499c00 appears to be swapped out

Thread 0xfffffc0017499c00:
More (n if no)?warning: thread 0xfffffc0017499c00 appears to be swapped out
warning: thread 0xfffffc0017499c00 appears to be swapped out
warning: thread 0xfffffc0017499c00 appears to be swapped out
warning: thread 0xfffffc0017499c00 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc0017499c00 appears to be swapped out
warning: thread 0xfffffc0017499c00 appears to be swapped out
   1 msg_dequeue(0xfffffc000046ba08, 0x28, 0x0, 0x3ffc01d0588, 0xffffffffa18678f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0x1ab1330, 0x0, 0x28, 0x7e, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1ab1330, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]
warning: thread 0xfffffc000d327800 appears to be swapped out

Thread 0xfffffc000d327800:
warning: thread 0xfffffc000d327800 appears to be swapped out
warning: thread 0xfffffc000d327800 appears to be swapped out
warning: thread 0xfffffc000d327800 appears to be swapped out
warning: thread 0xfffffc000d327800 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
More (n if no)?More (n if no)?warning: thread 0xfffffc000d327800 appears to be swapped out
   1 msg_dequeue(0xfffffc000046ba08, 0x28, 0x0, 0x0, 0xffffffffa188f8f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0x1ac9330, 0x0, 0x28, 0x81, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1ac9330, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]
warning: thread 0xfffffc0017498c00 appears to be swapped out

Thread 0xfffffc0017498c00:
warning: thread 0xfffffc0017498c00 appears to be swapped out
warning: thread 0xfffffc0017498c00 appears to be swapped out
warning: thread 0xfffffc0017498c00 appears to be swapped out
warning: thread 0xfffffc0017498c00 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc0017498c00 appears to be swapped out
warning: thread 0xfffffc0017498c00 appears to be swapped out
   1 msg_dequeue(0x0, 0x28, 0x0, 0xfffffc000ab340a0, 0xffffffffa189f8f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0x1ae1330, 0x0, 0x28, 0x84, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1ae1330, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]More (n if no)?More (n
if no)?warning: thread 0xfffffc0017498000 appears to be swapped out

Thread 0xfffffc0017498000:
warning: thread 0xfffffc0017498000 appears to be swapped out
warning: thread 0xfffffc0017498000 appears to be swapped out
warning: thread 0xfffffc0017498000 appears to be swapped out
warning: thread 0xfffffc0017498000 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc0017498000 appears to be swapped out
warning: thread 0xfffffc0017498000 appears to be swapped out
   1 mpsleep(0xfffffc0016023a00, 0x11a, 0xfffffc0000625f90, 0x0, 0x0)
["../../../../src/kernel/bsd/kern_synch.c":446, 0xfffffc000042c0c4]
   2 sosleep(0xfffffc0016023a00, 0xffffffffa18af720, 0xffffffff0000011a, 0x0,
0xfffffc000025a85c) ["../../../../src/kernel/bsd/uipc_socket2.c":692, 0xfffffc0000259668]
   3 accept1(0xfffffc000ab34210, 0xffffffffa18af8c8, 0xffffffffa18af8b8, 0x1, 0x0)
["../../../../src/kernel/bsd/uipc_syscalls.c":374, 0xfffffc000025a8d8]
   4 oaccept(0xffffffffa18af8b8, 0x1, 0x0, 0xfffffc000047d19c, 0xfffffc000047c588)
["../../../../src/kernel/bsd/uipc_syscalls.c":319, 0xfffffc000025a7ac]
   5 syscall(0x3, 0x3ffbfcada60, 0x0, 0x41, 0x63)
["../../../../src/kernel/arch/alpha/syscall_trap.c":519, 0xfffffc000047c584]
   6 _Xsyscall(0x8, 0x3ff8051a5c8, 0x3ffffef30d0, 0x17, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]More (n if no)?More (n
if no)?warning: thread 0xfffffc00117f7000 appears to be swapped out

Thread 0xfffffc00117f7000:
warning: thread 0xfffffc00117f7000 appears to be swapped out
warning: thread 0xfffffc00117f7000 appears to be swapped out
warning: thread 0xfffffc00117f7000 appears to be swapped out
warning: thread 0xfffffc00117f7000 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc00117f7000 appears to be swapped out
warning: thread 0xfffffc00117f7000 appears to be swapped out
   1 mpsleep(0xfffffc001ca55600, 0x11a, 0x18, 0x0, 0x0)
["../../../../src/kernel/bsd/kern_synch.c":446, 0xfffffc000042c0c4]
   2 sosleep(0xfffffc001ca55600, 0x1, 0xfffffc000000011a, 0x0, 0xfffffc000ee49d00)
["../../../../src/kernel/bsd/uipc_socket2.c":692, 0xfffffc0000259668]
   3 sosbwait(0xfffffc001ca55660, 0xfffffc001ca55600, 0x0, 0xfffffc001ca55600, 0x18)
["../../../../src/kernel/bsd/uipc_socket2.c":668, 0xfffffc00002595d4]
   4 soreceive(0xfffffc0000436d34, 0x0, 0xffffffffa18b7768, 0xfffffc00002558d0, 0x0)
["../../../../src/kernel/bsd/uipc_socket.c":1390, 0xfffffc00002579a8]
   5 soo_read(0xffffffffa18b7708, 0x18, 0xfffffc001c0b9300, 0xfffffc000ab34688, 0x0)
["../../../../src/kernel/bsd/sys_socket.c":160, 0xfffffc00002525d4]
   6 rwuio(0xfffffc000ab34210, 0xffffffffa18b4000, 0xffffffffa18b78b8, 0xffffffffa18b7768, 0x0)
["../../../../src/kernel/bsd/sys_generic.c":1069, 0xfffffc00002523dc]More (n if no)?More (n if
no)?   7 read(0xfffffc0000000001, 0xffffffffa18b7758, 0x0, 0x100
   8 syscall(0x3, 0x3ffbfcae1f8, 0x6a, 0x41, 0x3)
["../../../../src/kernel/arch/alpha/syscall_trap.c":519, 0xfffffc000047c584]
   9 _Xsyscall(0x8, 0x3ff8051c018, 0x3ffffef30d0, 0x18, 0x1afd738)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]
warning: thread 0xfffffc0017499400 appears to be swapped out

Thread 0xfffffc0017499400:
warning: thread 0xfffffc0017499400 appears to be swapped out
warning: thread 0xfffffc0017499400 appears to be swapped out
warning: thread 0xfffffc0017499400 appears to be swapped out
warning: thread 0xfffffc0017499400 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
warning: thread 0xfffffc0017499400 appears to be swapped out
warning: thread 0xfffffc0017499400 appears to be swapped out
   1 mpsleep(0xfffffc001bb1ee00, 0x11a, 0x3a2, 0x0, 0x0)
["../../../../src/kernel/bsd/kern_synch.c":446, 0xfffffc000042c0c4]
   2 sosleep(0xfffffc001bb1ee00, 0x1, 0xfffffc000000011a, 0x0, 0xfffffc0000460528)
["../../../../src/kernel/bsd/uipc_socket2.c":692, 0xfffffc0000259668]
   3 sosbwait(0xfffffc001bb1ee60, 0xfffffc001bb1ee00, 0x0, 0xfffffc001bb1ee00, 0x1)
["../../../../src/kernel/bsd/uipc_socket2.c":668, 0xfffffc00002595d4]
   4 soreceive(0xffffffffa18bf740, 0x0, 0xffffffffa18bf768, 0x0, 0x0)
["../../../../src/kernel/bsd/uipc_socket.c":1390, 0xfffffc00002579a8]More (n if no)?More (n if
no)?   5 soo_read(0xffffffffa18bf708, 0xe, 0xfffffc001c0b9300, 0xfffffc000ab34688, 0xfffff
   6 rwuio(0xfffffc000ab34210, 0x1, 0xffffffffa18bf8b8, 0xffffffffa18bf768, 0x0)
["../../../../src/kernel/bsd/sys_generic.c":1069, 0xfffffc00002523dc]
   7 read(0xfffffc0000000001, 0xffffffffa18bf758, 0xffffffffa18bf8b8, 0x100000001, 0x0)
["../../../../src/kernel/bsd/sys_generic.c":1021, 0xfffffc00002522e8]
   8 syscall(0x3, 0x3ff8051f174, 0x0, 0x0, 0x3)
["../../../../src/kernel/arch/alpha/syscall_trap.c":519, 0xfffffc000047c584]
   9 _Xsyscall(0x8, 0x3ff8051c018, 0x40000000ae0, 0xe, 0x1b14f08)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]
warning: thread 0xfffffc00117f6400 appears to be swapped out

Thread 0xfffffc00117f6400:
warning: thread 0xfffffc00117f6400 appears to be swapped out
warning: thread 0xfffffc00117f6400 appears to be swapped out
warning: thread 0xfffffc00117f6400 appears to be swapped out
warning: thread 0xfffffc00117f6400 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc00117f6400 appears to be swapped out
warning: thread 0xfffffc00117f6400 appears to be swapped out
   1 msg_dequeue(0xfffffc000046ba08, 0x28, 0x0, 0x18000, 0xffffffffa18c78f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]More (n if no)?More (n if
no)?   2 msg_receive_trap(0x1b2cbf0, 0x0, 0x28, 0x90, 0x0) ["../../../../src/kernel/
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1b2cbf0, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]
warning: thread 0xfffffc00117f7400 appears to be swapped out

Thread 0xfffffc00117f7400:
warning: thread 0xfffffc00117f7400 appears to be swapped out
warning: thread 0xfffffc00117f7400 appears to be swapped out
warning: thread 0xfffffc00117f7400 appears to be swapped out
warning: thread 0xfffffc00117f7400 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc00117f7400 appears to be swapped out
warning: thread 0xfffffc00117f7400 appears to be swapped out
   1 mpsleep(0xfffffc001ca55300, 0x11a, 0xfffffc0000236dd1, 0x0, 0x0)
["../../../../src/kernel/bsd/kern_synch.c":446, 0xfffffc000042c0c4]
   2 sosleep(0xfffffc001ca55300, 0x1, 0xfffffc000000011a, 0x0, 0x0)
["../../../../src/kernel/bsd/uipc_socket2.c":692, 0xfffffc0000259668]
   3 sosbwait(0xfffffc001ca55360, 0xfffffc001ca55300, 0x0, 0xfffffc001ca55300, 0x1)
["../../../../src/kernel/bsd/uipc_socket2.c":668, 0xfffffc00002595d4]
   4 soreceive(0xffffffffa18cf740, 0x0, 0xffffffffa18cf768, 0x0, 0x0)
["../../../../src/kernel/bsd/uipc_socket.c":1390, 0xfffffc00002579a8]
   5 soo_read(0xffffffffa18cf708, 0x19, 0xfffffc001c0b9300, 0xfffffc000ab34688,
0xffffffffa18cf740) ["../../../../src/kernel/bsd/sys_socket.c":160, 0xfffffc00002525d4
More (n if no)?More (n if no)?]
   6 rwuio(0xfffffc000ab34210, 0x0, 0xffffffffa18cf8b8, 0xffffffffa18cf768, 0x0)
["../../../../src/kernel/bsd/sys_generic.c":1069, 0xfffffc00002523dc]
   7 read(0xfffffc0000000001, 0xffffffffa18cf758, 0xffffffffa18cf8b8, 0x100000001, 0x0)
["../../../../src/kernel/bsd/sys_generic.c":1021, 0xfffffc00002522e8]
   8 syscall(0x3, 0x3ff8051f174, 0x0, 0x0, 0x3)
["../../../../src/kernel/arch/alpha/syscall_trap.c":519, 0xfffffc000047c584]
   9 _Xsyscall(0x8, 0x3ff8051c018, 0x40000000ae0, 0x19, 0x1b44f08)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]
warning: thread 0xfffffc001fe7b800 appears to be swapped out

Thread 0xfffffc001fe7b800:
warning: thread 0xfffffc001fe7b800 appears to be swapped out
warning: thread 0xfffffc001fe7b800 appears to be swapped out
warning: thread 0xfffffc001fe7b800 appears to be swapped out
warning: thread 0xfffffc001fe7b800 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc001fe7b800 appears to be swapped out
warning: thread 0xfffffc001fe7b800 appears to be swapped out
   1 msg_dequeue(0x1f7b18, 0x28, 0x0, 0x3ffc01d0588, 0xffffffffa18d78f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0x1b5cbf0, 0x0, 0x28, 0x96, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]More (n if no)?More (n if
no)?   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1b5cbf0, 0x0)
["../../../../src/kernel/arch/alpha
warning: thread 0xfffffc001fe7bc00 appears to be swapped out

Thread 0xfffffc001fe7bc00:
warning: thread 0xfffffc001fe7bc00 appears to be swapped out
warning: thread 0xfffffc001fe7bc00 appears to be swapped out
warning: thread 0xfffffc001fe7bc00 appears to be swapped out
warning: thread 0xfffffc001fe7bc00 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc001fe7bc00 appears to be swapped out
warning: thread 0xfffffc001fe7bc00 appears to be swapped out
   1 msg_dequeue(0xfffffc000046ba08, 0x28, 0x0, 0xfffffc0009a7db80, 0xffffffffa18df8f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0x1b741c0, 0x0, 0x28, 0x99, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1b741c0, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":1249, 0xfffffc000046bd18]
warning: thread 0xfffffc001215c800 appears to be swapped out

Thread 0xfffffc001215c800:
warning: thread 0xfffffc001215c800 appears to be swapped out
More (n if no)?More (n if no)?warning: thread 0xfffffc001215c800 appears to be swapped out
warning: thread 0xfffffc001215c800 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc001215c800 appears to be swapped out
warning: thread 0xfffffc001215c800 appears to be swapped out
   1 mpsleep(0xfffffc0008d0a300, 0x11a, 0x38, 0x0, 0x0)
["../../../../src/kernel/bsd/kern_synch.c":446, 0xfffffc000042c0c4]
   2 sosleep(0xfffffc0008d0a300, 0x1, 0xfffffc000000011a, 0x0, 0x0)
["../../../../src/kernel/bsd/uipc_socket2.c":692, 0xfffffc0000259668]
   3 sosbwait(0xfffffc0008d0a360, 0xfffffc0008d0a300, 0x0, 0xfffffc0008d0a300, 0x0)
["../../../../src/kernel/bsd/uipc_socket2.c":668, 0xfffffc00002595d4]
   4 soreceive(0xffffffffa18e7740, 0x0, 0xffffffffa18e7768, 0x3fffff2a8a0, 0x0)
["../../../../src/kernel/bsd/uipc_socket.c":1390, 0xfffffc00002579a8]
   5 soo_read(0xffffffffa18e7708, 0x1b, 0xfffffc001c0b9300, 0xfffffc000ab34688,
0xffffffffa18e7740) ["../../../../src/kernel/bsd/sys_socket.c":160, 0xfffffc00002525d4]
   6 rwuio(0xfffffc000ab34210, 0x0, 0xffffffffa18e78b8, 0xffffffffa18e7768, 0x0)
["../../../../src/kernel/bsd/sys_generic.c":1069, 0xfffffc00002523dc]
   7 read(0x8, 0xffffffffa18e7758, 0x0, 0x800000001, 0x0)
["../../../../src/kernel/bsd/sys_generic.c":1021, 0xfffffc00002522e8]
   8 syscall(0x1b87ea0, 0x1, 0x1, 0x21, 0x3)
["../../../../src/kernel/arch/alpha/syscall_trap.c":519, 0xfffffc000047c584]
   9 _Xsyscall(0x8, 0x3ff8051c018, 0x40000000ae0, 0x1b, 0x1b8c8b8)
["../../../../src/kernel/arch/alpha/locore.s":1094, 0xfffffc000046bbc4]More (n if no)?More (n
if no)?
Thread 0xfffffc000be68c00:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1cd5350, 0x100)
["../../../../src/kernel/arch/alpha/locore.s":1205, 0xfffffc000046bcb8]
warning: thread 0xfffffc001fe15800 appears to be swapped out

Thread 0xfffffc001fe15800:
warning: thread 0xfffffc001fe15800 appears to be swapped out
warning: thread 0xfffffc001fe15800 appears to be swapped out
warning: thread 0xfffffc001fe15800 appears to be swapped out
warning: thread 0xfffffc001fe15800 appears to be swapped out
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
warning: thread 0xfffffc001fe15800 appears to be swapped out
warning: thread 0xfffffc001fe15800 appears to be swapped out
   1 msg_dequeue(0xfffffc000046ba08, 0x28, 0x0, 0xfffffc0009a7db80, 0xffffffffa193f8f8)
["../../../../src/kernel/kern/ipc_basics.c":865, 0xfffffc00002cc448]
   2 msg_receive_trap(0x1d2b990, 0x0, 0x28, 0xa2, 0x0)
["../../../../src/kernel/kern/ipc_basics.c":1230, 0xfffffc00002ccc3c]More (n if no)?More (n if
no)?   3 _Xsyscall(0x8, 0x3ff8052e574, 0x3ffc01cb690, 0x1d2b990, 0x0)
["../../../../src/kernel/arch/alpha

Thread 0xfffffc001fe14400:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931, 0xfffffc0000461594]
   1 thread_dowait(0xfffffc0001812400, 0x1, 0xfffffc001ca34600, 0xfffffc0000000000,
0xfffffc00002db2a8) ["../../../../src/kernel/kern/thread.c":2100, 0xfffffc00002ddcac]
   2 task_dowait(task = 0xfffffc000ab34030, must_wait = 179519488)
["../../../../src/kernel/kern/task.c":809, 0xfffffc00002db438]
   3 task_suspend_self(must_wait = 1) ["../../../../src/kernel/kern/task.c":1281,
0xfffffc00002dbe00]
   4 procfs_trace_signals(0xfffffc000ab34210, 0xffffffffa1977728, 0xffffffffa1977750,
0xffffffffa1977710, 0x4283d4) ["../../../../src/kernel/procfs/procfs_subrs.c":473,
0xfffffc00002843d0]
   5 psig(sig = 5) ["../../../../src/kernel/bsd/kern_sig.c":3846, 0xfffffc0000428e94]
   6 trap() ["../../../../src/kernel/arch/alpha/trap.c":1386, 0xfffffc000047dad4]
   7 _XentIF(0x8, 0x3ff80569ce4, 0x3ffffef30d0, 0x1, 0x141d79240)
["../../../../src/kernel/arch/alpha/locore.s":688, 0xfffffc000046b890]

Thread 0xfffffc001fe14c00:
More (n if no)?>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1931,
0xfffffc0000461594]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x3ff80567b00, 0x3ffc01e3500, 0xffffffffffffff35, 0xffffffffffffff35)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]

Thread 0xfffffc001fe15400:
>  0 thread_block() ["../../../../src/kernel/kern/sched_prim.c":1919, 0xfffffc0000461524]
   1 issig() ["../../../../src/kernel/bsd/kern_sig.c":3262, 0xfffffc0000427ad4]
   2 trap() ["../../../../src/kernel/arch/alpha/trap.c":1385, 0xfffffc000047dac0]
   3 exception_exit(0x8, 0x12004d05c, 0x14003ae70, 0x1d91b78, 0x0)
["../../../../src/kernel/arch/alpha/locore.s":839, 0xfffffc000046b9a8]
(dbx) 

    Well, this U hang is related to procfs trying to halt the
    task on a signal. All the other threads in issig() are
    waiting for that signal trace to complete. So, one other
    thread in the task is not cooperating, casuing the hang.
    
    But this may not be related to the original problem, which
    I'm assuming happens without the debugger involved, right?
    You should at least be sure that you've got all the latest
    procfs patches. There were some hang fixes over that
    last few months, and more coming for the 3.2 space.
    
    Procfs issues aside, not sure we can tell much more from these
    traces. If you a ps mp STUCK-PID, can you find *two* threads
    in the U state? One is the thread taking the signal -- it would
    help to know if there is another U thread and what it's doing.

.0> By experience, we have seen this kind of behaviour being caused by thread 
.0> stack overflow

It's been a long time since I've worked on a V3.2 system, but my recollection is
that a stack overflow resulted in a quick termination, with a message like
"sendsig: cannot grow stack".

.0> we would have prefered a more explicit behaviour, such a somehow signalling 
.0> "Thread Stack Overflow" somewhere, but ...

Jeff put something like that into V4.0...I hope you like it!  ;-)

.0> we achieved to go a bit further by increasing the size of some of the 
.0> application threads (namely, those related to DECss7) from 32kb to more than 
.0> 200kb, but the same misbehaviour still occurs at a different point.

Probably what happened is that by increasing the size of your stacks you changed
the location where something was happening, so that it then happened elsewhere
or at a different time.  (That is, increasing the size of the stacks didn't fix
anything, it just moved it.)

.0> Did you already hear of such kind of troubles with thread stack size ? 

I am not aware of any such troubles associated with lack of stack space. 
However, if a program makes very large stack allocations (i.e., uses frames
(e.g., local variable storage) larger than 8192 bytes), then it's possible for
the program to "stride" across the guard page at the end of one thread's stack
and corrupt the adjactent memory (e.g., another thread's stack, or possibly heap
storage) without being detected.  Once you've corrupted memory, nearly anything
could happen.

.0> Is there a mean by which we can configure either the OS or the DECthreads 
.0> library to automatically allocate a stack with a user-defined size (much 
.0> greater than 32kb) ?

There is no way to change the default thread stack size.  If you want a larger
stack, you have to request the size that you want at each creation.

.0> What do you think would be a more reasonable default stack size value ?

The "reasonable" size is entirely dependent upon what the thread is doing.  It
would be wonderful if each API provider published the amount of stack space that
it's routines required -- then the developer could add it all up and be sure of
creating the thread with the right size stack.  However, that's hard to do, and
it changes over time...  It appears to be a trial and error process.  Hopefully,
we'll be able to provide the grand and final solution to this problem in the
"Steel" release...stay tuned.  :-)

In the meantime, increasing the size of the stack guard area.  This will help
detect stack overflow corruptions (by terminating the process with a SEGV), and
it might show you the problem.


				Webb

> The "reasonable" size is entirely dependent upon what the thread is doing.  It
> would be wonderful if each API provider published the amount of stack space that
> it's routines required -- then the developer could add it all up and be sure of
> creating the thread with the right size stack.

	Even more wonderful would be if the system (linker, runtime loader,
	and threads) could do this all automatically.

	For example, the linker (or whoever does global optimizations)
	could in its flow analysis could also look at the stack space
	requirements for each routine and determine a reasonable worse
	case calling path (obviously if there is alot of recursion
	to unknown depths then #pragma's and/or linker options can
	guide the analysis) and store that value somewhere in the
	header of a shared binary and shared objects.

	Then the runtime loader could automatically "add it all up"
	as it loads the binary and required shared objects (another
	obvious is that threaded programs that load libraries on
	the fly will still have to fend for themselves).  when the
	application then creates a new thread "with the default stack
	size", the threads library can get the info from some place
	the runtime loader stored it and use that as the default.

	Though even easier and just, if not more wonderful would be
	growable (and possibly shrinkable) stacks, which someone
	(probably the other, ie. base system, Jeff) said was being
	considered for a future release of Digital UNIX?

In Steel, we're getting kernel support for thread stacks that will behave
like the default process stack. We've been allocating relatively small stacks
for threads because whatever we allocate has been "committed" in the memory
system, just like any normal malloc() call -- and, on an eager swap system,
is immediately reserved in the swapfile. Running around with a lot of threads
having large stacks is EXPENSIVE, and we simply can't justify a larger stack
default.

In Steel, we expect to be able to ALLOCATE a large stack (1Mb or even 10Mb)
for each thread, but COMMIT only a few pages of it. The swapfile space will
thus grow only as the program USES stack. You'll almost never need to set the
stacksize attribute unless you really want a smaller stack than the default.
(One unfortunate consequence is that all the existing code that had trouble
with the 23Kb default and set "large" stacksizes will suddenly be actually
setting SMALL stacksizes. I may be able to avoid that, for example, by
"retiring" the existing stacksize setting routine, and changing it to ignore
any setting over 23Kb; and moving the <pthread.h> prototype for stacksize to
a new routine name, for newly linked code, that really does what you ask...
but I need to think through all the possible consequences.)

	/dave

    I guess you've got source access. so take a look if you're using
    the alloca family somehere, which would be a candidate for using large 
    stack.
    on the other hand, to obtain the numbers of stack use isn't that hard
    with atom. What I did (it was for a process, and not yet threadsafe)
    was go down the atom hierachy (object, function, block, instruction).
    as there is no special command for stack, I searched for command
    lda using $sp for ra and rb. in case ofthis, i passed the offset to
    the analysis routine. this dumps out your stack requirements and the
    cleanups, which could be an aid to resolve youre culprit.
    
    Pit

Thank you very much to all of you, guys ...

I will get my customer informed of your answers (especially the one
related to `procfs' patches, since it made me remember that they have
other pbs related to this ;-)

| .0> By experience, we have seen this kind of behaviour being caused by thread 
| .0> stack overflow
| 
| It's been a long time since I've worked on a V3.2 system, but my recollection is
| that a stack overflow resulted in a quick termination, with a message like
| "sendsig: cannot grow stack".

Webb, you surely do not have the possibility to test our DECss7 platform 
during that time ;-) Just remember that we are using all together DECthreads,
C++ and Ada RTLs !!!

That said, I think that "dynamically growing thread stacks" is the only
reasonable solution to this issue. So, please let me know as soon as
your Steel has been quenched ;-)

Regards,

Xav

> In the meantime, increasing the size of the stack guard area.  This will help
> detect stack overflow corruptions (by terminating the process with a SEGV), and
> it might show you the problem.

My customer asked me for the rationale behing this action. I suspect that it
implies something like, if the Stack Guard Area Size is strictly greater than the
actual stack size, we ensure the presence of "holes" between each thread's stack,
which in turn would lead to SEGV in case of stack overflow. Is this correct ?

Thx,

Xav

    The customer informed me that after increasing the stack size the
    process still consumes much too much system resources.
    Also a problem call was opened thru the MCS hotline (Nice #34691).
    As the customer is in a big time pressure could we speed up here and
    directly setup a conversation between Unix engineering and the customer
    ?? 
    
    Tks & Rgds
    Daniel.

.8> My customer asked me for the rationale behing this action.

I believe I answered that in .3:

.3> if a program makes very large stack allocations (i.e., uses frames (e.g.,
.3> local variable storage) larger than 8192 bytes), then it's possible for
.3> the program to "stride" across the guard page at the end of one thread's
.3> stack and corrupt the adjactent memory (e.g., another thread's stack, or
.3> possibly heap storage) without being detected.

Thus, the guard size should be set to a value at least as large as the
largest demand that any single routine makes for stack space.  Since the
customer is having troubles, it would be a good idea to double or quaddruple
the guard size until the problem is located.  (Even if the problem turns out
not to be a stack overflow, increasing the guard size will increase the
likelihood that the corruption hits a guard page, and the resulting SEGV will
make clear the source.)


				Webb

|.8> My customer asked me for the rationale behing this action.
| 
|I believe I answered that in .3:
| 
|.3> if a program makes very large stack allocations (i.e., uses frames (e.g.,
|.3> local variable storage) larger than 8192 bytes), then it's possible for
|.3> the program to "stride" across the guard page at the end of one thread's
|.3> stack and corrupt the adjactent memory (e.g., another thread's stack, or
|.3> possibly heap storage) without being detected.
|
|Thus, the guard size should be set to a value at least as large as the
|largest demand that any single routine makes for stack space.  Since the
|customer is having troubles, it would be a good idea to double or quaddruple
|the guard size until the problem is located.  (Even if the problem turns out
|not to be a stack overflow, increasing the guard size will increase the
|likelihood that the corruption hits a guard page, and the resulting SEGV will
|make clear the source.)

Well, we were asking for the "rationale" as an explanation of the underlying
reasons for which we should expect a corruption to provoke a SEGV. I mean, 
I think I well understand the basics, but I would also appreciate to explain
the actual technical reason (that's why I was imagining such "memory holes"
in process virtual address space).

In practice, as it is quite impossible to know what is the "largest demand
that any single routine makes for stack space", I was wondering what could
be a good attempt. For instance, if I propose to the customer to increase
stack size to 1 MB and guard size to 2MB, does it make any sense ?

Thanks,

Xav

.11> we were asking for [...] an explanation of the underlying reasons for which 
.11> we should expect a corruption to provoke a SEGV. 

Ah, I see (I think) -- the use of the word "rationale" confused me.  The "guard
area" is one or more pages of memory whose page protection denies any access;
thus, any attempt to read or write from these pages will produce a SEGV. 
Because no thread should ever be attempting to access these areas (i.e., they
are outside the stack boundaries in otherwise unreferenced memory), the SEGV
allows immediate detection of problems.

.11> In practice, as it is quite impossible to know what is the "largest demand
.11> that any single routine makes for stack space", I was wondering what could
.11> be a good attempt. 

It's far from impossible, but I will grant that it's inconveniently difficult.

.11> For instance, if I propose to the customer to increase stack size to 1 MB 
.11> and guard size to 2MB, does it make any sense ?

I can't think of a good reason for the guard size to be larger than the stack
size (unless you're desperate and you have address space to burn).  For guard
size, I was thinking in the 16Kb-32Kb range, but if you can afford larger that's
great (for debugging purposes).  As for stack size, what you want depends on
what the application does (and how much memory (e.g., swap space) is available).

I'm surprised that you cannot run this under the debugger and make some basic
determinations about how much stack space each thread actually requires.  (For
example, you could take the "largest seen so far" and add a cushion to that and
use the resulting value.)


				Webb

Following is an atom instrumentation file to track memory references.
    you can delete a couple of stuff, as you are only interested in 
    the stack values. of course, you have to perform a quick check
    if threads use lda with REG_SP too. 
    you should than be able to printout the function, line, stack required/
    released in a simple analysis routine. 
    I just grabbed this, and know that it is by now mean complete, but it
    should at least help you to obtain the data what you're looking for.
    oh, uncomment the switch (IsTragetProc) block, as this is used for 
    diffrent purpose
    
    Pit
    
    
/*
 *  Instrumentation file for a memory tracer
 *
 *  Goal is to keep track of memory references that we obtain
 *  a map of memory usage afterwards.
 *  
 *  
 *  general flow
 *       foreach obj
 *		collect/print TextStartAddress/Size
 *			      InitDataStartAddress/Size
 *			      UninitDataStartAddress/Size
 *			      Number of Procs/Blocks/Insts
 *			      
 *	         proc
 *		collect	      ProcFrameSize
 *			      I/F Regmask && RegOffset
 *			      GP if used && size of prologue
 *			      LocalOffset to procs local vars
 *			      Frame/Pcreg
 *			      Number of Blocks/Insts
 *			      IsRegFrame
 *
 *			      addr && size of calls to alloc
 *			      mmap and shm family
 *
 *		Block/Inst
 *			      Load/Stores && Read/Writes are
 *			      passed to analysis routine that
 *			      we can determine the actual memory
 *			      references being used.
 *			      
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <cmplrs/atom.inst.h>
#include <alpha/inst.h>
#include "st.h"

#define INST_DEBUG
/*
 *  forward declaration
 */
static const char *SafeProcName(Proc *);
int isForkProc(Proc *);
void AddCallProtos();
int IsInstSafe(Proc *);
const char * Undemangle(const char *);
static char theprogname[128];

/*
 * AddCallProtos
 */
void AddCallProtos() 
{
   AddCallProto("OpenFile(char *, int)");
   AddCallProto("CloseFile()");
   AddCallProto("ObjInfo(long, long, long, long, long, long)");
   AddCallProto("BeforeFork()");
   AddCallProto("AfterFork()");
   AddCallProto("Before_malloc(REGV)");
   AddCallProto("After_malloc(REGV)");
   AddCallProto("Before_calloc(REGV,REGV)");
   AddCallProto("After_calloc(REGV)");
   AddCallProto("Before_realloc(REGV,REGV)");
   AddCallProto("After_realloc(REGV)");
   AddCallProto("Before_free(REGV)");
   AddCallProto("Before_sbrk(REGV)");
   AddCallProto("After_sbrk(REGV)");
   AddCallProto("Before_brk(REGV)");
   AddCallProto("Before_mmap(REGV, REGV, int, int, int, REGV)");
   AddCallProto("After_mmap(REGV)");
   AddCallProto("LoadRef(VALUE, long, int)");
   AddCallProto("LoadAddRef(REGV, int, long, int)");
   AddCallProto("StoreRef(VALUE, long, int)");
   AddCallProto("StartObj(int)");
   AddCallProto("CloseObj(int)");
}


unsigned
InstrumentAll(int argc, char **argv)
{
   Obj *obj;
   Proc *p;
   Block *b;
   Inst *inst;
   const char *ofn;		/* Object file name */
   long addr;
   unsigned long textstart, textsize, datastart, datasize, bssstart, bsssize;
   int rega, regb, regc, disp, opcode, func;
   int p_ind;
   InstRegUsageVec RegVec;
   union alpha_instruction ai;
   ProcRes  target;
   int procnum;
   int in_sh_lib = 0;
   int lookup_shlib(const char *);

   AddCallProtos();

   for(obj = GetFirstObj(); obj != NULL; obj = GetNextObj(obj)){
       if(BuildObj(obj))
		return(1);
       in_sh_lib = 0;
       ofn = GetObjName(obj);

#ifdef INST_DEBUG
       printf("Object <%s> <%s>\n",GetObjName(obj), ofn);
#endif

	if(lookup_shlib(ofn))
       		in_sh_lib = 1;

	if(!in_sh_lib){
		textstart  = GetObjInfo(obj, ObjTextStartAddress);
		textsize = GetObjInfo(obj, ObjTextSize);
		datastart = GetObjInfo(obj, ObjInitDataStartAddress);
		datasize = GetObjInfo(obj, ObjInitDataSize);
		bssstart = GetObjInfo(obj, ObjUninitDataStartAddress);
		bsssize = GetObjInfo(obj, ObjUninitDataSize);
	}

       for(procnum = 0,p = GetFirstObjProc(obj); p != NULL; p = GetNextProc(p)){
    	    if (!strcmp(SafeProcName(p),"main")) {
		/* following is ugly, but I haven't discovered my error 
		 * yet. __start contains 3 jsr's. first two are local
		 * (nil) when resolved by ResolveTargetProc. Third is resolved 
		 * as exit. stepping manually through start, shows that 
		 * register t12 in
		 * jsr     ra, (t12), 0x12000129c(zero)
		 * contains 0x12000149c (the startaddress of main in my testproc)
		 */
   		b=GetFirstBlock(p); 
       		inst=GetFirstInst(b);
		strcpy(theprogname,GetObjName(obj));
	    }
	    if (isForkProc(p)){
		    AddCallProc(p, ProcBefore, "BeforeFork");
		    AddCallProc(p, ProcAfter, "AfterFork");
		    continue;
	    }


	    if(!in_sh_lib){

#ifdef INST_DEBUG
	    printf("\n FUNC: %s\n", SafeProcName(p));
#endif

/**********
	    	if(IsInstSafe(p))
		    continue;
**********/


   for (b=GetFirstBlock(p); b; b=GetNextBlock(b)) 
       for (inst=GetFirstInst(b); inst; inst=GetNextInst(inst)) {

           /* get the opcode, followed by getting the displacement
            * disp<15:14> differs in the meaning as
            *              Meaning      Predict. Target            Stack action
            *       0 0    JMP           PC + { 4 * disp<13:0> }       ---
            *       0 1    JSR           PC + { 4 * disp<13:0> }     Push PC
            *       1 0    RET           prediction stack            Pop
            *       1 1    JSR_COROUT.   prediction stack            Pop, Push
PC
            * see Alpha ref manual for further explanation
            */
		opcode = GetInstInfo(inst,InstOpcode);
		disp = GetInstInfo(inst,InstMemDisp);
		func = (disp >> 14) & 0x3;

		ai.word = GetInstInfo(inst, InstBinary);
/**
printf("disp = %d ra = %d rb = %d op = %x\n", ai.m_format.memory_displacement,
ai.m_format.ra, ai.m_format.rb, ai.m_format.opcode);
**/

		if ((opcode == op_jsr && func == 1 ) || opcode== op_bsr) {
                	ResolveTargetProc(inst, &target);
#ifdef INST_DEBUG
                printf("name <%s> ", target.name);
                printf("obj 0x%p ", target.obj);
                printf("proc 0x%p ", target.proc);
                printf("inst 0x%p\n", target.inst);
#endif

                if(target.name && target.proc == NULL) { /* lib function */
#ifdef INST_DEBUG
		printf("entering switch %s\n", target.name);
#endif
                	switch(p_ind = IsTargetProc(target.name)){
				case TYP_MALLOC:
				printf("switch malloc\n");
	    		    AddCallInst(inst, InstBefore, "Before_malloc",REG_16);
			    AddCallInst(inst, InstAfter, "After_malloc",REG_0);
					break;
				case TYP_CALLOC:
			    AddCallInst(inst, InstBefore, "Before_calloc",REG_16,REG_17);
			    AddCallInst(inst, InstAfter, "After_calloc",REG_0);
					break;
				case TYP_REALLOC:
			    AddCallInst(inst, InstBefore, "Before_realloc",REG_16,REG_17);
			    AddCallInst(inst, InstAfter, "After_realloc",REG_0);
					break;
				case TYP_FREE:
			    AddCallInst(inst, InstBefore, "Before_free",REG_16);
					break;
				case TYP_SBRK:
			    AddCallInst(inst, InstBefore, "Before_sbrk",REG_16);
			    AddCallInst(inst, InstAfter, "After_sbrk", REG_0);
					break;
				case TYP_BRK:
			    AddCallInst(inst, InstBefore, "Before_brk",REG_16);
					break;
				case TYP_MMAP:
			    AddCallInst(inst, InstBefore, "Before_mmap",
				REG_16, REG_17, REG_18, REG_19, REG_20, REG_21);
			     /*  addr   len     prot    flag     fd     off */
			    AddCallInst(inst, InstAfter, "After_mmap", REG_0);
					break;
				default: /* no default */
					break;
                }
	    }
    	}

                /* check for stack adjustment */
		if(opcode == op_lda && ai.m_format.ra == REG_SP
		   && ai.m_format.rb == REG_SP)
			AddCallInst(inst,InstBefore,"LoadAddRef",
				REG_SP, ai.m_format.memory_displacement,
				InstPC(inst), procnum);

		if(IsInstType(inst,InstTypeLoad)) {
			AddCallInst(inst,InstBefore,"LoadRef",
				EffAddrValue, InstPC(inst), procnum);
		}
		else if(IsInstType(inst,InstTypeStore)) {
			AddCallInst(inst,InstBefore,"StoreRef",EffAddrValue,
				 InstPC(inst), procnum);
		}
	    } /* for GetFirstInst */
	  } /* if in_sh_lib */
	  procnum++;
        } /* for GetFirstObjProc */

	WriteObj(obj);

   	AddCallObj(ObjBefore,"StartObj", procnum);
   	AddCallObj(ObjAfter,"CloseObj", procnum);
   } /* GetFirstObj  */

   AddCallProgram(ProgramBefore,"OpenFile", theprogname, 1);
   AddCallProgram(ProgramAfter,"CloseFile");
}

int IsTargetProc(char *pname)
{
	int res = 0;

	printf("enter with name %s\n", pname);

	if(!strcmp("malloc", pname))
		res = TYP_MALLOC;
	else if(!strcmp("calloc", pname))
		res = TYP_CALLOC;
	else if(!strcmp("realloc", pname))
		res = TYP_REALLOC;
	else if(!strcmp("free", pname))
		res = TYP_FREE;
	else if(!strcmp("__sbrk", pname))
		res = TYP_SBRK;
	else if(!strcmp("__brk", pname))
		res = TYP_BRK;
	else if(!strcmp("mmap", pname))
		res = TYP_MMAP;
	printf("return %d\n", res);
	return(res);
}

/*
 * lookup_shlib
 * determines if shared libraries are fully traced too
 * currently only libc.so is compared. 
 */
int lookup_shlib(const char * shname)
{
   if (!strcmp(shname, "/usr/shlib/libc.so")) 
	return(1);

   return(0);
}

/*
 * isMemProc 
 * compares if the fork procedure is used as current function.
 * if so, we have to create a new file for the new process
 * in the anal routine
 */
int isMemProc(Proc* p) {
	/* alloc family */
    if (!strcmp(SafeProcName(p),"malloc"))  
	return TYP_MALLOC;
    else if(!strcmp(SafeProcName(p),"_malloc"))
	return TYP_MALLOC;
    else if(!strcmp(SafeProcName(p),"calloc"))
	return TYP_CALLOC;
    else if(!strcmp(SafeProcName(p),"realloc"))
	return TYP_REALLOC;
    else if(!strcmp(SafeProcName(p),"free"))
	return TYP_FREE;
    else if(!strcmp(SafeProcName(p),"__sbrk"))
	return TYP_SBRK;
    else if(!strcmp(SafeProcName(p),"__brk"))
	return TYP_BRK;
    else if(!strcmp(SafeProcName(p),"mmap"))
	return TYP_MMAP;
     else 
        return 0;
    
}

/*
 * isForkProc 
 * compares if the fork procedure is used as current function.
 * if so, we have to create a new file for the new process
 * in the anal routine
 */
int isForkProc(Proc* p) {
    if (!strcmp(SafeProcName(p),"__fork") || !strcmp(SafeProcName(p),"__vfork")  )  
	return 1;
     else 
        return 0;
    
}


/*
 * see apropos demangle. c compiler demangles the function while
 * adding underscores. 
 */
const char * Undemangle(const char * str)
{
	const char *p;

	p = str;
	while(*p == '_')
		p++;	

	return(p);
}
		
/*
 * load_locked / store cond shouldn't be instrumented as branches
 * out of such a pair tend to be disasterous. 
 */
int IsInstSafe(Proc * p)
{
   Block *b;
   Inst *i;
   int op;

   for (b=GetFirstBlock(p); b; b=GetNextBlock(b)) {
       for (i=GetFirstInst(b); i; i=GetNextInst(i)) {
           op = GetInstInfo(i, InstOpcode);
           if (op==op_ldl_l || op==op_ldq_l) {
    		printf("found ldx_l in %s\n",SafeProcName(p));  
                return 1;
            }
        }
    }
    return 0;
}

	
/*
 * some strippe are stripped and can't return a name. In this case
 * we issue the trem proc_at_0x<addr> instead
 */
static const char *SafeProcName(Proc *p)
{
    const char *	name;
    static char 	buf[128];

    name = ProcName(p);
    if (name) 
	return(name);
    sprintf(buf, "_proc_at_0x%lx", ProcPC(p));
    return(buf);
}

|.11> we were asking for [...] an explanation of the underlying reasons for which 
|.11> we should expect a corruption to provoke a SEGV. 
|
|Ah, I see (I think) -- the use of the word "rationale" confused me.  The "guard
|area" is one or more pages of memory whose page protection denies any access;
|thus, any attempt to read or write from these pages will produce a SEGV. 
|Because no thread should ever be attempting to access these areas (i.e., they
|are outside the stack boundaries in otherwise unreferenced memory), the SEGV
|allows immediate detection of problems.

Well, I do appreciate your explanation and guess that will help us in using this
guard size in a bit more "reasonable" fashion ;-)

|I'm surprised that you cannot run this under the debugger and make some basic
|determinations about how much stack space each thread actually requires.  (For
|example, you could take the "largest seen so far" and add a cushion to that and
|use the resulting value.)

>Following is an atom instrumentation file to track memory references.
>    you can delete a couple of stuff, as you are only interested in 
>    the stack values. of course, you have to perform a quick check
>    if threads use lda with REG_SP too. 
>    you should than be able to printout the function, line, stack required/
>    released in a simple analysis routine. 
>    I just grabbed this, and know that it is by now mean complete, but it
>    should at least help you to obtain the data what you're looking for.
>    oh, uncomment the switch (IsTragetProc) block, as this is used for 
>    diffrent purpose
    
We'll try to investigate in the directions you gave us.

Thank you very much for your help,

Xav

    Re 9248.13 by COL01::LINNARTZ
    
    Where can I obtain the file st.h included by the memory tracer atom
    instrumentation program?
    
Peter.

/usr/include/st.h is a new file in V4.0.   

I don't see anything in that atom tool that requires the st.h stuff.  Did
you try removing it?

    Removing the call to st.h gives the following types of error. All the
    references to TYP_???? symbols and the two calls to AddCallObj have
    errors. I am using V3.2G. 
    
Peter.
    
    
    
    > cc atom_mem.c
    /usr/lib/cmplrs/cc/cfe: Error: atom_mem.c, line 196: 'TYP_MALLOC'
    undefined, reoccurrences will not be reported
                                    case TYP_MALLOC:
                                    -----^
    /usr/lib/cmplrs/cc/cfe: Error: atom_mem.c, line 201: 'TYP_CALLOC'
    undefined, reoccurrences will not be reported
                                    case TYP_CALLOC:
                                    -----^
    
    
    and
    
    /usr/lib/cmplrs/cc/cfe: Warning: atom_mem.c, line 253: illegal
    combination of pointer and integer
            AddCallObj(ObjBefore,"StartObj", procnum);
     ---    -----------^
    /usr/lib/cmplrs/cc/cfe: Warning: atom_mem.c, line 253: illegal
    combination of pointer and integer
            AddCallObj(ObjBefore,"StartObj", procnum);
     ---    ---------------------^
    /usr/lib/cmplrs/cc/cfe: Warning: atom_mem.c, line 253: illegal
    combination of pointer and integer
            AddCallObj(ObjBefore,"StartObj", procnum);
     ---    ---------------------------------^
    
      

TYP_MALLOC does not show up in any header file on my system.   You've got me...

    I just posted it all as I didn't want to check it out. I mentioned to
    comment the TargetProg function. currently st.h just contains 
    #define TYP_MALLOC      1
    #define TYP_CALLOC      2
    #define TYP_REALLOC     3
    #define TYP_FREE        4
    #define TYP_SBRK        5
    #define TYP_BRK         6
    #define TYP_MMAP        7
    
    and the templates from the analysis routine, which needs some
    additional work. sorry that I've choosen an existing name by accident,
    but I only wanted to show one way to track stack operation.
    
    To get rid of the warning's, take a look at the atom reference manual
    or "caldec::wrl_atom".
    
    Pit