[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference clt::cma

Title:DECthreads Conference
Moderator:PTHRED::MARYSTEON
Created:Mon May 14 1990
Last Modified:Fri Jun 06 1997
Last Successful Update:Fri Jun 06 1997
Number of topics:1553
Total number of notes:9541

1502.0. "condition-variable morphing problem on V4.0B ???" by TAEC::GARNERO () Tue Mar 11 1997 08:35


Hello,

I have got a problem which seems (to me) similar to the problem
known as IPMT Case MGO102540 (CFS.46628) [condition-variable morphing],
where a thread was blocking on a pthread_mutex_unlock call. 
This problem has been fixed on V3.2x.

When the problem occurs, the idle time remaining on each CPU is 0.

As soon as the process (where the problem occurs, ss7_fep_ccitt_c.exe) 
is killed (kill -9), the system crashes.

This problem occurs only on multicpus machines.

Could someone tell me if a patched pthread library (on V4.0B) fixing
this problem already exists ?

Configuration:
  Digital Unix V4.0B 
    . with patch 24.00 OSF410-400131 (kit-0002).
    . + unofficial patch fixing sigwait problem (EINTR returned even if all 
        signals masked)
  Alpha 4100 with 4 CPUs
  pthread interface used: POSIX 1003.1c

Many thanks in advance,
Pierre

****************************************************************

# dbx /usr/var/ss7/platform_c/bin/ss7_fep_ccitt_c.exe
dbx version 3.11.10
Type 'help' for help.

main:    Source not available
(dbx) r
%SS7-I-RULE_LOADING, Start loading MTP3 rules file - V3.0
%SS7-I-FILEOPEN, File opened, start loading
%SS7-I-RULE_END, End of loading phase
DECthreads bugcheck (version V3.13-435), terminating execution.
vpUpcallThreadUnblocked: (os/kern) invalid argument (4) nxm_resched(49,0)



****** impossible to do anything else in this session ******

Then, under another debug session:

# dbx -pid 928 /usr/var/ss7/platform_c/bin/ss7_fep_ccitt_c.exe
dbx version 3.11.10
Type 'help' for help.

thread 0x19 stopped at>*[swtch_pri, 0x3ff8053eb1c]      ret     r31, (r26), 1
(dbx) where
>  0 swtch_pri(0x3ffc0080ba0, 0x3ff00000000, 0x3ff805605e0, 0x3ff8055d558,
0x3ff8057c4b4) [0x3ff8053eb1c]
   1 pthread_mutex_unblock(0x140e77160, 0x140155cd0, 0x140129630, 0x9, 0x0)
[0x3ff80568ee4]
   2 __pthread_mutex_unlock(0x140155cd0, 0x140129630, 0x9, 0x0, 0x3ff00000000)
[0x3ff8057c588]
   3 (unknown)() [0x3ff80575ab4]
   4 free(0x3ff800d1fc8, 0x0, 0x1205f5e7c, 0x140de8950, 0x140f1cab8)
[0x3ff800d1fc4]
   5 FI_q_remove_from_head(0x0, 0x1402329e0, 0x0, 0x140867298, 0x0) [0x1205f5e78]
   6 (unknown)() [0x1200aff7c]
   7 ADA$MTP2_BOARD$000C(0x140129630, 0x1401231f0, 0x140071020, 0x1400739e0,
0x1203f4178) [0x1203e6ad4]
   8 (unknown)() [0x1203f3de8]
   9 ADA$MTP2_BOARD$000L(0x1402329e0, 0x3ffffffab98, 0x0, 0x1402329e0,
0x1203e5a40) [0x1203ebc24]
  10 end_of_qio_read(0x0, 0x1402329e0, 0x1203e5a40, 0x140155980, 0x3ffbffd1150)
[0x1203e5a3c]
  11 ast$deliver(0x0, 0x0, 0x0, 0x0, 0x3ffbffc604c) [0x3ffbffd114c]
  12 FV_read_service__13TC_io_channelXv(0x3ffbffc6074, 0x1408608e0, 0x0,
0x1408608e0, 0x1) [0x3ffbffc6904]
  13 (unknown)() [0x3ffbffc6070]
  14 thdBase(0x0, 0x0, 0x0, 0x1, 0x45586732) [0x3ff80574714]

(dbx) tstack
*********** nothing returned *********






Crash dump analysis (crash occurs when the process is killed):


#
# Crash Data Collection (Version 1.4)
#
_crash_data_collection_time: Tue Mar 11 14:04:37 MET 1997
_current_directory: /
_crash_kernel: /var/adm/crash/vmunix.7
_crash_core: /var/adm/crash/vmcore.7
_crash_arch: alpha
_crash_os: Digital UNIX
_host_version: Digital UNIX V4.0B  (Rev. 564); Tue Mar 11 11:22:24 MET 1997 
_crash_version: Digital UNIX V4.0B  (Rev. 564); Tue Mar 11 11:22:24 MET 1997 

thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
_crashtime:  struct {
    tv_sec = 858085045
    tv_usec = 859663
} 
_boottime:  struct {
    tv_sec = 858082520
    tv_usec = 608766
} 
_config:  struct {
    sysname = "OSF1"
    nodename = "prever.vbe.dec.com"
    release = "V4.0"
    version = "564"
    machine = "alpha"
} 
_cpu:  49 
_system_string:  0xffffffffff800f58 = "AlphaServer 4100 5/300 2MB" 
_ncpus:  4 
_avail_cpus:  4 
_partial_dump:  1 
_physmem(MBytes):  255 
_panic_string:  0xfffffc000066b6a0 = "kernel memory fault" 
_paniccpu:  2 
_panic_thread:  0xfffffc000ff0fb80 
_preserved_message_buffer_begin: 
struct {
    msg_magic = 0x63061
    msg_bufx = 0xe10
    msg_bufr = 0xc22
    msg_bufc = "Alpha boot: available memory from 0x126e000 to 0xfffe000
Digital UNIX V4.0B  (Rev. 564); Tue Mar 11 11:22:24 MET 1997 
physical memory = 256.00 megabytes.
available memory = 237.56 megabytes.
using 975 buffers containing 7.61 megabytes of memory
Master cpu at slot 0.
Firmware revision: 3.0
PALcode: Digital-UNIX/OSF version 1.21
AlphaServer 4100 5/300 2MB
Module 1214:334 not in pci option table, can't configure it
Module 1214:334 not in pci option table, can't configure it
pci1 at mcbus0 slot 5
psiop0 at pci1 slot 1
Loading SIOP: script c0001700, reg 5020400, data c000d610
scsi0 at psiop0 slot 0
rz5 at scsi0 target 5 lun 0 (LID=0) (DEC     RRD45   (C) DEC  0436)
tu0: DECchip 21040-AA: Revision: 2.4
tu0 at pci1 slot 2
tu0: DEC TULIP Ethernet Interface, hardware address: 00-00-F8-20-F9-A5
tu0: auto sensing: selected 10Base2|5 (AUI) port
psiop1 at pci1 slot 3
Loading SIOP: script c0013700, reg 5020200, data c0027a10
scsi1 at psiop1 slot 0
rz8 at scsi1 target 0 lun 0 (LID=1) (DEC     RZ28D    (C) DEC 0008) (Wide16)
rz9 at scsi1 target 1 lun 0 (LID=2) (DEC     RZ28D    (C) DEC 0008) (Wide16)
gpc0 at eisa0
pci0 at mcbus0 slot 4
eisa0 at pci0
ace0 at eisa0
ace1 at eisa0
lp0 at eisa0
fdi0 at eisa0
fd0 at fdi0 unit 0
trio0 at pci0 slot 2
trio0: S3 Trio64 (SVGA) - Plug N' Play - 1.0 Mb
No FRU table entry found in console-rpb
--Not creating a FRU table errorlog packet
lvm0: configured.
lvm1: configured.
kernel console: trio0
dli: configured
ATM Subsystem configured with 4 restart threads
ATM UNI 3.x signalling: configured
ATM IP interface: configured
ADVFS: using 2322 buffers containing 18.14 megabytes of memory
Starting secondary cpu 1
Starting secondary cpu 2
Starting secondary cpu 3
LDBL: Failure to config resolve bus pci0, status = 22
ptstr_probe: called.
ptstr_probe: active_boards = 0
ptstr_probe: PTSTR0: device is in slot 4
ptstr_probe: PTSTR0: device is in bus 1
ptstr_probe: sc->sc_memptr = 4c00000
ptstr_probe: Base Address Register 0 = 0x5020100
ptstr_probe: Base Address Register 1 = 0x100101
ptstr_probe: Base Address Register 2 = 0x4800000
ptstr_probe: Sparse base = 0xfffffcfa90000000
ptstr_probe: Dense  base = 0xfffffcfb04800000
ptstr0 at pci1 slot 4ptstr_attach: sc->sc_hid = 0xfde2c80
 - Attached
ptstr_probe: called.
ptstr_probe: active_boards = 1
ptstr_probe: PTSTR1: device is in slot 5
ptstr_probe: PTSTR1: device is in bus 1
ptstr_probe: sc->sc_memptr = 4400000
ptstr_probe: Base Address Register 0 = 0x5020000
ptstr_probe: Base Address Register 1 = 0x100001
ptstr_probe: Base Address Register 2 = 0x4000000
ptstr_probe: Sparse base = 0xfffffcfa80000000
ptstr_probe: Dense  base = 0xfffffcfb04000000
ptstr1 at pci1 slot 5ptstr_attach: sc->sc_hid = 0xfde2ea0
 - Attached
ptstr_cfg: strmod_add returned 67108864
Node ID is 00-00-f8-20-f9-a5 (from device tu0)
dna_netman: configured
wan_utilities: configured
ctf_base: configured
dna_dli: configured
Node UID is 3eb54a56-9a09-11d0-821a-0000f820f9a5
dna_base: configured
dna_xti: configured
SuperLAT. Copyright 1994 Meridian Technology Corp. All rights reserved.

trap: invalid memory ifetch access from kernel mode


trap: invalid memory read access from kernel mode

    faulting virtual address:     0x0000000000000008
    pc of faulting instruction:   0xfffffc00004626bc
    ra contents at time of fault: 0xfffffc0000459540
    sp contents at time of fault: 0xffffffff8fa03188

panic (cpu 2): kernel memory fault
device string for dump = SCSI 1 3 0 1 100 0 0.
DUMP.prom: dev SCSI 1 3 0 1 100 0 0, block 262144
device string for dump = SCSI 1 3 0 1 100 0 0.
DUMP.prom: dev SCSI 1 3 0 1 100 0 0, block 262144
"
} 
_preserved_message_buffer_end: 
_kernel_process_status_begin: 
  PID	COMM
00000	kernel idle
00001	init
00003	kloadsrv
00092	update
00194	syslogd
00196	binlogd
00256	routed
00281	rwhod
00364	portmap
00366	nfsiod
00369	rpc.statd
00371	rpc.lockd
00377	automount
00401	dnalimd
00404	dnaevld
00415	ctfd
00445	dnascd
00446	dnansd
00447	dnaksd
00451	dnsadv
00493	dtssd
00499	dnanoded
00536	osaknmd
00575	sendmail
00623	snmpd
00632	os_mibs
00663	advfsd
00674	inetd
00707	cron
00735	lpd
00773	dtlogin
00786	getty
00790	Xdec
00799	dtlogin
00813	dxconsole
00814	dtgreet
00874	rlogind
00875	csh
00879	telnetd
00880	csh
00903	ss7nsd
00909	ss7_bep_ccitt.ex
00916	ss7_dir_ccitt.ex
00920	sh
00922	csh
00924	sh
00926	dbx
00928	ss7_fep_ccitt.ex
00931	telnetd
00932	csh
00941	csh
00942	dbx
00946	csh
00954	csh
00957	csh
_kernel_process_status_end: 
_current_pid:  0 
_current_tid:  0xfffffc000ff0f8c0 
_proc_thread_list_begin: 
thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
thread 0xfffffc000ff0fb80 stopped at   [stop_secondary_cpu:513
,0xfffffc00004f8f68]	 Source not available
thread 0xfffffc000fde4000 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde42c0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde4580 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde4840 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde4b00 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde4dc0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde5080 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde5340 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde5600 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde58c0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde5b80 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001328000 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00013282c0 stopped at   [stop_secondary_cpu:513
,0xfffffc00004f8f68]	 Source not available
thread 0xfffffc0001328580 stopped at   [thread_run:2469 ,0xfffffc00002a9648]	
Source not available
thread 0xfffffc0001328840 stopped at   [thread_run:2469 ,0xfffffc00002a9648]	
Source not available
thread 0xfffffc0001328b00 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001328dc0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001329080 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001329340 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001329600 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001329b80 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001278000 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00012782c0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001278580 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001278840 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001278b00 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001278dc0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001279080 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00044bd340 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00044bd080 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00044bd600 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00044bd8c0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00044bdb80 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000366e000 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000366e2c0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0f600 stopped at   [stop_secondary_cpu:513
,0xfffffc00004f8f68]	 Source not available
thread 0xfffffc000ff0f340 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0f080 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0edc0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0eb00 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0e840 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0e580 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0e2c0 stopped at   [thread_run:2469 ,0xfffffc00002a9648]	
Source not available
_proc_thread_list_end: 

warning: Files compiled -g3: parameter values probably wrong
_dump_begin: 
>  0 boot(0x400000000, 0x5900000058, 0x5b000004b0, 0x5d0000005c, 0x5f0000005e)
["../../../../src/kernel/arch/alpha/machdep.c":2484, 0xfffffc00005026cc]

   1 panic(s = 0xfffffc00006688c8 = "event_timeout: panic request")
["../../../../src/kernel/bsd/subr_prf.c":707, 0xfffffc000027b5dc]
pcpu = 0xfffffc00006c99c0
i = 6719688
mycpu = 0
spl = 4

   2 event_timeout(func = 0xfffffc00005011d0, arg = 0xffffffff8f9fb048, timeout
= 15) ["../../../../src/kernel/arch/alpha/cpu.c":1005, 0xfffffc00004f9d60]
prevcc = 4169741081
nextcc = 7
timer = 0
limit = 4491017955

   3 simple_lock_miss(0xfffffc000059dd44, 0xfffffc00006c7d78, 0x59dd41,
0x59dd41, 0x72f) ["../../../../src/kernel/arch/alpha/lockprim.s":1244,
0xfffffc00005011a0]

   4 tuintr(0x3, 0xfffffc00002383a1, 0x4, 0xfffffc000ff0f8c0, 0x21)
["../../../../src/kernel/io/dec/netif/if_tu.c":2968, 0xfffffc000059dd40]

   5 intr_dispatch_no_post(0x4, 0xfffffc000ff0f8c0, 0x21, 0x300000002f,
0xfffffc00004fe290) ["../../../../src/kernel/arch/alpha/hal/shared_intr.c":234,
0xfffffc0000599f3c]

   6 _XentInt(0x2, 0xfffffc000027b9e0, 0xfffffc000068e3b0, 0xffffffff8f9fb278,
0x56) ["../../../../src/kernel/arch/alpha/locore.s":1049, 0xfffffc00004fe28c]

   7 prf(bufptr = 0xffffffff8f9fb278, va = struct {
    _a0 = 0xffffffff8f9fb330
    _offset = 8
}) ["../../../../src/kernel/bsd/subr_prf.c":955, 0xfffffc000027b9dc]
fmt = 0xfffffc0000669d29
flags = -1024
orig_bufptr = 0xffffffff8f9fb278
rd = 0xfffffc000ff0f8c0

   8 printf(va_alist = -4398039786200)
["../../../../src/kernel/bsd/subr_prf.c":361, 0xfffffc000027abac]
valist = struct {
    _a0 = 0xffffffff8f9fb330
    _offset = 0
}
s = 6724904
ps = 6922808
cpu = 0
buf = {
    [0] '^W'
    [1] '^@'
    [2] '^@'
    [3] '^@'
    [4] '^@'
    [5] '^@'
    [6] '^@'
    [7] '^@'
    [8] '^@'
    [9] '^@'
    [10] '^@'
    [11] '^@'
    [12] '^@'
    [13] '^@'
    [14] '^@'
    [15] '^@'
    [16] '^W'
    [17] '^@'
    [18] '"'
    [19] '^@'
    [20] '^@'
    [21] '^@'
    [22] '^@'
    [23] '^@'
    [24] '^W'
    [25] '^@'
    [26] '^@'
    [27] '^@'
    [28] '^@'
    [29] '^@'
    [30] '^@'
    [31] '^@'
    [32] '^8'
    [33] '\303'
    [34] 'l'
    [35] '^@'
    [36] '^@'
    [37] '\374'
    [38] '\377'
    [39] '\377'
    [40] 'p'
    [41] '^W'
    [42] '^@'
    [43] '^@'
    [44] '^@'
    [45] '^@'
    [46] '^@'
    [47] '^@'
    [48] '8'
    [49] '\242'
    [50] 'i'
    [51] '^@'
    [52] '^@'
    [53] '\374'
    [54] '\377'
    [55] '\377'
    [56] '^'
    [57] '^A'
    [58] 'i'
    [59] '^@'
    [60] '^@'
    [61] '\374'
    [62] '\377'
    [63] '\377'
    [64] 'h'
    [65] 'k'
    [66] 'h'
    [67] '^@'
    [68] '^@'
    [69] '\374'
    [70] '\377'
    [71] '\377'
    [72] '\300'
    [73] '^8'
    [74] '\360'
    [75] '^O'
    [76] '^@'
    [77] '\374'
    [78] '\377'
    [79] '\377'
    [80] '!'
    [81] '^@'
    [82] '^@'
    [83] '^@'
    [84] '^@'
    [85] '^@'
    [86] '^@'
    [87] '^@'
    [88] '\340'
    [89] '^A'
    [90] 'i'
    [91] '^@'
    [92] '^@'
    [93] '\374'
    [94] '\377'
    [95] '\377'
    [96] 'P'
    [97] '^@'
    [98] '^@'
    [99] '^@'
    [100] '^@'
    [101] '^@'
    [102] '^@'
    [103] '^@'
    [104] '/'
    [105] '^G'
    [106] '^@'
    [107] '^@'
    [108] '^@'
    [109] '^@'
    [110] '^@'
    [111] '^@'
    [112] '^@'
    [113] '^@'
    [114] '^@'
    [115] '^@'
    [116] '^@'
    [117] '^@'
    [118] '^@'
    [119] '^@'
    [120] '^B'
    [121] '^@'
    [122] '^@'
    [123] '^@'
    [124] '^@'
    [125] '^@'
    [126] '^@'
    [127] '^@'
}

   9 boot(0x0, 0xfffffc000ff0f8c0, 0x2100000021, 0x7900000021,
0xfffffc0000000008) ["../../../../src/kernel/arch/alpha/machdep.c":2399,
0xfffffc0000502558]

  10 panic(s = 0xfffffc0000668708 = "cpu_ip_intr: panic request")
["../../../../src/kernel/bsd/subr_prf.c":791, 0xfffffc000027b77c]
pcpu = 0xfffffc0000728d90
i = 6924112
mycpu = 0
spl = 5

  11 cpu_ip_intr() ["../../../../src/kernel/arch/alpha/cpu.c":647,
0xfffffc00004f92f0]
mycpu = 0
ipi_mask = 1
ipi_maskp = 0xfffffc0000200854
percpu = (nil)

  12 _XentInt(0x4, 0xfffffc000026051c, 0xfffffc000068e3b0, 0x4, 0x2604dc)
["../../../../src/kernel/arch/alpha/locore.s":1076, 0xfffffc00004fe2cc]

  13 malloc(size = 256) ["../../../../src/kernel/bsd/kern_malloc.c":894,
0xfffffc0000260518]
kbp = 0xfffffc00006c9fc0
s = 4
caller = 0xfffffc000059e7d0

  14 tu_receive_int(0xfffffc00006c7d78, 0xfffffffffc6900c4, 0x18000, 0x1,
0xaa0f00002b0009) ["../../../../src/kernel/io/dec/netif/if_tu.c":3465,
0xfffffc000059e7d0]

  15 tuintr(0x1, 0xfffffc00002383a1, 0x2, 0xfffffc000132bfc0,
0xfffffffffffff317) ["../../../../src/kernel/io/dec/netif/if_tu.c":3008,
0xfffffc000059dde8]

  16 intr_dispatch_no_post(0x2, 0xfffffc000132bfc0, 0xfffffffffffff317, 0x0,
0xfffffc00004fe290) ["../../../../src/kernel/arch/alpha/hal/shared_intr.c":234,
0xfffffc0000599f3c]

  17 _XentInt(0x1, 0xfffffc0000254b78, 0xfffffc000068e3b0, 0x1, 0x254b38)
["../../../../src/kernel/arch/alpha/locore.s":1049, 0xfffffc00004fe28c]

  18 softclock_scan() ["../../../../src/kernel/bsd/kern_clock.c":1315,
0xfffffc0000254b74]
p1 = 0xfffffc0003cf1ca0
arg = 0xfffffc000132bfc0 = "^�\234q"
func = 0xfffffc00004594f8
a = -3305
t = struct {
    tv_sec = 316034
    tv_usec = 0
}
head = 0x6fe338
myprocessor = 0xfffffc0000200400
type = 2
s = 5

  19 hardclock(pc = 0xfffffc00004699f0 = " ")
["../../../../src/kernel/bsd/kern_clock.c":1101, 0xfffffc000025468c]
thread = 0x2e59f1
p = 0xfffffc000ff0b170
dosoftscan = 1
myprocessor = 0xfffffc0000200400
msp = 0x1
t = struct {
    tv_sec = 858085045
    tv_usec = 854663
}
time_update = 267432304
doresettodr = 316034
item_list = struct {
    function = 18446739675665139796
    out_flags = 8
    in_flags = 0
    rtn_status = 1
    next_function = 0xfffffc000af2cab8
    input_data = 18446739675670157680
    output_data = 18446739675669964384
}
sys_time = struct {
    seconds = -4398041296312
    microseconds = 8
}
user_time = struct {
    seconds = -4398043907828
    microseconds = 14
}

  20 _XentInt(0x0, 0xfffffc00004699f0, 0xfffffc000068e3b0, 0x0, 0x4699c0)
["../../../../src/kernel/arch/alpha/locore.s":1032, 0xfffffc00004fe264]

  21 csq_lateral(sqh = (nil), sq = 0xfffffc0000719cf0)
["../../../../src/kernel/streams/str_synch.c":1010, 0xfffffc00004699ec]
psqh = 0xfffffc0000719d20
_ssavpri = 0

  22 runq_run() ["../../../../src/kernel/streams/str_runq.c":108,
0xfffffc00004625c0]
sq = 0xfffffc000db72700
q = 0x4
_ssavpri = 0

  23 netisr_thread() ["../../../../src/kernel/net/netisr.c":841,
0xfffffc00002b7230]
isr = 0xfffffc0000462550
netisr = 0xfffffc00006cd488
handled = 12
s = 0
thread = 0xfffffc000ff0f8c0

_dump_end: 
_kernel_thread_list_begin: 
thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
thread 0xfffffc000ff0fb80 stopped at   [stop_secondary_cpu:513
,0xfffffc00004f8f68]	 Source not available
thread 0xfffffc000fde4000 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde42c0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde4580 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde4840 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde4b00 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde4dc0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde5080 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde5340 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde5600 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde58c0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000fde5b80 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001328000 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00013282c0 stopped at   [stop_secondary_cpu:513
,0xfffffc00004f8f68]	 Source not available
thread 0xfffffc0001328580 stopped at   [thread_run:2469 ,0xfffffc00002a9648]	
Source not available
thread 0xfffffc0001328840 stopped at   [thread_run:2469 ,0xfffffc00002a9648]	
Source not available
thread 0xfffffc0001328b00 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001328dc0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001329080 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001329340 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001329600 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001329b80 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001278000 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00012782c0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001278580 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001278840 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001278b00 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001278dc0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc0001279080 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00044bd340 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00044bd080 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00044bd600 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00044bd8c0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc00044bdb80 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000366e000 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000366e2c0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0f600 stopped at   [stop_secondary_cpu:513
,0xfffffc00004f8f68]	 Source not available
thread 0xfffffc000ff0f340 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0f080 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0edc0 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0eb00 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0e840 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0e580 stopped at   [thread_block:2097 ,0xfffffc00002a8e80]	
Source not available
thread 0xfffffc000ff0e2c0 stopped at   [thread_run:2469 ,0xfffffc00002a9648]	
Source not available
_kernel_thread_list_end: 
_savedefp:  (nil) 
_kernel_memory_fault_data_begin:  
struct {
    fault_va = 0x3ff800d3a98
    fault_pc = 0x3ff800d3a98
    fault_ra = 0xfffffc0000462694
    fault_sp = 0xffffffff8fa03968
    access = 0xffffffffffffffff
    status = 0x0
    cpunum = 0x2
    count = 0x2
    pcb = 0xffffffff8fa03a38
    thread = 0xfffffc000ff0fb80
    task = 0xfffffc000ff0aa80
    proc = 0xfffffc000ff0aca0
} 
_kernel_memory_fault_data_end:  
_uptime: .70 hours

thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
paniccpu: 0x2 
machine_slot[paniccpu]: struct {
    is_cpu = 0x1
    cpu_type = 0xf
    cpu_subtype = 0x16
    running = 0x1
    cpu_ticks = {
        [0] 0x126ed
        [1] 0x0
        [2] 0x1450ba
        [3] 0x18bec2
        [4] 0x3ae
    }
    clock_freq = 0x4b0
    error_restart = 0x0
    cpu_panicstr = 0xfffffc000066b6a0 = "kernel memory fault"
    cpu_panic_thread = 0xfffffc000ff0fb80
} 
tset machine_slot[paniccpu].cpu_panic_thread: 
Begin Trace for machine_slot[paniccpu].cpu_panic_thread: 
thread 0xfffffc000ff0fb80 stopped at   [stop_secondary_cpu:513
,0xfffffc00004f8f68]	 Source not available

warning: Files compiled -g3: parameter values probably wrong
>  0 stop_secondary_cpu(do_lwc = 0x0)
["../../../../src/kernel/arch/alpha/cpu.c":507, 0xfffffc00004f8f64]
   1 panic(s = 0xfffffc00006688c8 = "event_timeout: panic request")
["../../../../src/kernel/bsd/subr_prf.c":703, 0xfffffc000027b5d0]
   2 event_timeout(func = 0xfffffc000027b800, arg = 0xfffffc00006ccc30, timeout
= 0x5) ["../../../../src/kernel/arch/alpha/cpu.c":1005, 0xfffffc00004f9d60]
   3 xcpu_puts(s = 0xfffffc000069e3b0, prfbufp = 0xfffffc00006ccc30)
["../../../../src/kernel/bsd/subr_prf.c":844, 0xfffffc000027b864]
   4 printf(va_alist = 0xfffffc000061e618)
["../../../../src/kernel/bsd/subr_prf.c":379, 0xfffffc000027ac60]
   5 panic(s = 0xfffffc000066b6a0 = "kernel memory fault")
["../../../../src/kernel/bsd/subr_prf.c":753, 0xfffffc000027b728]
   6 trap() ["../../../../src/kernel/arch/alpha/trap.c":1512, 0xfffffc000050a568]
   7 _XentMM(0x1, 0xfffffc00004626bc, 0xfffffc000068e3b0, 0xfffffc000af2c200,
0x2608f4) ["../../../../src/kernel/arch/alpha/locore.s":1424, 0xfffffc00004fe634]
   8 qenable(q = 0xfffffc000132ba40)
["../../../../src/kernel/streams/str_runq.c":144, 0xfffffc00004626b8]
   9 str_timeout(0xfffffffffffff2fa, 0x1, 0xfffffc0000254b8c,
0xfffffc00004594f8, 0xfffffc0000201000)
["../../../../src/kernel/streams/str_env.c":394, 0xfffffc0000459530]
  10 softclock_scan() ["../../../../src/kernel/bsd/kern_clock.c":1316,
0xfffffc0000254b88]
  11 hardclock(pc = 0xfffffc000027c840 = "/")
["../../../../src/kernel/bsd/kern_clock.c":1101, 0xfffffc000025468c]
  12 _XentInt(0x0, 0xfffffc000027c840, 0xfffffc000068e3b0, 0x336666, 0x30)
["../../../../src/kernel/arch/alpha/locore.s":1032, 0xfffffc00004fe264]
  13 printn(bufptr = 0xffffffff8fa03531, bufend = 0xffffffff8fa03588, n = 0x30,
b = 0x10) ["../../../../src/kernel/bsd/subr_prf.c":1244, 0xfffffc000027c83c]
  14 prf(bufptr = 0xffffffff8fa0352c, va = struct {
    _a0 = 0xffffffff8fa035c0
    _offset = 0x10
}) ["../../../../src/kernel/bsd/subr_prf.c":1005, 0xfffffc000027bb60]
  15 printf(va_alist = 0xfffffc000066b5e0)
["../../../../src/kernel/bsd/subr_prf.c":361, 0xfffffc000027abac]
  16 trap() ["../../../../src/kernel/arch/alpha/trap.c":1504, 0xfffffc000050a534]
  17 _XentMM(0x0, 0x3ff800d3a98, 0xfffffc000068e3b0, 0xfffffc0001dcda00,
0x462678) ["../../../../src/kernel/arch/alpha/locore.s":1424, 0xfffffc00004fe634]
End Trace for machine_slot[paniccpu].cpu_panic_thread: 

thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
"cpu_data" is not an array
thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
_stack_trace[0]_begin: 
>  0 boot(0x400000000, 0x5900000058, 0x5b000004b0, 0x5d0000005c, 0x5f0000005e)
["../../../../src/kernel/arch/alpha/machdep.c":2484, 0xfffffc00005026cc]
   1 panic(s = 0xfffffc00006688c8 = "event_timeout: panic request")
["../../../../src/kernel/bsd/subr_prf.c":707, 0xfffffc000027b5dc]
   2 event_timeout(func = 0xfffffc00005011d0, arg = 0xffffffff8f9fb048, timeout
= 15) ["../../../../src/kernel/arch/alpha/cpu.c":1005, 0xfffffc00004f9d60]
   3 simple_lock_miss(0xfffffc000059dd44, 0xfffffc00006c7d78, 0x59dd41,
0x59dd41, 0x72f) ["../../../../src/kernel/arch/alpha/lockprim.s":1244,
0xfffffc00005011a0]
   4 tuintr(0x3, 0xfffffc00002383a1, 0x4, 0xfffffc000ff0f8c0, 0x21)
["../../../../src/kernel/io/dec/netif/if_tu.c":2968, 0xfffffc000059dd40]
   5 intr_dispatch_no_post(0x4, 0xfffffc000ff0f8c0, 0x21, 0x300000002f,
0xfffffc00004fe290) ["../../../../src/kernel/arch/alpha/hal/shared_intr.c":234,
0xfffffc0000599f3c]
   6 _XentInt(0x2, 0xfffffc000027b9e0, 0xfffffc000068e3b0, 0xffffffff8f9fb278,
0x56) ["../../../../src/kernel/arch/alpha/locore.s":1049, 0xfffffc00004fe28c]
   7 prf(bufptr = 0xffffffff8f9fb278, va = struct {
    _a0 = 0xffffffff8f9fb330
    _offset = 8
}) ["../../../../src/kernel/bsd/subr_prf.c":955, 0xfffffc000027b9dc]
   8 printf(va_alist = -4398039786200)
["../../../../src/kernel/bsd/subr_prf.c":361, 0xfffffc000027abac]
   9 boot(0x0, 0xfffffc000ff0f8c0, 0x2100000021, 0x7900000021,
0xfffffc0000000008) ["../../../../src/kernel/arch/alpha/machdep.c":2399,
0xfffffc0000502558]
  10 panic(s = 0xfffffc0000668708 = "cpu_ip_intr: panic request")
["../../../../src/kernel/bsd/subr_prf.c":791, 0xfffffc000027b77c]
  11 cpu_ip_intr() ["../../../../src/kernel/arch/alpha/cpu.c":647,
0xfffffc00004f92f0]
  12 _XentInt(0x4, 0xfffffc000026051c, 0xfffffc000068e3b0, 0x4, 0x2604dc)
["../../../../src/kernel/arch/alpha/locore.s":1076, 0xfffffc00004fe2cc]
  13 malloc(size = 256) ["../../../../src/kernel/bsd/kern_malloc.c":894,
0xfffffc0000260518]
  14 tu_receive_int(0xfffffc00006c7d78, 0xfffffffffc6900c4, 0x18000, 0x1,
0xaa0f00002b0009) ["../../../../src/kernel/io/dec/netif/if_tu.c":3465,
0xfffffc000059e7d0]
  15 tuintr(0x1, 0xfffffc00002383a1, 0x2, 0xfffffc000132bfc0,
0xfffffffffffff317) ["../../../../src/kernel/io/dec/netif/if_tu.c":3008,
0xfffffc000059dde8]
  16 intr_dispatch_no_post(0x2, 0xfffffc000132bfc0, 0xfffffffffffff317, 0x0,
0xfffffc00004fe290) ["../../../../src/kernel/arch/alpha/hal/shared_intr.c":234,
0xfffffc0000599f3c]
  17 _XentInt(0x1, 0xfffffc0000254b78, 0xfffffc000068e3b0, 0x1, 0x254b38)
["../../../../src/kernel/arch/alpha/locore.s":1049, 0xfffffc00004fe28c]
  18 softclock_scan() ["../../../../src/kernel/bsd/kern_clock.c":1315,
0xfffffc0000254b74]
  19 hardclock(pc = 0xfffffc00004699f0 = " ")
["../../../../src/kernel/bsd/kern_clock.c":1101, 0xfffffc000025468c]
  20 _XentInt(0x0, 0xfffffc00004699f0, 0xfffffc000068e3b0, 0x0, 0x4699c0)
["../../../../src/kernel/arch/alpha/locore.s":1032, 0xfffffc00004fe264]
  21 csq_lateral(sqh = (nil), sq = 0xfffffc0000719cf0)
["../../../../src/kernel/streams/str_synch.c":1010, 0xfffffc00004699ec]
  22 runq_run() ["../../../../src/kernel/streams/str_runq.c":108,
0xfffffc00004625c0]
  23 netisr_thread() ["../../../../src/kernel/net/netisr.c":841,
0xfffffc00002b7230]
_stack_trace[0]_end: 

thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
"cpu_data" is not an array
thread 0xfffffc00013282c0 stopped at   [stop_secondary_cpu:513
,0xfffffc00004f8f68]	 Source not available

warning: Files compiled -g3: parameter values probably wrong
_stack_trace[1]_begin: 
>  0 stop_secondary_cpu(do_lwc = 1)
["../../../../src/kernel/arch/alpha/cpu.c":507, 0xfffffc00004f8f64]
   1 panic(s = 0xfffffc0000668708 = "cpu_ip_intr: panic request")
["../../../../src/kernel/bsd/subr_prf.c":733, 0xfffffc000027b67c]
   2 cpu_ip_intr() ["../../../../src/kernel/arch/alpha/cpu.c":647,
0xfffffc00004f92f0]
   3 _XentInt(0x0, 0xfffffc00002aa9b8, 0xfffffc000068e3b0, 0x3fff, 0x1)
["../../../../src/kernel/arch/alpha/locore.s":1076, 0xfffffc00004fe2cc]
   4 idle_thread() ["../../../../src/kernel/kern/sched_prim.c":3324,
0xfffffc00002aa9b4]
_stack_trace[1]_end: 

thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
"cpu_data" is not an array
thread 0xfffffc000ff0fb80 stopped at   [stop_secondary_cpu:513
,0xfffffc00004f8f68]	 Source not available

warning: Files compiled -g3: parameter values probably wrong
_stack_trace[2]_begin: 
>  0 stop_secondary_cpu(do_lwc = 0)
["../../../../src/kernel/arch/alpha/cpu.c":507, 0xfffffc00004f8f64]
   1 panic(s = 0xfffffc00006688c8 = "event_timeout: panic request")
["../../../../src/kernel/bsd/subr_prf.c":703, 0xfffffc000027b5d0]
   2 event_timeout(func = 0xfffffc000027b800, arg = 0xfffffc00006ccc30, timeout
= 5) ["../../../../src/kernel/arch/alpha/cpu.c":1005, 0xfffffc00004f9d60]
   3 xcpu_puts(s = 0xfffffc000069e3b0, prfbufp = 0xfffffc00006ccc30)
["../../../../src/kernel/bsd/subr_prf.c":844, 0xfffffc000027b864]
   4 printf(va_alist = -4398040095208)
["../../../../src/kernel/bsd/subr_prf.c":379, 0xfffffc000027ac60]
   5 panic(s = 0xfffffc000066b6a0 = "kernel memory fault")
["../../../../src/kernel/bsd/subr_prf.c":753, 0xfffffc000027b728]
   6 trap() ["../../../../src/kernel/arch/alpha/trap.c":1512, 0xfffffc000050a568]
   7 _XentMM(0x1, 0xfffffc00004626bc, 0xfffffc000068e3b0, 0xfffffc000af2c200,
0x2608f4) ["../../../../src/kernel/arch/alpha/locore.s":1424, 0xfffffc00004fe634]
   8 qenable(q = 0xfffffc000132ba40)
["../../../../src/kernel/streams/str_runq.c":144, 0xfffffc00004626b8]
   9 str_timeout(0xfffffffffffff2fa, 0x1, 0xfffffc0000254b8c,
0xfffffc00004594f8, 0xfffffc0000201000)
["../../../../src/kernel/streams/str_env.c":394, 0xfffffc0000459530]
  10 softclock_scan() ["../../../../src/kernel/bsd/kern_clock.c":1316,
0xfffffc0000254b88]
  11 hardclock(pc = 0xfffffc000027c840 = "/")
["../../../../src/kernel/bsd/kern_clock.c":1101, 0xfffffc000025468c]
  12 _XentInt(0x0, 0xfffffc000027c840, 0xfffffc000068e3b0, 0x336666, 0x30)
["../../../../src/kernel/arch/alpha/locore.s":1032, 0xfffffc00004fe264]
  13 printn(bufptr = 0xffffffff8fa03531, bufend = 0xffffffff8fa03588, n = 48, b
= 16) ["../../../../src/kernel/bsd/subr_prf.c":1244, 0xfffffc000027c83c]
  14 prf(bufptr = 0xffffffff8fa0352c, va = struct {
    _a0 = 0xffffffff8fa035c0
    _offset = 16
}) ["../../../../src/kernel/bsd/subr_prf.c":1005, 0xfffffc000027bb60]
  15 printf(va_alist = -4398039779872)
["../../../../src/kernel/bsd/subr_prf.c":361, 0xfffffc000027abac]
  16 trap() ["../../../../src/kernel/arch/alpha/trap.c":1504, 0xfffffc000050a534]
  17 _XentMM(0x0, 0x3ff800d3a98, 0xfffffc000068e3b0, 0xfffffc0001dcda00,
0x462678) ["../../../../src/kernel/arch/alpha/locore.s":1424, 0xfffffc00004fe634]
_stack_trace[2]_end: 

thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
"cpu_data" is not an array
thread 0xfffffc000ff0f600 stopped at   [stop_secondary_cpu:513
,0xfffffc00004f8f68]	 Source not available

warning: Files compiled -g3: parameter values probably wrong
_stack_trace[3]_begin: 
>  0 stop_secondary_cpu(do_lwc = 1)
["../../../../src/kernel/arch/alpha/cpu.c":507, 0xfffffc00004f8f64]
   1 panic(s = 0xfffffc0000668708 = "cpu_ip_intr: panic request")
["../../../../src/kernel/bsd/subr_prf.c":761, 0xfffffc000027b748]
   2 cpu_ip_intr() ["../../../../src/kernel/arch/alpha/cpu.c":647,
0xfffffc00004f92f0]
   3 _XentInt(0x0, 0xfffffc00002e9974, 0xfffffc000068e3b0, 0xfffffc0002a05e00,
0x2e995c) ["../../../../src/kernel/arch/alpha/locore.s":1076, 0xfffffc00004fe2cc]
   4 dli_intr(0xfffffc00002e98a0, 0xfffffc000069ac28, 0xfffffc00006c9970,
0xfffffc000069a660, 0xffffffff00000001)
["../../../../src/kernel/dli/dli_input.c":252, 0xfffffc00002e9970]
   5 netisr_thread() ["../../../../src/kernel/net/netisr.c":841,
0xfffffc00002b7230]
_stack_trace[3]_end: 

thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
_kdbx_sum_start:
Hostname : prever.vbe.dec.com
cpu: AlphaServer 4100 5/300 2MB	avail: 4
Boot-time:	Tue Mar 11 13:15:20 1997
Time:	Tue Mar 11 13:57:25 1997
Kernel : OSF1 release V4.0 version 564 (alpha)
_kdbx_sum_end:
_kdbx_swap_start:

       Swap device name              Size       In Use       Free
--------------------------------  ----------  ----------  ----------
/dev/rz9b                            512000k          8k     511992k
                                      64000p          1p      63999p
--------------------------------  ----------  ----------  ----------
Total swap partitions:    1          512000k          8k     511992k
                                      64000p          1p      63999p
_kdbx_swap_end:
_kdbx_proc_start:
Addr        PID   PPID  PGRP  UID   NICE SIGCATCH P_SIG    Event       Flags
=========== ===== ===== ===== ===== ==== ======== ======== ===========
============
k0x0ff0aca0     0     0     0     0    0 00000000 00000000        NULL in sys
k0x0132e220     1     0     1     0    0 307a62ff 00000000        NULL in
contign pagv
k0x0132f720     3     1     3     0    0 00004000 00000000        NULL in pagv
k0x044a3720    92     1    92     0    0 00002000 00000000        NULL in pagv
k0x044ba220   194     1   194     0    0 60086001 00000000        NULL in pagv
k0x044baca0   196     1   196     0    0 00004001 00000000        NULL in pagv
k0x044bb720   256     1   256     0    0 60006003 00000000        NULL in pagv
k0x0374eca0   281     1   281     0    0 00002001 00000000        NULL in pagv
k0x0374e220   364     1   364     0    0 00080628 00000000        NULL in pagv
k0x0374f720   366     1   366     0    0 00000000 00000000        NULL in pagv
k0x03fe0ca0   369     1     0     0    0 00002000 00000000        NULL in pagv
ctty
k0x03fe0220   371     1   371     0    0 00002000 00000000        NULL in pagv
k0x03fe1720   377     1     0     0    0 00084001 00000000        NULL in pagv
k0x0366c220   401     1     0     0    0 00084003 00000000        NULL in pagv
k0x0366d720   404   401   404     0    0 20082000 00000000        NULL in pagv
k0x044a2220   415     1   415     0    0 00000001 00000000        NULL in pagv
k0x03846ca0   445   401     0     0    0 00887efb 00000000        NULL in pagv
k0x03847720   446   445     0     0    0 00084000 00000000        NULL in pagv
k0x029c4220   447   445     0     0    0 00080000 00000000        NULL in pagv
k0x029c5720   451   401   451     0    0 00484007 00000000        NULL in pagv
k0x044a2ca0   493     1     0     0    0 00004edf 00000000        NULL in pagv
k0x0ed24ca0   499   401     0     0    0 00000000 00000000        NULL in pagv
k0x03082220   536     1   536     0    0 00000000 00000000        NULL in pagv
k0x0ed24220   575     1     0     0    0 00086000 00000000        NULL in pagv
k0x0ee5aca0   623     1   623     0    0 20004002 00000000        NULL in pagv
k0x0dbb4220   632     1   632     0    0 00004002 00000000        NULL in pagv
k0x0ee5b720   663     1   140     0   -1 60027eff 00000000        NULL in pagv
ctty
k0x0dbb5720   674     1   674     0    0 40082001 00000000        NULL in pagv
k0x0ee5a220   707     1   707     0    0 00002000 00000000        NULL in pagv
k0x03083720   735     1   735     0    0 00084007 00000000        NULL in pagv
k0x0e159720   773     1   773     0    0 20084003 00000000        NULL in pagv
k0x0132eca0   786     1   786     0    0 00000000 00000000        NULL in pagv
ctty
k0x0e158220   790   773   790     0    0 00004003 00000000        NULL in pagv
k0x0175a220   799   773   799     0    0 20004002 00000000        NULL in pagv
k0x04406220   813     1   804     0    0 00000000 00000000        NULL in pagv
k0x0175aca0   814   799   814     0    0 00004000 00000000        NULL in pagv
k0x0ed25720   874   674   874     0    0 00084027 00000000        NULL in pagv
k0x03082ca0   875   874   875    21    0 01882003 00000000        NULL in pagv
ctty
k0x01906220   879   674   879     0    0 00084025 00000000        NULL in pagv
k0x01907720   880   879   880    21    0 01882003 00000000        NULL in pagv
ctty
k0x0e158ca0   903   674   903     0    0 67003eff 00000000        NULL in pagv
k0x0366cca0   909     1   883    21    0 67003eff 00000000        NULL in pagv
ctty
k0x0d9d3720   916     1   883    21    0 67003eff 00000000        NULL in pagv
ctty
k0x01906ca0   920   880   920    21    0 00003efb 00000000        NULL in pagv
ctty
k0x0175b720   922   920   922     0    0 00082002 00000000        NULL in pagv
ctty
k0x0d9d2220   924   922   924     0    0 00003efb 00000002        NULL in pagv
ctty
k0x0d9d2ca0   926   924   924     0    0 0000068e 00000000        NULL in pagv
ctty
k0x04407720   928   926   924     0    0 67003eff 00000000        NULL in exit
pagv ctty
k0x03a24ca0   931   674   931     0    0 00084025 00000000        NULL in pagv
k0x03a25720   932   931   932    21    0 01882003 00000000        NULL in pagv
ctty
k0x09f18220   941   932   941     0    0 00082002 00000000        NULL in pagv
ctty
k0x09f18ca0   942   941   942     0    0 0000068e 00000000        NULL in pagv
ctty
k0x04406ca0   946   875   946   415    0 01882003 00000000        NULL in pagv
ctty
k0x029c4ca0   954   946   954    21    0 01882003 00000000        NULL in pagv
ctty
k0x03846220   957   954   957     0    0 00082002 00000000        NULL in pagv
ctty
_kdbx_proc_end:

thread 0xfffffc000ff0f8c0 stopped at  [boot:2484 ,0xfffffc00005026cc]	 Source
not available
Audit subsystem disabled

No audit data to be saved
#
_crash_data_collection_finished:
T.RTitleUserPersonal
Name
DateLines
1502.1SMURF::DENHAMDigital UNIX KernelTue Mar 11 1997 09:1216
    You have a strange crash dump there:
    
    _kernel_memory_fault_data_begin:
    struct {
        fault_va = 0x3ff800d3a98
        fault_pc = 0x3ff800d3a98     <<<<<<<<< !!!!
        fault_ra = 0xfffffc0000462694
        fault_sp = 0xffffffff8fa03968
        access = 0xffffffffffffffff
        status = 0x0
        cpunum = 0x2
     
    You'd best get me pointers to the compressed dump files
    so I can see what's up.
    
    
1502.2Application memory corruptor?WTFN::SCALESDespair is appropriate and inevitable.Tue Mar 11 1997 09:3615
Pierre, there are two separate problems here:  there's your thread spinning
in mutex un[b]lock and there's the system crasher.  The second is a problem
in the kernel -- i.e., there's nothing that DECthreads _should_ be able to do
that will crash the system.  The first occurs because the thread is unable to
acquire the DECthreads scheduler spinlock.  While this could be the result of
a problem inside DECthreads, nearly every time this sort of problem has been
reported it's been a memory corruptor in the application.

So, once you've made the system dump available to Jeff, you might want to
examine your code for use of uninitialized local pointer variables, because,
in the absence of any other data, I would suspect a memory corruptor in the
application.


					Webb
1502.3Application memory corruptor ? May be not...TAEC::GARNEROTue Mar 11 1997 11:0224
RE.-1:

Webb, just talking about the mutex unlock, we (DECss7) have already
experienced a similar problem [IPMT MGO102540 (CFS.46628)] on V3.2G.

After 3 months of investigation, it has been proved that the problem
was located in the pthread library.

I am doing, at this moment, a test (on V4.0B) similar to the one which
revealed the bug in the V3.2G pthread library (draft 4 interface) and 
I guess that it is the same problem.

Of course, some things have changed in the DECss7 application as we are 
now using the POSIX 1003.1c pthread interface.

Anyway, is it sure that the MGO102540 problem is no more in the V4.0B
pthread library ????

At this moment, I am waiting for the official patches fixing this problem
on V3.2x but is there any plan to fix the problem on V4.0B ?

Thanks 

Pierre
1502.4SMURF::DENHAMDigital UNIX KernelTue Mar 11 1997 12:174
    Just FYI the dump in .0 -- looks like a streams-related problem.
    Streams guru is investigating.... The crash doesn't even occur
    on any of the ss7 threads; it's in one of the kernel's network
    protocol threads.
1502.5Maybe not, but there's little data either way.WTFN::SCALESDespair is appropriate and inevitable.Tue Mar 11 1997 13:5614
.3> we (DECss7) have already experienced a similar problem [...] on V3.2G.

True.  However, the implementation of thread blocking and unblocking on V4 is
completely different from that on V3, so, while the one stack trace which you've
provided makes it look like a similar problem, there isn't much reason to
believe that it's the -same- problem.

Furthermore, that stack trace would also occur in the corruption case (as well
as a couple of more pathological cases), so there are several possible
explanations.  In the absence of more data or a small reproducer test program,
there's not much more we can say.


				Webb
1502.6just a hint to save time of anybodyTAEC::GARNEROWed Mar 12 1997 01:4230
RE.-1

Webb,

your are right. I am not sure at 100% that the DECss7 code is correct.

Anyway, my previous experience on this kind of problem shows that
making a small reproducer is quite impossible (we have already tried
to achieve a reproducer when investigating the problem on V3.2G 
without any success). 

Now, during this last investigation, we have reviewed the DECss7 code
at lot of time, and we haven't found any memory corruption problem.
Remember that an external consultant spend 1 week to investigate the
problem here, at Valbonne. Its conclusion was that there was no problem
on the DECss7 code (related to this problem of course :-)))

And as the problem only occurs on multicpu machine (I have performed
a similar test on a monocpu machine with success) I really think 
that memory corruption is not the problem here.

Now, I think we could easily make a small test that way:

 . if the problem is located in the condition-variable morphing 
   optimization (as I suspect), could you generate for me a libpthread.so 
   which is not implementing this optimization (I think that the pthread
   code implementing this optimization is flagged) ? 

Thanks in advance for your support,
Pierre
1502.7DCETHD::BUTENHOFDave Butenhof, DECthreadsWed Mar 12 1997 08:1514
>And as the problem only occurs on multicpu machine (I have performed
>a similar test on a monocpu machine with success) I really think 
>that memory corruption is not the problem here.

I don't intend to argue about any of the other points, but this statement
deserves some comment.

In fact, you've gotten it entirely backwards. Because the problem only occurs
on a multiprocessor, it is far more likely to be memory corruption. A
uniprocessor, running only one thread at a time, is very limited in the ways
it can creatively corrupt memory. A multiprocessor, running multiple threads
at the same time, asynchronously, has far more opportunities for "creativity".

	/dave
1502.8TAEC::GARNEROWed Mar 12 1997 11:0313
 Could someone comfirm that the libpthread.so delivered with V4.0B contains 
 the condition-variable morphing optimization ?

 If yes, is it possible to generate a libpthread.so without this 
 optimization, please ?

 I could check in 5 minutes if the problem is located in portion of code.

 Many thanks in advance.

 Pierre

1502.9Condition variable morphing is not turned on in Digital UNIX V4.0*PTHRED::PORTANTEPeter Portante, DTN 381-2261, (603)881-2261, MS ZKO2-3/Q18Wed Mar 12 1997 12:046
Pierre,

I have confirmed that in Digital UNIX V4.0* the condition wait morphing
optimization is not enabled.

-Peter
1502.10TAEC::GARNEROThu Mar 13 1997 02:0716
As I get the following error message :

DECthreads bugcheck (version V3.13-435), terminating execution.
vpUpcallThreadUnblocked: (os/kern) invalid argument (4) nxm_resched(49,0)

I think that the problem encountered is the same than the one described
in notes 1457,1481,1505.

Could someone tell me when the patch for V4.0B will be available ?

Is it possible to get an unofficial patch now, in order for us to go ahead
in our tests of the DECss7 product (because this problem is blocking) ?

Thanks,

Pierre
1502.11This bugcheck has nothing to do with your corruptor.WTFN::SCALESDespair is appropriate and inevitable.Thu Mar 13 1997 10:4420
Pierre, the nxm_resched() bugcheck reported in notes 1457 and 1481 is
completely unrelated to the problem reported in this topic.  (I'm assuming
that you think, or hope, that they are related, since you posted your reply
here instead of to one of those topics.)

I will allow that a random memory corruptor in your application could cause
this bugcheck, but it's unlikely in the extreme (the corruptor would have to
overwrite a particular field in the thread control block at a particular
time, and the probability of this is much lower than corrupting the
scheduling spinlock).  Thus, applying the patch for the nxm_resched()
bugcheck will not fix the corruption problem which you are seeing.

.0> Could someone tell me when the patch for V4.0B will be available ?

I believe that Peter made it clear in 1505 that there would be an annoucement
in this conference when the patch is available.  If you need more than that,
you should contact the Digital Unix patch/kitting folks.


					Webb