[Search for users]
[Overall Top Noters]
[List of all Conferences]
[Download this site]
| Title: | ase |
|
| Moderator: | SMURF::GROSSO |
|
| Created: | Thu Jul 29 1993 |
| Last Modified: | Fri Jun 06 1997 |
| Last Successful Update: | Fri Jun 06 1997 |
| Number of topics: | 2114 |
| Total number of notes: | 7347 |
I have a customer that is experiencing problems with Oracle
Parallel Server on a TruCluster. He's running 2 4100's, each
with 512MB, Digital UNIX 4.0b and TCR 1.4. Oracle is version
7.3.2.3. He has all of the 4.0b patches and Oracle patches.
When he tries to start up oracle, he gets the following:
SVRMGR> connect internal
Connected to an idle instance.
SVRMGR> startup parallel
ORACLE instance started.
Total System Global Area 321485824 bytes
Fixed Size 52424 bytes
Variable Size 310579000 bytes
Database Buffers 9830400 bytes
Redo Buffers 1024000 bytes
Database mounted.
ORA-00449: background process 'LCK0' unexpectedly terminated with error
600
ORA-00600: internal error code, arguments: [], [], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [9030], [600], [], [], [], [],
[], []
SVRMGR>
He contacted Oracle, and they seem to think it's an issue with DLM,
so I had him check the messages file and he found the following:
Apr 30 16:48:19 qedscm1 vmunix: dlm_agent: suspending lock activity
Apr 30 16:48:19 qedscm1 vmunix: cnxagent: added node qedmc1
Apr 30 16:48:24 qedscm1 vmunix: dlm_agent: resuming lock activity
Apr 30 17:06:12 qedscm1 vmunix: malloc failed: bucket size=256, # of
failures=10
0
Apr 30 17:06:12 qedscm1 vmunix: malloc failed: bucket size=256, # of
failures=20
0
Apr 30 17:06:12 qedscm1 vmunix: malloc failed: bucket size=256, # of
failures=30
0
Apr 30 17:06:12 qedscm1 vmunix: malloc failed: bucket size=256, # of
failures=40
0
Apr 30 17:06:12 qedscm1 vmunix: malloc failed: bucket size=256, # of
failures=50
I know DLM uses 256 byte buckets, so I had him do some monitoring
while starting up Oracle. I wrote a script uing vmstat -M to
see what was happening with 256 byte buckets and DLM. Here's what
he saw (Note: this is just the 256 buckets and DLM extracted - all
of it would be too long):
bucket# element_size elements_in_use elements_free bytes_in_use
7 256 1242 134 317952
7 256 1254 122 321024
7 256 1277 99 326912
7 256 1288 88 329728
7 256 1314 62 336384
7 256 1325 51 339200
7 256 1336 40 342016
7 256 1347 29 344832
7 256 1358 18 347648
7 256 1378 30 352768
7 256 1380 28 353280
7 256 1391 17 356096
7 256 1402 6 358912
7 256 1413 27 361728
7 256 1424 16 364544
7 256 1435 5 367360
7 256 1446 26 370176
7 256 1457 15 372992
7 256 1468 4 375808
7 256 1479 25 378624
7 256 1490 14 381440
7 256 1501 3 384256
7 256 1512 24 387072
7 256 1523 13 389888
7 256 1535 1 392960
7 256 1545 23 395520
7 256 1556 12 398336
7 256 1567 33 401152
7 256 1578 22 403968
7 256 1600 32 409600
7 256 1608 24 411648
7 256 1616 16 413696
7 256 1622 10 415232
7 256 1633 31 418048
7 256 1644 20 420864
7 256 1667 29 426752
7 256 1687 9 431872
7 256 1695 1 433920
7 256 1708 20 437248
7 256 1718 10 439808
7 256 1730 30 442880
7 256 1742 18 445952
7 256 1752 8 448512
7 256 1765 27 451840
7 256 1775 17 454400
7 256 1787 5 457472
7 256 1798 26 460288
7 256 1809 15 463104
7 256 1820 4 465920
7 256 1832 24 468992
7 256 1843 13 471808
7 256 1861 27 476416
7 256 1869 19 478464
7 256 11135 33 2850560
7 256 11145 23 2853120
7 256 11156 12 2855936
7 256 11167 33 2858752
7 256 11178 22 2861568
7 256 17175 9 4396800
7 256 17187 29 4399872
7 256 17197 19 4402432
7 256 17208 8 4405248
7 256 17219 29 4408064
7 256 28594 14 7320064
7 256 28604 4 7322624
7 256 28616 24 7325696
7 256 28626 14 7328256
7 256 28637 35 7331072
7 256 35453 35 9075968
7 256 35464 24 9078784
7 256 35474 14 9081344
7 256 35486 34 9084416
7 256 35497 23 9087232
7 256 41349 27 10585344
7 256 41360 16 10588160
7 256 41371 5 10590976
7 256 46189 19 11824384
7 256 46200 8 11827200
7 256 46211 29 11830016
7 256 56946 14 14578176
7 256 56958 2 14581248
7 256 56968 24 14583808
7 256 56979 13 14586624
7 256 56990 2 14589440
7 256 63196 4 16178176
7 256 63207 25 16180992
7 256 63218 14 16183808
7 256 63229 3 16186624
7 256 63240 24 16189440
7 256 74067 13 18961152
7 256 74078 2 18963968
7 256 74089 23 18966784
7 256 74100 12 18969600
7 256 74111 1 18972416
7 256 74122 22 18975232
7 256 78033 15 19976448
7 256 78045 3 19979520
7 256 78055 25 19982080
7 256 82756 28 21185536
7 256 82767 17 21188352
7 256 82779 5 21191424
7 256 92384 0 23650304
7 256 92395 21 23653120
7 256 92406 10 23655936
7 256 92417 31 23658752
7 256 92428 20 23661568
7 256 92440 8 23664640
7 256 96746 22 24766976
7 256 96757 11 24769792
7 256 96768 32 24772608
7 256 101460 12 25973760
7 256 101471 33 25976576
7 256 101483 21 25979648
7 256 105753 7 27072768
7 256 105764 28 27075584
7 256 111844 28 28632064
7 256 111856 16 28635136
7 256 111866 6 28637696
7 256 111877 27 28640512
7 256 115789 19 29641984
7 256 115800 8 29644800
7 256 115811 29 29647616
7 256 115823 17 29650688
7 256 115833 7 29653248
7 256 120196 28 30770176
7 256 120216 8 30775296
7 256 120218 6 30775808
7 256 124445 3 31857920
7 256 124455 25 31860480
7 256 124466 14 31863296
7 256 128771 29 32965376
7 256 128782 18 32968192
7 256 128793 7 32971008
7 256 128804 28 32973824
7 256 128816 16 32976896
7 256 133883 5 34274048
7 256 133894 26 34276864
7 256 133905 15 34279680
7 256 133916 4 34282496
7 256 133928 24 34285568
7 256 139175 25 35628800
7 256 141031 25 36103936
7 256 141042 14 36106752
7 256 141053 3 36109568
7 256 141064 24 36112384
7 256 143756 20 36801536
7 256 143767 9 36804352
7 256 143778 30 36807168
7 256 148359 25 37979904
7 256 148370 14 37982720
As you can see, the 256 byte buckets dropped dramatically when he started
Oracle. The DLM memory usage stats looked like this:
DRD = 61504 DLM = 264064 CNX = 12928
DRD = 61504 DLM = 264064 CNX = 12928
DRD = 61504 DLM = 266944 CNX = 12928
DRD = 61504 DLM = 266944 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 275008 CNX = 12928
DRD = 61504 DLM = 277312 CNX = 12928
DRD = 61504 DLM = 5632320 CNX = 12928
DRD = 61504 DLM = 5632320 CNX = 12928
DRD = 61504 DLM = 5632320 CNX = 12928
DRD = 61504 DLM = 5632320 CNX = 12928
DRD = 61504 DLM = 5632320 CNX = 12928
DRD = 61504 DLM = 9170368 CNX = 12928
DRD = 61504 DLM = 9170368 CNX = 12928
DRD = 61504 DLM = 9170368 CNX = 12928
DRD = 61504 DLM = 9170368 CNX = 12928
DRD = 61504 DLM = 9170368 CNX = 12928
DRD = 61504 DLM = 15904704 CNX = 13392
DRD = 61504 DLM = 15904704 CNX = 13392
DRD = 61504 DLM = 15904704 CNX = 13392
DRD = 61504 DLM = 15904704 CNX = 13392
DRD = 61504 DLM = 15904704 CNX = 13392
DRD = 61504 DLM = 19930816 CNX = 13392
DRD = 61504 DLM = 19930816 CNX = 13392
DRD = 61504 DLM = 19930816 CNX = 13392
DRD = 61504 DLM = 19930816 CNX = 13392
DRD = 61504 DLM = 19930816 CNX = 13392
DRD = 61504 DLM = 23385344 CNX = 12928
DRD = 61504 DLM = 23385344 CNX = 12928
DRD = 61504 DLM = 23385344 CNX = 12928
DRD = 61504 DLM = 26236096 CNX = 13712
DRD = 61504 DLM = 26236096 CNX = 13712
DRD = 61504 DLM = 26236096 CNX = 13712
DRD = 61504 DLM = 32585152 CNX = 12928
DRD = 61504 DLM = 32585152 CNX = 12928
DRD = 61504 DLM = 32585152 CNX = 12928
DRD = 61504 DLM = 32585152 CNX = 12928
DRD = 61504 DLM = 32585152 CNX = 12928
DRD = 61504 DLM = 36251776 CNX = 12928
DRD = 61504 DLM = 36251776 CNX = 12928
DRD = 61504 DLM = 36251776 CNX = 12928
DRD = 61504 DLM = 36251776 CNX = 12928
DRD = 61504 DLM = 36251776 CNX = 12928
DRD = 61504 DLM = 42654848 CNX = 12928
DRD = 61504 DLM = 42654848 CNX = 12928
DRD = 61504 DLM = 42654848 CNX = 12928
DRD = 61504 DLM = 42654848 CNX = 12928
DRD = 61504 DLM = 42654848 CNX = 12928
DRD = 61504 DLM = 42654848 CNX = 12928
DRD = 61504 DLM = 44958592 CNX = 12928
DRD = 61504 DLM = 44958592 CNX = 12928
DRD = 61504 DLM = 44958592 CNX = 12928
DRD = 61504 DLM = 47741952 CNX = 13392
DRD = 61504 DLM = 47741952 CNX = 13392
DRD = 61504 DLM = 47741952 CNX = 13392
DRD = 61504 DLM = 53416704 CNX = 13248
DRD = 61504 DLM = 53416704 CNX = 13248
DRD = 61504 DLM = 53416704 CNX = 13248
DRD = 61504 DLM = 53416704 CNX = 13248
DRD = 61504 DLM = 53416704 CNX = 13248
DRD = 61504 DLM = 53416704 CNX = 13248
DRD = 61504 DLM = 55964928 CNX = 12928
DRD = 61504 DLM = 55964928 CNX = 12928
DRD = 61504 DLM = 55964928 CNX = 12928
DRD = 61504 DLM = 58734912 CNX = 13392
DRD = 61504 DLM = 58734912 CNX = 13392
DRD = 61504 DLM = 58734912 CNX = 13392
DRD = 61504 DLM = 61253952 CNX = 12928
DRD = 61504 DLM = 61253952 CNX = 12928
DRD = 61504 DLM = 64848256 CNX = 12928
DRD = 61504 DLM = 64848256 CNX = 12928
DRD = 61504 DLM = 64848256 CNX = 12928
DRD = 61504 DLM = 64848256 CNX = 12928
DRD = 61504 DLM = 67160768 CNX = 13376
DRD = 61504 DLM = 67160768 CNX = 13376
DRD = 61504 DLM = 67160768 CNX = 13376
DRD = 61504 DLM = 67160768 CNX = 13376
DRD = 61504 DLM = 67160768 CNX = 13376
DRD = 61504 DLM = 69733056 CNX = 12928
DRD = 61504 DLM = 69733056 CNX = 12928
DRD = 61504 DLM = 69733056 CNX = 12928
DRD = 61504 DLM = 72226432 CNX = 13248
DRD = 61504 DLM = 72226432 CNX = 13248
DRD = 61504 DLM = 72226432 CNX = 13248
DRD = 61504 DLM = 74773504 CNX = 12928
DRD = 61504 DLM = 74773504 CNX = 12928
DRD = 61504 DLM = 74773504 CNX = 12928
DRD = 61504 DLM = 74773504 CNX = 12928
DRD = 61504 DLM = 74773504 CNX = 12928
DRD = 61504 DLM = 77770304 CNX = 12928
DRD = 61504 DLM = 77770304 CNX = 12928
DRD = 61504 DLM = 77770304 CNX = 12928
DRD = 61504 DLM = 77770304 CNX = 12928
DRD = 61504 DLM = 77770304 CNX = 12928
DRD = 61504 DLM = 80868160 CNX = 13392
DRD = 61504 DLM = 81956032 CNX = 12928
DRD = 61504 DLM = 81956032 CNX = 12928
DRD = 61504 DLM = 81956032 CNX = 12928
DRD = 61504 DLM = 81956032 CNX = 12928
DRD = 61504 DLM = 83549440 CNX = 12928
DRD = 61504 DLM = 83549440 CNX = 12928
DRD = 61504 DLM = 83549440 CNX = 12928
DRD = 61504 DLM = 86255488 CNX = 12928
DRD = 61504 DLM = 86255488 CNX = 12928
free pages looked like this:
47021 free pages
46935 free pages
46718 free pages
46645 free pages
45355 free pages
44413 free pages
44115 free pages
43419 free pages
42828 free pages
42238 free pages
40331 free pages
40219 free pages
39518 free pages
38805 free pages
38114 free pages
37470 free pages
36827 free pages
36181 free pages
35435 free pages
34926 free pages
34279 free pages
33639 free pages
33000 free pages
32368 free pages
31600 free pages
31072 free pages
30423 free pages
29788 free pages
29273 free pages
26501 free pages
25771 free pages
25771 free pages
25124 free pages
25124 free pages
24363 free pages
23761 free pages
22729 free pages
22247 free pages
21425 free pages
21424 free pages
20663 free pages
20269 free pages
19522 free pages
18969 free pages
18200 free pages
17686 free pages
17063 free pages
16431 free pages
15648 free pages
15139 free pages
14391 free pages
13881 free pages
13748 free pages
13089 free pages
13089 free pages
13089 free pages
13088 free pages
13069 free pages
12680 free pages
12680 free pages
12611 free pages
12611 free pages
12611 free pages
11838 free pages
11838 free pages
11773 free pages
11773 free pages
11773 free pages
11368 free pages
11368 free pages
11368 free pages
11367 free pages
11252 free pages
10838 free pages
10838 free pages
10795 free pages
10521 free pages
10521 free pages
10423 free pages
9651 free pages
9651 free pages
9651 free pages
9651 free pages
9650 free pages
9227 free pages
9227 free pages
9158 free pages
9158 free pages
9158 free pages
8386 free pages
8386 free pages
8386 free pages
8360 free pages
8343 free pages
8343 free pages
8083 free pages
8083 free pages
8083 free pages
7825 free pages
7825 free pages
7824 free pages
7165 free pages
7037 free pages
7037 free pages
6985 free pages
6985 free pages
6985 free pages
6724 free pages
6724 free pages
6708 free pages
6321 free pages
6321 free pages
6321 free pages
6063 free pages
6062 free pages
5547 free pages
5547 free pages
5531 free pages
5532 free pages
5272 free pages
5272 free pages
5272 free pages
5272 free pages
5232 free pages
4972 free pages
4952 free pages
4890 free pages
4632 free pages
4631 free pages
4578 free pages
4320 free pages
4320 free pages
4260 free pages
4260 free pages
4244 free pages
3986 free pages
3985 free pages
3982 free pages
3982 free pages
3863 free pages
3605 free pages
3349 free pages
3332 free pages
3332 free pages
3331 free pages
3200 free pages
3200 free pages
3200 free pages
2813 free pages
2813 free pages
NOTE: the above three outputs are synchronized in time.
I'm having him run a script that displays 'sysconfig -q' in a loop
while he start up Oracle. I'll post those results when I have them.
Does anyone have any ideas?
John McDonald
Atlanta CSC
[Posted by WWW Notes gateway]
| T.R | Title | User | Personal
Name | Date | Lines |
|---|
| 2042.1 | More data | NNTPD::"[email protected]" | John McDonald | Mon May 05 1997 11:40 | 124 |
| I had the customer run 'sysconfig -q dlm' in a loop while trying
to start up Oracle. I've extracted some of the relevant areas
(Note: these are synchronized in time):
dlm_lkbs_allocated = 0
dlm_lkbs_allocated = 0
dlm_lkbs_allocated = 0
dlm_lkbs_allocated = 0
dlm_lkbs_allocated = 0
dlm_lkbs_allocated = 0
dlm_lkbs_allocated = 5
dlm_lkbs_allocated = 19
dlm_lkbs_allocated = 5728
dlm_lkbs_allocated = 27666
dlm_lkbs_allocated = 48620
dlm_lkbs_allocated = 67857
dlm_lkbs_allocated = 86994
dlm_lkbs_allocated = 106215
dlm_lkbs_allocated = 124195
dlm_lkbs_allocated = 142056
dlm_lkbs_allocated = 159734
dlm_lkbs_allocated = 174663
dlm_lkbs_allocated = 188184
dlm_lkbs_allocated = 203107
dlm_lkbs_allocated = 216866
dlm_lkbs_allocated = 227577
dlm_lkbs_allocated = 227577
dlm_lkbs_allocated = 227577
dlm_lkbs_allocated = 227577
dlm_lkbs_allocated = 227577
dlm_lkbs_allocated = 206315
dlm_lkbs_allocated = 177990
dlm_lkbs_allocated = 149067
dlm_lkbs_allocated = 119031
dlm_lkbs_allocated = 88510
dlm_lkbs_allocated = 57330
dlm_lkbs_allocated = 25584
dlm_lkbs_allocated = 0
dlm_lkbs_allocated = 0
dlm_lkbs_allocated = 0
dlm_tot_lkids = 8191
dlm_tot_lkids = 8191
dlm_tot_lkids = 8191
dlm_tot_lkids = 8191
dlm_tot_lkids = 8191
dlm_tot_lkids = 8191
dlm_tot_lkids = 8191
dlm_tot_lkids = 8191
dlm_tot_lkids = 8191
dlm_tot_lkids = 28159
dlm_tot_lkids = 48639
dlm_tot_lkids = 68095
dlm_tot_lkids = 87039
dlm_tot_lkids = 106495
dlm_tot_lkids = 124415
dlm_tot_lkids = 142335
dlm_tot_lkids = 159743
dlm_tot_lkids = 175103
dlm_tot_lkids = 188415
dlm_tot_lkids = 203263
dlm_tot_lkids = 217087
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_tot_lkids = 227839
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 0
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
dlm_no_msg_mem = 1
It's interesting that when DLm hits its largest number of locks (227K,
about 21 lines down) is when the dlm_no_msg_mem count goes to 1. It
would appear that DLM is running out of resources, specifically 256
bytes buckets. Is there any way to increase them?
John McDonald
Atlanta CSC
[Posted by WWW Notes gateway]
|
| 2042.2 | out of curiosity | ALFAM7::GOSEJACOB | | Mon May 05 1997 12:50 | 8 |
| What is your 'gc_db_locks' init.ora parameter set to?
I was just wondering how many DLM locks you are trying to allocate upon
Oracle startup. BTW if 'gc_db_locks' is not set Oracle will allocate
1 lock per database block in the SGA. What happens when you set
'gc_db_locks' to something like 100?
Martin
|
| 2042.3 | I'll give it a try | NNTPD::"[email protected]" | John McDonald | Mon May 05 1997 12:56 | 9 |
| Martin,
Thanx for the reply. I'll have the customer try that and see what
happens.
John McDonald
999634
[Posted by WWW Notes gateway]
|