Conference decwet::networker

575.0. "Problems cloning and recovering big indexes" by EEMELI::TAKKUNEN (Takku von Wei�bier) Mon Apr 14 1997 05:07

Problems with Networker V4.2A indexes and cloning of them.

Customer:
  - Nokia Telecommunications, Finland
  - [email protected]

Configuration:
  - Networker version V4.2a with patches BRXP1SOAKIT421 and BRXP3SOAKIT421
  - Digital UNIX V4.0a with ping patch
  - AlphaServer 2100 with 2 CPU:s, TL812 (4*TZ88), 2*TZ887, 1*TZ88
  - 21 clients - NT and HP-UX


PROBLEM DESCRIPTITION

All clients are in "Full clone group". Full backup is done once per month
and group has automatic cloning feature enabled:
  - backup of data (around 300 Gbytes full backup) and cloning works ok
  - backup of indexes seems to work ok
  - cloning of the indexes fails always

PROBLEM ANALYSIS

There are problems with the indexes of two clients (elvis and ncbsr06ou)
whose index-sizes are big - elvis >6 Gb and ncsbsr06ou >4 Gb. Looks like
if the index size grows over 2 Gt the cloning fails. We are afraid that
recovering these indexes would also fail (we have not tested it).

Are there patches available?

MORE INFORMATION

Next some information gathered with different commands.

  nwadmin -> Indexes
  - elvis      6543 Mb 80%
  - ncsbsr06ou 4275 Mb 64%

  nwadmin -> Save-set -> Recover
  - aslak (server)
    /nsr/index/elvis      full 10.12.96 3976 Mb
                          full 22.12.96  692 Mb
                          full 04.04.97  348 Mb
    /nsr/index/ncsbsr06ou full 23.11.96 3187 Mb
                          full 23.12.96 3525 Mb
                          full 25.01.97 3214 Mb
                          full 04.04.97 2421 Mb

  # mminfo -v -c aslak -q "name=/nsr/index/elvis,level=full" -t 11/01/96 
  NOTICE - actual tape and clone tape
 volume        client       date     time      size         ssid fl  lvl name
nisse.0151     aslak      11/23/96  9:56:44 2000 MB        25932 cr full
/nsr/index/elvis 3404Mb (save recover)
nisse.0152     aslak      11/28/96  8:24:33 2000 MB        26541 cr full
/nsr/index/elvis 3415Mb (save recover)
nisse.0165     aslak      12/10/96  8:39:46 18014398509 GB 28470 cr full
/nsr/index/elvis
nisse.0212     aslak      12/22/96 21:37:55  692 MB        40754 cb full
/nsr/index/elvis
nisse.0279     aslak       1/25/97  5:45:40 1064 MB        90115 cb full
/nsr/index/elvis
nisse.0358     aslak       2/22/97  5:57:40 1377 MB        95662 cb full
/nsr/index/elvis
nisse.0426     aslak       3/28/97  6:37:58  864 MB       102358 cb full
/nsr/index/elvis
nisse.0431     aslak       4/01/97  8:23:22  865 MB     102700 cb full
/nsr/index/elvis
nisse_c.036    aslak      11/23/96  9:56:44 2000 MB      25932 cr full
/nsr/index/elvis
nisse_c.049    aslak      12/22/96 21:37:55  692 MB      40754 cb full
/nsr/index/elvis
nisse_c.068    aslak       1/25/97  5:45:40 1064 MB      90115 cb full
/nsr/index/elvis
nisse_c.085    aslak       2/22/97  5:57:40 1377 MB      95662 cb full
/nsr/index/elvis
nisse_c.106    aslak       3/28/97  6:37:58  864 MB     102358 cb full
/nsr/index/elvis
nisse_c.110    aslak       4/01/97  8:23:22  865 MB     102700 cb full
/nsr/index/elvis

  # mminfo -v -c aslak -q "name=/nsr/index/ncsbsr06ou,level=full" -t 11/01/96   
               
  NOTICE - actual tape and clone tape
 volume        client       date     time      size          ssid fl  lvl name
nisse.0149     aslak      11/23/96 15:14:08 2000 MB         25944 cr full
/nsr/index/ncsbsr06ou
nisse.0152     aslak      11/28/96  9:56:41 2000 MB         26552 cr full
/nsr/index/ncsbsr06ou
nisse.0210     aslak      12/23/96  6:14:22 18014398509 GB  41129 cb full
/nsr/index/ncsbsr06ou
nisse.0282     aslak       1/25/97 16:39:01 18014398509 GB  90148 cb full
/nsr/index/ncsbsr06ou
nisse.0357     aslak       2/22/97 14:48:13 18014398509 GB  95685 cb full
/nsr/index/ncsbsr06ou
nisse.0426     aslak       3/28/97 15:22:47 18014398508 GB 102380 cb full
/nsr/index/ncsbsr06ou
nisse.0430     aslak       4/01/97  8:25:57 18014398508 GB 102701 cb full
/nsr/index/ncsbsr06ou
nisse_c.037    aslak      11/23/96 15:14:08 2000 MB         25944 cr full
/nsr/index/ncsbsr06ou
nisse_c.050    aslak      12/23/96  6:14:22 18014398509 GB  41129 cb full
/nsr/index/ncsbsr06ou
nisse_c.069    aslak       1/25/97 16:39:01 18014398509 GB  90148 cb full
/nsr/index/ncsbsr06ou
nisse_c.088    aslak       2/22/97 14:48:13 18014398509 GB  95685 cb full
/nsr/index/ncsbsr06ou
nisse_c.105    aslak       3/28/97 15:22:47 18014398508 GB 102380 cb full
/nsr/index/ncsbsr06ou
nisse_c.110    aslak       4/01/97  8:25:57 18014398508 GB 102701 cb full
/nsr/index/ncsbsr06ou

  When doing backup with command
  savegrp -O -v -l full -c elvis -c ncsbsr06ou -c aslak "Full clone"
  daemon.log contains following:
 4/04/97  9:05:56 nsrd: savegroup info: starting Full clone (with 3 clients)
 4/04/97  9:05:56 nsrd: savegroup notice: Full clone completed, 3 clients (All
Succeeded)
 4/04/97  9:06:17 nsrd: savegroup info: starting Full clone (with 3 clients)
 4/04/97  9:06:17 nsrd: savegroup info: Full clone running on elvis, ncsbsr06ou
 4/04/97  9:06:29 nsrd: aslak:/nsr/index/ncsbsr06ou saving to pool 'Default'
(testi)
 4/04/97  9:06:29 nsrd: aslak:/nsr/index/elvis saving to pool 'Default' (testi)
 4/04/97  9:07:04 nsrd: index notice: nsrim has finished checking the media db
 4/04/97  9:11:18 nsrd: savegroup info: Full clone running on elvis, ncsbsr06ou
 4/04/97  9:16:18 nsrd: savegroup info: Full clone running on elvis, ncsbsr06ou
 4/04/97  9:21:19 nsrd: savegroup info: Full clone running on elvis, ncsbsr06ou
 4/04/97  9:26:21 nsrd: savegroup info: Full clone running on elvis, ncsbsr06ou
 4/04/97  9:31:22 nsrd: savegroup info: Full clone running on elvis, ncsbsr06ou
 4/04/97  9:34:37 nsrd: aslak:/nsr/index/ncsbsr06ou continuing save to pool
'Default' (testi) 2417 MB
 4/04/97  9:34:37 nsrd: aslak:<1>/nsr/index/ncsbsr06ou saving to pool 'Default'
(testi)
 4/04/97  9:34:46 nsrd: aslak:<1>/nsr/index/ncsbsr06ou done saving to pool
'Default' (testi)
 4/04/97  9:34:52 nsrd: savegroup info: Full clone running on elvis
 4/04/97  9:39:53 nsrd: savegroup info: Full clone running on elvis
 4/04/97  9:42:54 nsrd: aslak:/nsr/index/elvis continuing save to pool 'Default'
(testi) 4529 MB
 4/04/97  9:42:54 nsrd: aslak:<1>/nsr/index/elvis saving to pool 'Default'
(testi)
 4/04/97  9:42:59 nsrd: aslak:<1>/nsr/index/elvis done saving to pool 'Default'
(testi)
 4/04/97  9:43:06 nsrd: savegroup info: Full clone running on aslak
 4/04/97  9:43:08 nsrd: aslak:/nsr/index/aslak saving to pool 'Default' (testi)
 4/04/97  9:43:12 nsrd: aslak:/nsr/index/aslak done saving to pool 'Default'
(testi)
 4/04/97  9:43:17 nsrd: savegroup info: Full clone running on aslak
 4/04/97  9:43:17 nsrd: aslak:bootstrap saving to pool 'Default' (testi)
 4/04/97  9:43:18 nsrmmdbd: media db is savings its data, this may take a while.
 4/04/97  9:44:01 nsrmmdbd: media db is open for business.
 4/04/97  9:44:21 nsrd: aslak:bootstrap done saving to pool 'Default' (testi) 16
MB
 4/04/97  9:44:32 nsrd: media info event: tz87 tape testi will be needed
 4/04/97  9:44:32 nsrd: media waiting event: waiting for tz87 tape testi
 4/04/97  9:45:05 nsrd: media event cleared: confirmed mount of testi on
/dev/nrmt4h
 4/04/97  9:47:38 nsrd: aslak:cloning session saving to pool 'Default Clone'
(nisse_c.111)
 4/04/97  9:49:30 nsrck: checking index for aslak
 4/04/97  9:49:30 nsrck: cross-checking index for aslak
 4/04/97  9:49:30 nsrd: index notice: cross-checking index for aslak
 4/04/97  9:49:40 nsrck: completed checking 1 client
 4/04/97  9:49:40 nsrd: index notice: completed checking 1 client
 4/04/97  9:54:32 nsrd: cloning session:1 of 6 save sets reading from testi 348
MB of 18446744073708 GB
 4/04/97 10:33:14 nsrd: media info: can not read record 12500 of file 13 on tz87
tape testi
 4/04/97 10:34:29 nsrmmd #3: MM_CLONEEND w/active saves
 4/04/97 10:34:30 nsrd: cloning session:save sets done reading 18446744073708 GB
 4/04/97 10:34:30 nsrd: aslak:cloning session done saving to pool 'Default
Clone' (nisse_c.111)
 4/04/97 10:34:30 nsrd: savegroup notice: Full clone completed, 3 clients (Save
Set Cloning Failed)
 4/04/97 10:35:23 nsrd: index notice: nsrim has finished checking the media db
 4/04/97 10:39:32 nsrd: aslak:/nsr/cores saving to pool 'Default' (testi)
 4/04/97 10:39:40 nsrd: aslak:/nsr/cores done saving to pool 'Default' (testi)
 4/04/97 10:40:20 nsrd: /dev/nrmt4h unmounted testi
 4/04/97 10:44:48 nsrck: checking index for aslak
 4/04/97 10:44:48 nsrck: cross-checking index for aslak
 4/04/97 10:44:48 nsrd: index notice: cross-checking index for aslak
 4/04/97 10:44:52 nsrck: completed checking 1 client
 4/04/97 10:44:52 nsrd: index notice: completed checking 1 client