Conference decwet::networker

    Hi
    
    I'm hard at work on automating our cloning process.  So hard at work,
    in fact, that I feel I'm needing to do too much to make this work.  So
    here are the details.
    
    At our site, our business needs determine that we save data for 1, 2 7,
    10 and 20 years.  We've decided that the best way to implement this is
    to create a separate clone pool for each retention period.  We'll
    assign a range of labeled media to each pool.  This will make it easy
    for our media coordinator to determine the retention for the media; the
    label range will tell the tale.  Also, by segregating our cloned data
    into media pools for each retention period, we won't need to perform
    saveset consolidations (when savesets with earlier retentions expire)
    to reclaim wasted space on the media.  By the by, we will only clone
    full savesets, to send these offsite for disaster recovery.
    
    In addition, we perform our backups during the night.  All tape drives
    are busy taking the data from the network; none are available for
    cloning as the data is copied.  So, our clones will be done during the
    day, which is the 'offshift' for backups.  So, we can't take advantage
    of NetWorker current implementation of automatic backups.
    
    Finally, each system on site has been assigned a media retention
    period; so, all data saved from a client has a known retention.  Thus,
    we could assign a clone media pool to each client.
    
    So here come the problems:
    
    1. clone media pools are assigned to groups, not clients.  Creating our
    groups by retention period, rather than other factors, is not desirable
    for us.
    
    2. I'm finding it very hard to find command line verbs that provide the
    functionality that I seek.  The logic flow that I THINK is good is
    this:
    
    Start unattended clone job:
    Determine date/time previous successful clone job ran (we'll scoop up
    all the backups done since then)
    {to keep the tape mount/dismount/movement to a minimum, perform the
    cloning by pool, cloning all clone pool clients to the same media at
    the same time, then moving on to the next clone pool}
    Get the list of clone pools.
    For each clone pool:
     Get the list of clients in the pool
     Get the list of full savesets for these clients since the last
      successful clone job
     Clone these savesets to the current clone pool
    If it's Wednesday (the day we send our clone tapes offsite), then
    export all of the clone tapes that have been written to from the TL822
    and mark their location to be 'offsite'
    End
    
    My chief problem at the moment is that clients don't have clone pools;
    groups have clone pools.  But I'm lucky in that each of our clients is
    in two groups: one group starts all our backup jobs at once, and is run
    nightly; the other is a one-client group, which is run when we need to
    redo the backups for a single client.  I guess we could add the clone
    pool information to this 'client group', and I could use mminfo?? to
    pull the information from there.
    
    But {whine whine} this is hard work, and sure wish NetWorker was (as it
    does in every other way) make this a easy process.
    
    Please comment on all of this: our business process, a clone pool
    decision, my logic flow.  I'm very interested in your thoughts.
    
    Thanks!
    tl

yeah... perhaps forwarding this to Legato as a feature request could 
 bear fruit downstream...

re: the script. Looks proper to me.

    Thanks for the feedback.  I don't expect immediate resolution.  My gut
    tells me this is a non-trivial implementation.  I'd be very happy to
    have Legato chew on this, and implement it someday down the road.
    
    Till then, I'd take any suggestions (gratefully) on commands to
    accomplish the steps I've listed.  Creating a list of only the clone
    pools, for instance, baffles me.
    
    Thanks!
    tl

    Sorry to whine about this.  How can I create a list of the clone pools,
    using a command level program?
    
    Thanks!
    tl

    How about this idea:  rather than use a date/time to figure out which
    savesets are new, and therefore need to be cloned, why don't I examine
    the instance values for the savesets in our production backup pool; if
    the instance is 0, then I need to create a clone.
    
    Sound good?  Any problems?
    
    Thanks!
    tl

sounds simpler than date/time...

    A little wading yielded the 'copies' value in the mminfo command, which
    is "The number of copies (instances or clones of the save set, all with
    the save save time and save set identifier)".  I took a look at this
    value in my data and, sure enough, the only savesets with a value of
    '1' were the savesets I needed to clone; all cloned savesets had a
    value of 2.  So, if we only clone savesets with a copies value of '1'
    for savesets in our production backup pool (I don't want to clone
    savesets in our clone pools after the original savesets expire), then
    that should do it.
    
    I'll post the code I think will work in a while.
    
    tl

    The following code will do most of what I need:  For all savesets saved
    into our production backup pool that are fulls, have been created in
    the last 4 weeks and have not yet been cloned, clone them to a clone
    pool:
    
    nsrclone -b Offsite 1 Year -S 'mminfo -r ssid \
    -q 'pool=Robot1 Daily Backup, \
        savetime = last month, \ <- this may need work
        level=full, \
        copies = 1, \
        client =  {select clients whose clone pool is Offsite 1 Year}'
    
    My problem is that I can't figure out how to create this selection
    group of clients.  Any thoughts?
    
    Thanks!
    tl
    

use nsradmin in a script, and awk to rip out parts needed for those clients
 in the groups that have "Offsite 1 Year"

# nsradmin < EOF
 show name
 print type: nsr group; clone pool:Offsite 1 Year
 quit
EOF
# awk ' <rip off name: part of non-blank lines, print out space seperated>'
# for i in <list-of-groups>; do
# nsradmin < EOF
 show name
 print type:nsr client; group ${i}
 quit
EOF
# done
# <and clean up above output to get just client names...>

    Hi Bruce
    
    Thanks for much for these suggestions.  I'm not strong (yet) on writing
    UNIX command scripts; you gave me a valuable leg up.
    
    In the meantime, I've discovered that:
    
    # mminfo -r 'ssid' \
    -q 'pool=Robot1 Daily Backups,level=full,copies=1' \
    -t 'last month' -ocn
    
    will produce a list of the save set ids for the full backups saved in
    the last month to the Robot1 Daily Backups pool which have not yet been
    cloned, sorted in client/disk order.
    
    # mminfo -r 'ssid,client,name,volume,savetime' \
    -q 'pool=Robot1 Daily Backups,level=full,copies=1' \
    -t 'last month' -ocn | more
    
    is a good way to check that the basic command is doing the right thing
    with your data.
    
    This can be fed to the nsrclone command in two ways:
    
    o write an output file from mminfo with '>', and use this input file
    with:
    
    nsrclone -b Offsite clone -f {file of ssids}
    
    o feed the mminfo output right into nsrclone with the -S option, as in:
    
    nsrclone -b Offsite clone -S mminfo -r 'ssid' \
        -q 'pool=Robot1 Daily Backups,level=full,copies=1' \
        -t 'last month' -ocn
    
    This command right here would work fine for a site with one
    clone/offsite pool.  We are planning to have one offsite pool for each
    retention period.  And we'll communicate which clients use which pool
    to NetWorker by adding clients to the pool client list.  Even though we
    won't be using NetWorker's automatic cloning, we'll still be able to
    use the GUIs for input of this data, and all the standard NetWorker
    command line commands to get it after its input.
    
    
    Thanks again, Bruce.  Back into the code I go yo ho.
    
    tl

    Hi
    
    I've taken a long look at this:
    
    # nsradmin < EOF
     show name
     print type: nsr group; clone pool:Offsite 1 Year
     quit
    EOF
    # awk ' <rip off name: part of non-blank lines, print out space
    seperated>'
    # for i in <list-of-groups>; do
    # nsradmin < EOF
     show name
     print type:nsr client; group ${i}
     quit
    EOF
    # done
    
    and also asked some experienced UNIX folks in my group.  We're all
    confused.  I'm assuming that I wouldn't put the # symbols in front of
    the actual commands in the script file; else, these would become
    comments, right?
    
    Also, the function of EOF is unclear to me.  Suffice to say that it
    doesn't seem to work:
    # cat t
     nsradmin < EOF
         show name
         print type: nsr group; clone pool:Offsite 1 Year
         quit
        EOF
    # csh -x t
    nsradmin
    NetWorker administration program.
    Portions Copyright � Digital Equipment Corporation 1996. All rights
    reserved.
    
    Restricted Rights: Use, duplication, or disclosure by the U.S.
    Government is subject to restrictions as set forth in subparagraph
    (c)(1)(ii) of DFARS 252.227-7013, or in FAR 52.227-19, or in FAR
    52.227-14 Alt. III, as applicable.
    
    Portions of this software are proprietary to and embody the
    confidential
    technology of Digital Equipment Corporation.  Posession, use, or
    copying of this software and media is authorized only pursuant to a
    valid written license from Digital or an authorized sublicensor.
    
    Use the "help" command for help, "visual" for full-screen mode.
    nsradmin> show name
    show: unable to change directory to /usr/users/crsupp/Mail/inbox: No
    such file or directory
    print type: nsr group
    print: Command not found.
    clone pool:Offsite 1 Year
    clone: Command not found.
    quit
    quit: Command not found.
    EOF
    EOF: Permission denied.
    #
    
    I'm sure I'm missing something obvious.  Please help me on my way.
    
    Thanks!
    tl

	It should be:

nsradmin << EOF
show name
print type:nsr client; group${i}
quit
EOF

	The double less than "<< string" passes multiline commands until
	a line starting with 'string'.  The EOF should be eaten by the
	shell.

    Roger
    
    Ah ha!  Thanks for the tip.  Executing:
    
    nsradmin> # cat t
    nsradmin << EOF
    show name
    print type:NSR pool; pool type:Backup clone
    quit
    EOF
    #
    
    I now see:
    
    # csh -x t
    nsradmin
    NetWorker administration program.
    Portions Copyright � Digital Equipment Corporation 1996. All rights
    reserved.
    
    Restricted Rights: Use, duplication, or disclosure by the U.S.
    Government is subject to restrictions as set forth in subparagraph
    (c)(1)(ii) of DFARS 252.227-7013, or in FAR 52.227-19, or in FAR
    52.227-14 Alt. III, as applicable.
    
    Portions of this software are proprietary to and embody the
    confidential
    technology of Digital Equipment Corporation.  Posession, use, or
    copying of this software and media is authorized only pursuant to a
    valid written license from Digital or an authorized sublicensor.
    
    Use the "help" command for help, "visual" for full-screen mode.
    nsradmin> nsradmin>                         name: Default Clone;
    
                            name: Offsite Clone;
    nsradmin> #
    
    The two 'name:' lines are the expected output.  Why, though, aren't the
    nsradmin commands used to generate these displayed during the -x
    execution?  This will make debugging more difficult.
    
    Thanks!
    tl

for the record, the "#" I used was bourne shell root prompt (I think)
 anyhow, it was an indication that a shell prompt was there... 

 oh yeah, I guess it's actually "%", not "#" - dang.  sorry.

    I probably have almost 30 hours invested in this effort, with precious
    little to show for it.  I still can't believe that something so
    obviously necessary is so incredibly difficult.  This has been an
    enormous learning curve for me, spanning not only the archane
    NetWorker command line functions, but UNIX script writing as well.  I
    say this to, I hope, cause product management to consider the
    opportunity of providing this functionality.  I think most professional
    users of cloning would afree that the current method of cloning at the
    time the backups are created is silly; it's a horribly inefficient use
    of backup resources during the critical backup window.  Please make
    this a formal part of our NetWorker product, not the sack of stuff that
    I've written.  Thanks!
    
    Now that I've got that off my chest, here is my current script.
    
    My game plan is to:
    o create a separate pool for each retention period;
    o add clients to the appropriate pool;
    o run the clone script every day during first shift (when backups
    aren't running).
    
    The clone script will:
    
    o make sure backups ran correctly (No clue yet how)
    o one by one, take each active backup clone pool
    o for each backup clone pool, find the clients that use it
    o for each of these clients, find the uncloned full savesets
    o clone these uncloned savesets to tapes assigned to the backup pool
    for these clients
    
    This should minimize the thrashing of tapes into/out of the drives,
    which would happen if we just cloned in, say, alphabetical client
    order.  This also makes use of existing NetWorker data fields and
    programs to the greatest degree possible (I think).
    
    1. Get list of enabled backup clone pools
    nsradmin << EOF
    show name
    print type:NSR group; autostart:Enabled; pool type:Backup clone
    EOF
    2. Put these into a clone pool array
    awk /name:/ {
    beginning = index ($0, ":")
    str_length = length ($0)
    pool_name[x] = substr ($0, beginning+2, str_length-beginning-2)
    print pool_name
    }
    3. For each pool in this clone pool array:
      a.  Get the list of clients that use that pool
      nsradmin> show clients
      nsradmin> print type:nsr pool; name {clone pool array[n]}
      b.  Put these into a client array
      awk . . .
      c.  For each client, select the full savesets from {client array[n]}
      nsrclone -b {clone pool array [n]} -S# mminfo -r 'ssid' -q
    'client,level=full,copies=1' -t 'last month' -ocn
    
    A good command to test what will be cloned is:
    
    # mminfo -r 'ssid,pool,client,name,volume,savetime' -q
    'level=full,copies=1' -t 'last month' -ocn | more 
        
    Current problems:
    1. I can't figure out how to pass the output of the first nsradmin
    command into the awk program.
    2.  I can't figure out how to place the output of the awk program into
    an array that can be read by the second nsradmin program.
    
    Any ideas?  Are there better ways to achieve what I seek?
    
    Thanks!
    tl
    tl

    I just tried to add a client to a Backup clone pool, and got the error
    message:
    
    	'No selection criterion are allowed for clone pools.'
    
    Does this mean that I can't specify clients for clone pools?
    
    tl

I just tried to add client "mrcofe.zso.dec.com" to pool Default CLone,
 and as expected got "NSR Pool "Default Clone" may not be modified"

So then I created a new pool, set the Pool type to "backup Clone", and
 label template to "Default Clone", set the client field to "mrcofe.zso.dec.com"
 and APPLY - worked.

am I doing what you want done??

    Yep, that's what I wanted to do.  I'm using NetWorker V4.2B, by the
    way, with most of the current patches.  I did what you did:
    
    Using the DU NetWorker Administrator interface, I
    o brought up the Pools window
    o clicked on Create
    o specified 'Terry test' for Name
    o specified 'Backup clone' for 'pool type'
    o specified 'Default clone' for 'Label template'
    o entered 'iss015' [a valid client] in Clients, and clicked Add
    When I click Apply, I see a pop-up box containing:
    
    No selection for criterion are allowed for clone pools.
    
    If you'd like to try it on my system, say the word.
    
    Thanks for taking the time to respond.
    
    tl
    
    p.s. I've worked around this problem by:
    
    o creating clone pools, one for every retention period (1 year, 2 year,
    etc.)
    o creating groups with the same name as these clone pools; these groups
    do not autostart, and never run
    o each client now belongs to two groups:
    - an autostart backup groups that runs nightly
    - a never-start clone group that serves as a link between a client and
    a clone pool.
    
    My script (see 240.*) is almost ready, and this isn't a real mind
    stretch from being able to add clients to clone pools.
    

Terry --

  I can see only 1 difference: you used a short name, I used a FQDN (long
 name) - could be a nit.

  re: access your system -- sorry, getting DMOV2 new FT kit out by next monday
 prevents me from accepting.

PS: glad you have a "workaround"