Conference vaxaxp::vmsnotes

    RE:  Dismounting shadow set members for back-up
    
    	I suggest you review some of the notes in VMSNOTES_V12 regarding
    the back-up scheme described in .0.  Your customer is not getting
    what they expect, i.e., usable, complete image back-up save sets.  
    
    RE: Vast difference in back-up between systems
    
    	Is DECps (or a similar performance monitoring tool) installed
    on these systems?  In my experience, DECps is excellent at helping
    a system manager to identify bottlenecks, SYSGEN parameters impac-
    ting performance, etc.  As a shot in the dark, I would make sure 
    that the drives are directly mounted on both nodes and not being 
    MSCP-served from one node to the other.  
    
    Greg
    

>      - Disk I/O queue lengths during backups are around 85-90

That's your problem. You've given backup such enormous quota's that it
will stuff the devices with as much i/o's as it's quota's allow. And then
comes your lonely application which wants to do a couple of i/o's which
are queued behind the i/o's of backup. Guess why you have to wait so long for
completion. Backup needs only the quota's to keep all devices busy at all
times for maximum performance, but creating queues on a device will not make
it spin any faster.

Check your backup quota's.

Jur.

    re .1
    
    Thanks for the suggestions. I'm especially concerned about your
    warning that the backups may be incomplete and/or unusable.  I had
    spent several hours reading dozens of notes from both VMSNOTES_v11
    and VMSNOTES_V12 related to "backup" and "shadow" before I entered
    the base note.  I got the impression that the customer was following
    the generally recommended procedure.  I'll go back and reread them
    with your warning in mind.  Do you have any pointers handy?
    
    We have DECPS running on both nodes. I've spent several hours pouring
    over reports.  The alphas are "loafing" at night;  CPU is almost idle;
    page faults are 3-5/sec during the hour of peak performance
    degradation; only 20% of memory is being used throughout the night;
    I/O rates to the disks are 26-28 I/O's per second; 
    
    Both alphas are directly connected to the drives, i.e. SHOW DEV DUA600
    show a host name of HSJ108 on both Alphas. (These drives happen to be
    MSCP served by one of the Alphas, but they are not being used through
    the MSCP server.)
    
    
    re .2
    
    I appreciate your observation, but I'm having a difficult time
    understanding it in light of the observed symptoms.  For example:
    
    	1) A backup during the day caused *no* performance spikes in the
    	   test procedure.
    
    	2) nightly backups do not cause performance spikes when the disk
    	   is on a different HSJ.  For example, I believe there have been
    	   two simultaneous backups going on on HSJ003 with no pathological
    	   performance noticed. (I'm going to revisit this to reconfirm.)
    	   
    If your observation were correct, I would expect a pathological
    slowdown during the backups from other HSJ's, or during my daytime
    backup from HSJ108.  However, this did not occur.  What am I missing?
    
    
    ---------------------------------
    
    In general there is a performance slowdown during backups.  My test
    time for file copy takes 2-3 times longer.  This is understandable,
    and is not the cause of my concern.
    
    However, I am worried when the copy takes 40-80 times longer.  This
    is especially worrisome when the slowdown occurs on a disk that is
    on HSJ108, but is *not* being backed up.
    
    NEW CLUE #1:  The problem moves to the CPU that is doing the backups.
    The evidence for this is anecdotal, but is going to be rigorously
    confirmed within the next few days.  In fact, the problem was initially
    observed on both Alphas when backups were running on both Alphas, and
    backing up disks from HSJ108 at the same time.  (Since then, the
    backups from HSJ108 have been completely serialized, and run from 
    a single Alpha.)
    
    NEW CLUE #2:  The problem affects the "SHOW DEVICE" command.  Normally
    the SHOW DEVICE command takes 2-3 seconds to complete.  However, 
    during the period of pathological performance this command can take
    20-30 seconds to complete.  (There has been anecdotal evidence that
    other commands are similarly affected, e.g. SHOW MEMORY, but I have
    not collected statistics on it.)
    
    A common thread is that the Alpha system disk(s) are also on HSJ108.
    We plan to move them off of HSJ108 on Feb. 9.  However, the I/O rate
    to the system disk averaged about 3-4/sec during the 2 hours of 
    pathological performance.  Once the backups got rolling it was
    hardly accessed.  It seems that the system disk might be a victim,
    but certainly not the cause of this situation.

    
:    Thanks for the suggestions. I'm especially concerned about your
:    warning that the backups may be incomplete and/or unusable. 

   Search for "interlock" via COMET notes searches.

   Splitting the shadowset has the same problems as the more typical
   (mis)use of the BACKUP/IGNORE=INTERLOCK qualifier -- splitting the
   shadowset has a rather shorter window when all applications must
   be quiescent, but there is still a window...

:    ...I got the impression that the customer was following
:    the generally recommended procedure.  I'll go back and reread them
:    with your warning in mind.  Do you have any pointers handy?

   Nope, what the customer is doing risks data corruption.

   It may or may not be a big risk (depending on what the applications
   are doing, and how the applications write to the disk(s)), but we
   should inform the customer of the potential risks -- we should
   certainly not tell the customer there are no risks.  (We're going
   to get the telephone support call when things go bad, after all.)

   If the applications can reliably recover from system crashes, it
   is possible that the applications may also correctly recover from
   the restoration of the split-shadowset backup...

   I'd still recommend shutting down the applications, and/or using
   on-line tools such as the Rdb RMU/BACKUP tool -- otherwise, it's
   possible the applications won't recover...

For the disks in question what is the response time during the problem
period?  You may need to DEFINE PSPA$GIVE_DEVICE_SERVICE TRUE in order to
see this.  Or you can graph the D_RESPONSE_TIME item for the disks.  This
will allow you to see if the disks themselves are behaving similarly.

Jilly

>    the base note.  I got the impression that the customer was following
>    the generally recommended procedure.  I'll go back and reread them
>    with your warning in mind.

The recommended procedure is: 1) Shut down application's use of the shadow
set(s).  2) DISMOUNT/NOUNLOAD shadowsets.  3) Remount shadowsets with one fewer
members. 4) Restart application.  5) Back up from the extra member. 6) Add it
back to the shadowset.

Before starting this, be sure all members are full members and no copies or
merges are pending or underway.

-Mike

 re: references which document the inconsistency of removed shadow members

  Volume Shadowing for OpenVMS

  Sec 4.6.1:

   "When you dismount an individual shadow set member, all
   outstanding I/O operations are completed and the member is
   removed from the set. However, note that files that are open
   on the shadow set virtual unit remain open because access
   to the shadow set continues. Because of this, file integrity on
   the removed physical unit cannot be guaranteed. Therefore,
   the disk is marked as being improperly dismounted so that
   a rebuild is initiated the next time the disk is mounted. To
   guarantee file integrity, you must dismount a virtual unit to
   dissolve the entire shadow set (described in  Section 4.6.2). See
   Section 6.1 for important information about the data consis-
   tency issues that result if you dismount an individual shadow
   set member."

 Just in case you missed it: "FILE INTEGRITY ON THE REMOVED PHYSICAL
 UNIT CANNOT BE GUARANTEED".

  Sec 6.1:

   "When you dismount an individual shadow set member,
   you produce a situation similar to a hardware disk failure.
   Because files remain open on the virtual unit, the removed
   physical unit is marked as  not being properly dismounted."

 
  						John Gillings, Sydney CSC

  (4th attempt to post this, for some reason cut/paste from bookreader
  doesn't get copied into the note until I delete leading whitespace -
  embedded nulls perhaps?)

    re .6
    
    And John would usually put in his note about adding a member first
    rather than just removing a member so that you are not running degraded
    (possibly running without shadowing protection at all) during the
    BACKUP.

[Posted by WWW Notes gateway]

Thanks to all for your suggestions. 

The customer is running DSM on OpenVMS.  They had originally been
shutting down DSM before doing the backups.   A few months ago
they were told by their Digital reseller that by doing a DSM "VOLINH"
command they would be able to safely split the shadow set and do
a backup.  (VOLINH means Volume Inhibit.)  The rest is history.

I apologize for omitting the VOLINH step of the customer procedure.
Based on that step, the customer believes that DSM has flushed all 
cache to disk, and that data consistency is not an issue.  I had 
started searching the DSM notes file to confirm/deny this belief, but
haven't gotten back to it yet.  (Does anyone in this conference
know about VOLINH?) 

Even if VOLINH works as the customer believes, there still seems
to be procedural problems with the way they manage the shadow set
during the backiup procedures (based on previous replies).

I will share this information with the customer and urge that they
reconsider their procedure.

----------------------------------------------


Over the weekend the backup procedure was moved to the other ALpha
to demonstrate that the problem followed the backups.  The results
were inconclusive;  the shape of the performance curve changed
significantly, suggesting there may be other factors involved.
I am still analyzing the results.

---------------

re .5

Thank you; I will try your suggestion.

  
[Posted by WWW Notes gateway]

    re .5
    
    Using TOP_REPONSE_TIME_DISK shows 3400ms respone time (3.4 seconds)
    during the backup to each disk as it is being backed up.  For example,
    the response time for DUA330 jumps from 0 to 3.5 seconds while it is
    being backed up, then it drops back to 0.  Next the response time
    for DUA410 jumps to 3.4 seconds while it is being backed up, then
    it drops back to zero when the backup completes.  This pattern
    continues for the next 4 drives.  (All 6 drives are on HSJ108)
    
    In the meantime my test procedure is doing a VMS copy to DUA440 every
    5 minutes.  The response time for DUA440 averages about 500ms during this
    four hour interval, with a peak at 2 seconds.  (This drive is also
    on HSJ108.)
    
    --------------------------
    
    I've finished analyzing the test results from this past weekend.
    There are 2 clear results:
    
    	1) The performance problem is related to the node which is
    	   running the backups.  There are 2 Alphas: Saturn & Viper.
    	   When the backups are run on Viper, the long disk response time
    	   occurs on viper (but *not* on Saturn).  When the backup
    	   runs on Saturn, then the long response times occur on Saturn,
    	   (but *not* on Viper).  This confirms earlier anecdotal evidence	
    	   that when the backups were run on both Saturn and Viper
    	   concurrently, that both Alphas had performance degradation.
    
    	2) The problem is not specifically an HSJ bottleneck, or a SCSI
    	   bandwidth issue, or a CI bandwidth issue.  I draw this
    	   conclusion as another aspect of #1 above.   When the backups
    	   are running on Viper, the TOP_RESPONSE_TIME_DISK is .5 - 2
    	   seconds on Viper, but only .06 seconds on Saturn.  This is
    	   basically the same results that my test procedure showed.
    
    	   So, while one machine is having terrible performance accessing
    	   a disk on HSJ108, the other machine is having normal performance
    	   accessing that same disk.
    
    	   -------------------------
    
    
    What is really puzzling is that we were able to do a backup during
    the day from DUA330 with no apparent affect on my test procedure.
    I'm going to rerun it to make sure I was looking at the "right"
    machine.
    
    Any other suggestions?
    
    Thanks,
    
    Jerry

Are you running the backup from different accounts? Or from batch maybe?
I still wonder what the backup account's quota's are.

Jur.

    
    I wonder if you are getting credit stall problems during the
    backup? Check the connection under SDA while you are running the
    backup... eample:
    sho conn/node=rabbit
    
    Connections via Port PAA0 to Node RABBIT
    ----------------------------------------
    
              --- Connection Descriptor Table (CDT) 81F706B0 ---
    
    State:  0002 open                 Local Process:       VMS$DISK_CL_DRVR
    Blocked State:  0000              Remote Node::Process:RABBIT::MSCP$DISK
    
    Local Con. ID   C0EA0005    Datagrams sent         0    Message queue     empty
    Remote Con. ID  47EC0002    Datagrams rcvd         0    Send Credit Q.    empty
    Receive Credit        10    Datagram discard       0    PB address     81F91980
    Send Credit           33    Messages Sent   12891183    PDT address    81F86B90
    Min. Rec. Credit      10    Messages Rcvd.  12891238    Error Notify   8244828C
    Pend Rec. Credit       0    Send Data Init.        0    Receive Buffer 81F9B2A0
    Initial Rec. Credit   10    Req Data Init.         0    Connect Data   82446CB1
    Rem. Sta.   000000000001    Bytes Sent             0    Aux. Structure 81F91480
    Rej/Disconn Reason     0    Bytes rcvd             0
    Queued for BDT         0    Total bytes map ********
    Queued Send Credit  ****
    
    
    
	How many send credits do you have? Do the disks being backed up
    have a non-zero RWAITCNT in thier UCBs... Under sda do a show device
    DUAxxxx,
    
    	What is the setting of, and I'm not sure of the exact name,
    but it's something like MAX_HOSTS on your HSJ. Lowering this toward
    the actual number of SYSTEMS on the CI can increase the number of
    credits you extend to each host. 
    
    KEvin
    
    

    re .15
    
    	The backups are done sequentially by a single command procedure.
    The procedure is submitted as a batch job from the OPER_BACKUP account.
    The OPER_BACKUP quotas are:
    
    	Maxjobs:         0  Fillm:      1000  Bytlm:      2500000
    	Maxacctjobs:     0  Shrfillm:      0  Pbytlm:           0
    	Maxdetach:       0  BIOlm:      1000  JTquota:      10240
    	Prclm:           2  DIOlm:     32700  WSdef:         1536
    	Prio:            4  ASTlm:     32750  WSquo:        22000
    	Queprio:         0  TQElm:        20  WSextent:     22000
    	CPU:        (none)  Enqlm:      1500  Pgflquo:     200000
    
    
    ----------------------
    
    re .16
    
    I started thinking along the same lines yesterday. I ran a procedure
    at 10 minute intervals last night to collect credit info (using
    SHOW CLUSTER).  There are 116 send credits available, but during the
    night, while the backups were running, the send credits dropped to 0
    a couple times for hsj108.
    
    I noticed that the send credits on HSJ107 (tape drives only) dropped
    to 0 shortly after the backups started.  Then they bounced around
    hitting zero several more times during the backup period.
    
    I observed that the tape HSJ send credits were zero more often than
    were the disk HSJ send credits.  I alos observed that the disk HSJ
    send credits tended to be unused, i.e. 115, when the tape HSJ send
    credits dropped to zero.
    
    Maybe we have a tape bottleneck?  There are other backups going on 
    from disks on other HSJ's to the tape drives on HSJ106/HSJ107.
    Hmmmmmmmm.  The DEC PS I/O rates and throughput didn't seem out of
    line when I looked at this a couple weeks ago, but maybe I better
    revisit it.  Any other things to consider from a tape point of
    view???  This might explain why there were no problems when I
    did a single backup (disk to tape) during the day....I was the
    only one using the tape drives!!
    
    
    
    RWAITCNT=0 during the day when backups are not running.  Is this
    a static or a dynamic value?
    
    MAX_HOSTS=7 on all HSJs.  There are currently 3 hosts on the CI.
    (may be 4 in the near future.)
    
    
    The tape configuration is:
    	
    	MUA1:	TKZ60	HSJ106/HSJ107
    	MUA2:	TKZ07	HSJ106/HSJ107
    	MUA3:	TZ87	HSJ106/HSJ107
    	MUA4:	TZ87	       HSJ107
    	MUA5:	TZ87	HSJ106/HSJ107
    	MUA6:	TZ87	       HSJ107
    
    In other words, 4 of the tape drives have a host name and an alternate
    host name (in SHOW DEV).  But two of the drives do NOT have and
    alternate host name.
    
    
    

With these backup quota's you end up queueing LOTS of i/o requests each waiting 
after each other, using all resources. Normal i/o's will be queued behind it, 
that explains your delays.

>    RWAITCNT=0 during the day when backups are not running.  Is this
>    a static or a dynamic value?

It's dynamic. This will be non-zero if requests are stalled, for example 
mountverifications or if you run out of credits.

Jur.

    re .18
    
    Thanks.  I lowered DIOLM  for the OPER_BACKUP account.  This
    was a SWAG based on an article from DSNlink titled "Setting up
    Parameters for the VMS BACKUP Operation".  It recommended that the
    minimum value for DIOLM should be the larger of (3*FILLM) or 4096.
    On our system this would be the value 4096.   However, I reasoned
    that 4096 outstanding Direct I/O's would still dominate the I/O queue,
    so I chose to use the value 50 for the new DIOLM.  (I am assuming
    that DIOLM is the quota that you had in mind also.)
    
    Using a value of 50 for DIOLM caused a slight improvement in the 
    performance problem.  However, I would still characterize it as
    pathologically poor.  For example, during the 4-5 hour backup period
    there were four samples that had a response time greater than 100
    seconds, when DIOLM was 32700.  When the DIOLM was lowered to 50
    there were still 4 sample that had a response time greater than 100
    seconds.  
    
    The response time is the time to do a VMS copy of a file to a disk
    on HSJ108.  The target of the copy is *not* being backed up, and 
    has no other I/O to it.  Normally this VMS copy takes 4-5 seconds.
    The disk that is being backed up is also on HSJ108.  
    The peak response time dropped from 190 seconds to 152 seconds,
    which is a (small) step in the right direction.  (Of course, the
    backup time increased from 4 hours to 5 1/4 hours.)
    
    -------------------------
    
    Questions:
    
    1) Is DIOLM=50 still too high (even though it is way below the number
       recommended in the DSNlink article)?
    
    2) What value of DIOlM do other folks use for backups?
    
    3) Is there anything else that should be investigated?
    
    
    Thanks,
    
    Jerry
    
    

    The backup DIOLM was lowered to 10 and the performance picture has
    improved some more.  However, it still seems unbelievably slow.  The
    copy time for my test procedure now averages only 50 seconds with peaks
    at 94 seconds.  ( The normal copy time is only 5-6 seconds when the
    backups are not running.)
    
    As soon as the last backup completes, the response time drops from 60
    seconds to 12 seconds.  At this point there are 6 shadow copies going on
    concurrently on HSJ108.  This seems reasonable.  
    After about 45 minutes the response time
    drops to 6 seconds with 4 shadow copies going on concurrently on
    HSJ108.  After another 1 1/2 hours the shadow copies complete, and
    the response time drops back to its "normal" time of 4 seconds.
    
    Conclusions
    -----------
    
    The combination of BACKUP/IMAGE plus 2 or more shadow copies on the
    same HSJ40 can cause performance to degrade by a factor of 10-15 times.
    This can even effect DCL commands such as "SHOW DEVICE DUA" which  
    can take 8-10 times longer than normal while the backups are going on.
    
    
    QUESTIONS
    ---------
    
    1) Is it "normal" for a single backup plus a few shadow copies to 
       degrade performance by a factor of 10-15 on VMS 6.2 accessing
       HSJ40's over the CI?
    
    2) Where's the bottleneck?
    
    3) Is there any reason to believe the performance would improve
       significantly by going to :
    
    	a) HSJ50?
    	b) VMS 7.0
        c) Some other backup quota
    
    
    
    
    Thanks,
    Jerry

    Based on the principle that if all you have is a hammer everything
    looks like a nail, if all you are is a filesystem engineer.....
    
    What is the `file characteristic' of the system - a single, huge
    coniguous file, or a gazillion small ones ?  fragmentation an issue (I
    wouldn't image so given the non-symetric performance).
    
    What sort of filesystem cache performance are you getting ?  This can
    slow down backup (and any other FS-type operations including sh dev).
    In particular make sure that you don't have the filesystem caches set
    to their minimum ("Max buffers in FCP cache" from show dev)
                                                               

Based on the symptom that IO to an otherwise idle (low utilization
anyway) disk is so heavily impacted by the backup to a separate disk,
my guess is some kind of resource exhaustion. Possibly a bottleneck.
IF (and this is a big IF) the CI is saturated 

    re .18
    
    The disks that are being backed up contain 1 to 4 large user files.
    
    Fragmentation is not "significant" on these volumes as the files
    were preallocated (and as you observed, the performance is not
    symetric).
    
    The Each of the 6 disks being backed up has a "maximum buffers in
    FCP cache" of 3618.  Is this high or low?  How can one alter this 
    value?
    
    Thanks,
    
    Jerry

    re .22
    
    Yes, a resource exhaustion seems to fit the evidence, or possibly
    some strange interaction between shadow copies and BACKUP.
    
    However, the evidence does not support hardware bandwidth bottlenecks.  
    For example,  the performance of the VMS COPY does *not* degrade 
    (pathologically) when the target disk is on a different HSJ on the
    same CI.  This seems to rule out CI bandwidth concerns, as well as
    confirm the DECPS reports that show the Alpha has more than enough
    horsepower to accomplish both the BACKUP and the COPY simultaneously.
    (This is an Alpha 8200, 4 CPU).
    
    Another example, the performance does not degrade when the VMS COPY is done
    to HSJ108 from a different CPU which is on the same CI.  This would
    seem to rule out CI bottleneck, HSJ bottleneck, SCSI bottleneck.
    
    I'm still interested in the FCP cache possibility mentioned in an
    earlier note.
    
    Thanks to all for your continued interest in this puzzle.
    
    Jerry

    
    > The disks that are being backed up contain 1 to 4 large user files.
    
    That pretty much rules out the filesystem.  Back to the drawing board...
    
>    The Each of the 6 disks being backed up has a "maximum buffers in
>    FCP cache" of 3618.  Is this high or low?  How can one alter this 
>    value?
    
     reasonable figure.  FWIW these are set by the ACP_*CACHE parameters
    and by rebooting (to set the default cache) or MOUNT/PROC=UNIQUE
    
    but you'll not see any benefit from these figures.
    
    	.rod

    
    You mentioned Shadowsets are being copied... Check to see if the
    sets are doing DCD copies. This is possible if both the source and
    copy target are on the same controller. I don't have a disk handy
    that is copying.. but if you use SDA and do a SHOW DEV DSAxxxx. On the
    second screen, look at the status bits for the individual members.
    If you are doing a DCD you will see either DCD or NULL in the status.
    Which you see depends on the version of OpenVMS and SDA that you have.

    If you are doing DCDs this could use a significant amount of resources
    in the HSJ. You can fail one of the Shadowset members to a different
    controller this will eliminate the DCD and see how that works.

    Kevin

In my observations, the backup utility is pretty bursty on large files.  Backup
apparently does well enough when there are zillions of little files or scattered
disk extents.  It reads some of INDEXF.SYS and then basically sorts into LBN
order.  It issues a ton of disk reads and then prepares the buffers for movement
to tape.  It seems like it does work sort of asynchronously in that it begins
the movement to tape whilst the disk reads are still pending.  But, I get the
distinct feeling that eventually it exhausts the sorted LBN's and has to pause
to gather more from INDEXF.SYS before continuing?

I suppose we forgive it when there's all this work for little files or numerous
extends to deal with but take exception when it is a small number of huge mostly
contiguous files.

I guess I don't think the burstiness of backup is the fundamental source of the
performance problem.  Reducing the DIOLM effectively restrained backup from
consuming the insufficent resource.  This is further supported by the evidence
of improving performance as the shadow copies retire.

The HSJ extends credits to each node equally independent of load.  This is why
the other node is able to make good progress during the strainful moments.

I'm thinking the flow control is bursty and that is the real source of your
pain.  Credits are used up and a queue builds, naturally.  Work gets done and
new credits are issued.  But these new credits show up at the waiting node more
or less all at once.  Sure enough a portion of the queue runs and the credits
are rapidly exhausted again.  But if only a few of the new credits could have
been moved to the node earlier it would have been much better.  Perhaps the new
credits are batched?

Sometimes flow control algorithms will include some form of congestion control. 
Perhaps the system detecting the exhausted credits delibrately slows the
delivery of work in an effort to avoid congestion?  Are any packets being lost
altogether?  Are any being retransmitted redundantly because the acks are being
delayed?