Conference cookie::aim$notes:disk_defrag

Menicacci

  I am not sure what the question is but I will try an address the
  following question:

  Why is there so much difference in the time it takes to
  defragment a disk with lots of small files that are deleted and
  created during the day?

  A full log file might have been better but the brief file shows
  that the time was spent in free space consolidation.  I cannot
  tell if any files were defragmented.  I see one file that had 30
  extents by the report.  However the report was done 15 hours
  after the defragmentation process ran.  The problem is not in
  defragmenting files.  After defragmenting files the process
  finds all of the files that cannot be moved and I will call
  these fences.  Between each fence is a consolidation area (note
  that _DSA7 had 56 consolidation areas).  DFO tries to make
  the largest free space it can.  DFO finds all of the potential
  consolidation areas from the smallest to the largestest.  Then
  DFO moves files from the largerest areas to the smaller areas
  to free up larger areas and fill in smaller areas.
  DFO then slides files down to make the largest free space in
  the largest consolidation area (Note that
  disk$user7 has a 2503485 block free space).  This free space
  keeps files from fragmented when creating a large file.  These
  two disks do not have problems with fragmented files.

  The reason that DFO takes so much time to consolidate free
  space is because the files are so small.  I do not know this
  for sure I can only go by what you say.  It would be nice if I
  added a report that told the average size of the files like I
  have for free space.  When DFO moves a file, it does about 15
  direct I/Os.  It moves the file, updates INDEXF.SYS, and
  updates BITMAP.SYS.  If DFO moves a 10,000 block file it might
  only take 15 I/Os.  So you can see if the disk has lots of
  small files the number of I/Os is going to be large.  Notice
  that the times are porportional to the number of I/Os.  As the
  I/Os go up the time goes up.

  Now that I have said all this, I would like to prove my point.
  I have been known to be wrong.  Could you do the following:

  1.  Get a defragmentation report before each defragmentation
  process. You can accomplish this with the prologue file:

  $ DEFRAGMENT SHOW disk:/VOLUME/OUT=VOL.BEFORE/HISTORY
  

  2.  Defragment the disk with /FULL/LOG= on it. (Not /brief as
  you have here)

  3.  Get a defragmentation report after each defragmentation.
  You can accomplish this with the epilogue file:

  $ DEFRAGMENT SHOW disk:/VOLUME/OUT=VOL.AFTER/HISTORY

  This will show the Total free extents before and after.  If you
  have lots of free extents, the free space is very fragmented,
  it will take a lot of I/Os to consolidate the free space.  The
  /full for the log file will show you something like

  Begin free space consolidation:                      10-MAR-1997 00:09:20.45
  There are 56 consolidation areas
  Beginning consolidation pass on area 56, LBN 634468 (268 blocks)
  Beginning consolidation pass on area 55, LBN 409384 (272 blocks)

  This means that it had 56 areas and that each area had to be
  consolidated.

  If you get something like:

  Begin free space consolidation:                      10-MAR-1997 00:09:20.45
  There are 56 consolidation areas
  Beginning consolidation pass on area 20, LBN 634468 (268 blocks)
  Beginning consolidation pass on area 19, LBN 409384 (272 blocks)

  You still had the same number of consolidation areas but some
  have already be packed from previous defragmentation processes
  and do not need to be done.  DFO starts with area 20 and
  works to area 0.  Area 0 is the largest consolidation area.

  Now you should have the before, log, and after file.
  If you can do this for a couple of nights so we can see if the
  state of the disk is making the difference in times.  Then post
  the files here.

  Does the above address the question or is there something else you
  want me to address?

  thanks
     jim amend  
    

	Jim,

thanks very much for your attention and mail.

I will give you the reports you requested.

I noticed that the defragmentation time was proportional to the number of 
IO's done. But I was not able to explain why this number of IO's was so 
different that's why I requested your help.

The wish of the customer is to understand why such differences, and then try to
reduce these times (if possible) cause all defragmentation processes must be 
finished during the night. 
 
Have a nice day.

Maria.

Maria

  I am not sure your customer is going to be able to reduce the
  time.  It is because of the deletion and creation of small
  files.  However,  If these disks have only small files, s/he
  might not need to defragment these disks that much.  S/he is
  not going to have that much fragmentation.

  Another company has mail routers like that and they cannot get
  done in a reasonable amount of time so they just defragment the
  disk without free space consolidation (/NOCONSOLIDATE).

  DFO consolidates freespace so that when new files are created,
  they have less chance of being fragmented.  However, in this
  case where the majority of these files are small and have a
  short life, it is not going to matter that much if there is 
  large contiguous free space. 

  Your customer might think about this and may just defragment
  once in a while.  If s/he only has the two disks, s/he might
  only defragment on both disks each night and consolidate
  on one disk each night or maybe on the weekend.  If it is not
  obvious how to do this, I have some ideas.

  thanks
    jim amend

	Hi Jim,

You will find on dior15(61.515)::reuters.bck, a saveset containing prologue,
log (full), epilogue files for DSA7 and DSA8 (both RZ29, cluster size 9).
(.avant means before, .apres means after).

Could you have a look just to be sure that there is not another problem and
that these elapsed times are not surprising you in that special case 
(small files being created all day long) ? 

In summary,

DSA7 (RZ29)

	        	03/14/97	03/17/97	03/18/97

Elapsed time    :       02:09:01.93	05:29:22.12	05:16:50.35
Direct I/O      :       96152		359169		393536
Total free 
extent (before) :       393		1273		559
Total free
extent (after)  :       38		34		27
# of conso area :       92		28              177
first area      :       48              18		145



DSA8 (RZ29)

	        	03/14/97	03/17/97	03/18/97

Elapsed time    :       00:42:59.85	05:47:01.11	04:41:03.11
Direct I/O      :       30054		336477		332635
Total free 
extent (before) :       400		1170		842
Total free
extent (after)  :       3		22		42
# of conso area :       22		22              78
first area      :       20		18		58



Customer has understood that the time spent by defrag is in consolidating
areas. 

I spoke with a DECmailworks specialist and he gave me some hints to have best
performance (basically, rebuilding a1mail.dir directories and convert/fdl
mail.x4m files).

I there is nothing we can do to have shorter elapsed times, I'll ask him not 
to use /CONSOLIDATE every night but every week.

Thanks for your input,

Maria.

The following is one complete before, log, and after report.

                   F r a g m e n t a t i o n    R e p o r t

DISK$USER8                                               14-MAR-1997 01:01:08.53

The fragmentation index is 1.8
      1 - 20.9 is excellent
     21 - 40.9 is good
     41 - 60.9 is fair
     61 - 80.9 is poor
     81 - 100 indicates a badly fragmented disk
Approximately 1.6 (out of 80.0 possible) is due to file fragmentation
Approximately 0.2 (out of 20.0 possible) is due to freespace fragmentation


Freespace Summary:
	Total free space:       5923899 blocks
	Percentage free:             70 (rounded)
	Total free extents:           3
	Maximum free extent:    3572181 blocks, LBN: 617832
	Minimum free extent:     694575 blocks, LBN: 6638733
	Average free extent:    1974633 blocks
	Median free extent:     1657143 blocks


File Fragmentation Summary:
	Number of files (with some allocation):  107838
	Total file extents on the disk:          108044
	Average number of file extents per file: 1.001910
	Median number of file extents per file:  1

Most Fragmented File:
	[USER.E_DEMALHERBE.A1MAIL$]MAIL.X4M;1 (7 extents)


                   F r a g m e n t a t i o n    R e p o r t

DISK$USER8                                               16-MAR-1997 18:38:14.15

The fragmentation index is 23.4
      1 - 20.9 is excellent
     21 - 40.9 is good
     41 - 60.9 is fair
     61 - 80.9 is poor
     81 - 100 indicates a badly fragmented disk
Approximately 3.4 (out of 80.0 possible) is due to file fragmentation
Approximately 20.0 (out of 20.0 possible) is due to freespace fragmentation


Freespace Summary:
	Total free space:       5985513 blocks
	Percentage free:             71 (rounded)
	Total free extents:        1170
	Maximum free extent:    2965608 blocks, LBN: 1224405
	Minimum free extent:          9 blocks, LBN: 7891164
	Average free extent:       5115 blocks
	Median free extent:          18 blocks


File Fragmentation Summary:
	Number of files (with some allocation):  106033
	Total file extents on the disk:          106397
	Average number of file extents per file: 1.003433
	Median number of file extents per file:  1

Most Fragmented File:
	[USER.F_GALLIAERDE.A1MAIL$]170C4UCR.002;1 (53 extents)


           F i l e    F r a g m e n t a t i o n    H i s t o g r a m

Extent
Count
------
         |
 8 To 53 |  (7)
       7 |  (3)
       6 |  (2)
       5 |  (4)
       4 |  (20)
       3 |  (34)
       2 |  (86)
       1 | ********************* (105877)
         +-------------------------------
          Number of files with a given number of extents
          Each * corresponds to 5000 files



             V o l u m e    F r e e s p a c e    H i s t o g r a m

Freespace
Size
(LBNs)
---------
                   |
 487845 To 2965608 |  (4)
   7722 To   98010 | * (5)
   3753 To    7002 | * (5)
   1062 To    3384 | * (5)
    648 To     819 | * (5)
    441 To     594 | * (5)
    369 To     432 | * (7)
    297 To     351 | * (5)
    225 To     279 | * (5)
    189 To     216 | * (6)
               180 | * (5)
    153 To     162 | * (9)
               144 | * (6)
               135 | * (7)
               126 | * (12)
               117 | * (5)
               108 | * (13)
                99 | * (7)
                90 | *** (27)
                81 | * (14)
                72 | *** (32)
                63 | *** (33)
                54 | ***** (51)
                45 | ****** (60)
                36 | *************** (145)
                27 | ********** (101)
                18 | *************************************** (387)
                 9 | ******************** (204)
                   +----------------------------------------------
                     Number of freespace extents of a given size
                      Each * corresponds to 10 freespace extents



 
 
Disk File Optimizer for OpenVMS DFG V2.2
    Time:                                            16-MAR-1997 18:38:07.96
    Process ID:                                      0000041C
    Node:                                            REPRS2
    Run time parameters:
        Defragmentation level:                       2
        Defragment placed files:                     No
        Defragment indexed files:                    Yes
        Log file report:                             Full
        Notify user by mail:                         No
        Notify user by opcom:                        Yes
        Default priority:                            4
        Minimum priority:                            3
        Query hotfile database:                      No
        RVN to defragment:                           0
        Consolidate freespace:                       Yes
        Perform write check:                         No
 
 
 
Defragmentation process start
 
	Process ID: 0000041C
	Device: _DSA8:
	Time: 16-MAR-1997 18:38:10.27
 
PROLOGUE subprocess beginning....
 
$ defrag show dsa8:/vol/hist/out=dsa6:[logs]dsa8.avant
Disk File Optimizer for OpenVMS DFG V2.2
� 1996, Digital Equipment Corporation
 
    Accounting information
        Buffered I/O count:                          95
        Direct I/O count:                            1305
        Page faults:                                 30440
        Peak working set:                            13143
        Charged CPU time:                            0 00:00:36.66
        Elapsed time:                                0 00:02:59.72
        Final status:                                10000001
 
PROLOGUE subprocess complete
 
    Volume name:             _DSA8:
    Media name:              HSX00
    Cluster factor:          9
    Number of blocks:        8380008
 
Begin file evaluation phase:                         16-MAR-1997 18:41:13.06
 
    Candidates for defragmentation (based on the defragmentation level):
 
            DISK$USER8:[USER.F_GALLIAERDE.A1MAIL$]170C4UCR.002;1 (53 extents)
            DISK$USER8:[USER.P_LABORIE.A1MAIL$]AQ1M2UCR.004;1 (14 extents)
            DISK$USER8:[USER.P_LABORIE.A1MAIL$]AQ1M2UCR.002;1 (11 extents)
            DISK$USER8:[USER.B_MATHIS.A1MAIL$]MAIL.OLD;28 (10 extents)
            DISK$USER8:[USER.E_DEMALHERBE.A1MAIL$]MAIL.OLD;27 (9 extents)
            DISK$USER8:[USER.S_ROUSSOTTE.A1MAIL$]MAIL.OLD;27 (8 extents)
            DISK$USER8:[USER.F_ANNABI.A1MAIL$]MAIL.OLD;24 (8 extents)
 
End file evaluation phase:                           16-MAR-1997 18:43:47.54
 
    Statistics:
        Number of files evaluated:                   7
        Number of candidates for defragmentation:    7
            (Does not include dormant and frequent files)
        Number of frequent files in list:            0
        Number of frequent-file candidates:          0
        Number of dormant files in list:             0
        Number of dormant-file candidates:           0
 
        Candidates evaluated by category:
            Indexed files:                           439
            Files with fixed placement:              0
            Contiguous files:                        0
            Less than or equal to window turn:       0
            Less than or equal to one header:        7
            Greater than one header:                 0
 
Begin defragmentation preprocessing phase:           16-MAR-1997 18:43:48.85
 
    Files preprocessed for defragmentation:
 
    Symbols preceding a file name indicate the following:
 
        Blank   - file has been preprocessed for defragmentation
         d      - file has been defragmented
         *      - file has been partly defragmented
        D       - file has been placed for dormant access
        Dd      - file has been defragmented and placed dormant
        D*      - file has been placed dormant and been partly defragmented
        F       - file has been placed for frequent access
        Fd      - file has been defragmented and placed frequent
        F*      - file has been placed frequent and been partly defragmented
        Moved   - file has been moved (number of extents unchanged)
        EMPTY   - file was empty; no action taken
        FAIL    - file's fragmentation cannot be reduced
        IN USE  - new disk space for file was in use; no action taken
        OPEN    - file was open; no action taken
        NOMOVE  - file may not be moved; no action taken
        UNAVL   - file was unavailable; no action taken
 
     d      DISK$USER8:[USER.F_GALLIAERDE.A1MAIL$]170C4UCR.002;1 (1 extent)
     d      DISK$USER8:[USER.P_LABORIE.A1MAIL$]AQ1M2UCR.004;1 (1 extent)
     d      DISK$USER8:[USER.P_LABORIE.A1MAIL$]AQ1M2UCR.002;1 (1 extent)
     d      DISK$USER8:[USER.E_DEMALHERBE.A1MAIL$]MAIL.OLD;27 (1 extent)
     d      DISK$USER8:[USER.B_MATHIS.A1MAIL$]MAIL.OLD;28 (1 extent)
     d      DISK$USER8:[USER.F_ANNABI.A1MAIL$]MAIL.OLD;24 (1 extent)
     d      DISK$USER8:[USER.S_ROUSSOTTE.A1MAIL$]MAIL.OLD;27 (1 extent)
 
End file defragmentation preprocessing phase:        16-MAR-1997 18:44:04.86
 
    Statistics:
        Number of files preprocessed:                7
        Number of files defragmented:                7
        Number of files partly defragmented:         0
        Number of frequent files placed:             0
        Number of dormant files placed:              0
 
Begin free space consolidation:                      16-MAR-1997 18:44:05.75
There are 22 consolidation areas
Beginning consolidation pass on area 18, LBN 7750467 (22689 blocks)
Beginning consolidation pass on area 17, LBN 579195 (37206 blocks)
Beginning consolidation pass on area 16, LBN 7773723 (41634 blocks)
Beginning consolidation pass on area 15, LBN 8328834 (51174 blocks)
Beginning consolidation pass on area 14, LBN 6344406 (51372 blocks)
Beginning consolidation pass on area 13, LBN 8265051 (63000 blocks)
Beginning consolidation pass on area 12, LBN 6396894 (74817 blocks)
Beginning consolidation pass on area 11, LBN 1035 (94689 blocks)
Beginning consolidation pass on area 10, LBN 1127043 (97173 blocks)
Beginning consolidation pass on area 9, LBN 441882 (137304 blocks)
Beginning consolidation pass on area 8, LBN 95733 (154782 blocks)
Beginning consolidation pass on area 7, LBN 8096301 (168741 blocks)
    IN USE  DISK$USER8:[USER.F_GALLIAERDE.A1MAIL$]121740.OLD;6 (1 extent)
    IN USE  DISK$USER8:[USER.F_GALLIAERDE.A1MAIL$]0Q28PMOQ.001;1 (1 extent)
    IN USE  DISK$USER8:[USER.S_LEROY.A1MAIL$]290R4KCR.X4H;1 (1 extent)
Beginning consolidation pass on area 6, LBN 250524 (171954 blocks)
FID = (104650, 13, 0)  --  No such file!
%SYSTEM-W-NOSUCHFILE, no such file
Beginning consolidation pass on area 5, LBN 7816356 (278298 blocks)
Beginning consolidation pass on area 4, LBN 7375770 (374688 blocks)
Beginning consolidation pass on area 3, LBN 616968 (509832 blocks)
Beginning consolidation pass on area 2, LBN 6471720 (904041 blocks)
    IN USE  DISK$USER8:[USER.F_GALLIAERDE.A1MAIL$]170C4UCR.002;1 (1 extent)
    IN USE  DISK$USER8:[USER.B_WALET.A1MAIL$]CS0EUQCR.002;1 (1 extent)
Beginning consolidation pass on area 1, LBN 4352058 (1991673 blocks)
    IN USE  DISK$USER8:[USER.M_ELKESLASSY.A1MAIL$]8N0JCLCO.X4H;1 (1 extent)
    IN USE  DISK$USER8:[USER.J_MEADE.A1MAIL$]EI0A5MSQ.001;1 (1 extent)
    IN USE  DISK$USER8:[USER.Q_FAWCITT.A1MAIL$]5B08TP4Q.003;1 (1 extent)
    IN USE  DISK$USER8:[USER.F_GALLIAERDE.A1MAIL$]IG135M4R.X4H;1 (1 extent)
    IN USE  DISK$USER8:[USER.F_GALLIAERDE.A1MAIL$]II218TCO.002;1 (1 extent)
    IN USE  DISK$USER8:[USER.P_LEVAN.A1MAIL$]E40RACOP.002;1 (1 extent)
Beginning consolidation pass on area 0, LBN 1224270 (2965743 blocks)
 
End free space consolidation:                        17-MAR-1997 00:28:12.69
Completion data for volume:
    Time:                                            17-MAR-1997 00:28:12.87
 
    Statistics:
 
    Total number of files evaluated:                 7
    Total number of candidates found:                7
    Total number of files defragmented:              7
    Total number of files partly defragmented:       0
    Total number of files placed frequent:           0
    Total number of files placed dormant:            0
 
    Accounting information
 
        Buffered I/O count:                          60766
        Direct I/O count:                            336477
        Page faults:                                 48709
        Peak working set:                            15584
        Charged CPU time:                            01:37:54.27
        Elapsed time:                                0 05:47:01.11
EPILOGUE subprocess beginning....
 
$ defrag show dsa8:/vol/hist/out=dsa6:[logs]dsa8.apres
Disk File Optimizer for OpenVMS DFG V2.2
� 1996, Digital Equipment Corporation
 
    Accounting information
        Buffered I/O count:                          92
        Direct I/O count:                            1337
        Page faults:                                 30419
        Peak working set:                            13474
        Charged CPU time:                            0 00:00:36.49
        Elapsed time:                                0 00:01:33.32
        Final status:                                10000001
 
EPILOGUE subprocess complete
 
Defragmentation process normal completion
 
	Process ID: 0000041C
	Device: _DSA8:
	Time: 17-MAR-1997 00:29:48.35
 



                   F r a g m e n t a t i o n    R e p o r t

DISK$USER8                                               17-MAR-1997 00:28:17.52

The fragmentation index is 2.6
      1 - 20.9 is excellent
     21 - 40.9 is good
     41 - 60.9 is fair
     61 - 80.9 is poor
     81 - 100 indicates a badly fragmented disk
Approximately 1.5 (out of 80.0 possible) is due to file fragmentation
Approximately 1.1 (out of 20.0 possible) is due to freespace fragmentation


Freespace Summary:
	Total free space:       5985432 blocks
	Percentage free:             71 (rounded)
	Total free extents:          22
	Maximum free extent:    2958138 blocks, LBN: 1231875
	Minimum free extent:         36 blocks, LBN: 1224270
	Average free extent:     272065 blocks
	Median free extent:       31176 blocks


File Fragmentation Summary:
	Number of files (with some allocation):  106067
	Total file extents on the disk:          106249
	Average number of file extents per file: 1.001716
	Median number of file extents per file:  1

Most Fragmented File:
	[USER.F_GALLIAERDE.A1MAIL$]MAIL.X4M;1 (7 extents)


           F i l e    F r a g m e n t a t i o n    H i s t o g r a m

Extent
Count
------
       |
     7 |  (1)
     6 |  (1)
     5 |  (4)
     4 |  (9)
     3 |  (27)
     2 |  (74)
     1 | ********************* (105951)
       +-------------------------------
        Number of files with a given number of extents
        Each * corresponds to 5000 files



             V o l u m e    F r e e s p a c e    H i s t o g r a m

Freespace
Size
(LBNs)
---------
                   |
           2958138 | * (1)
           1657152 | * (1)
            735156 | * (1)
            131913 | * (1)
            127692 | * (1)
            102087 | * (1)
             59904 | * (1)
             53019 | * (1)
             51993 | * (1)
             38646 | * (1)
             31176 | * (1)
             11502 | * (1)
              9090 | * (1)
              7479 | * (1)
              5121 | * (1)
              1404 | * (1)
              1206 | * (1)
              1053 | * (1)
               963 | * (1)
               630 | * (1)
                72 | * (1)
                36 | * (1)
                   +------
                    Number of freespace extents of a given size
                    Each * corresponds to 1 freespace extent

Maria

  The information you sent me looks pretty convincing that the
  time spend in consolidation is because of the small files.  On
  disks that do not have much activity, you will see that after a
  couple of days of consolidation, the beginning cosolidation
  pass with start a lot lower than the number of consolidation
  areas:

  There are 400 consolidation area
  Beginning consolidation pass on are 22

  This means that 388 areas have been packed and we do not have
  pack files into these areas.  I noticed on yours that it was

  There are 78 consolidation area
  Beginning consolidation pass on are 58

  There were only 20 areas that did not need files moved.


  Because the files are small, there is very little file
  fragmentation from day to day.  You don't have to worry about
  file fragmentation.

  However, one thing that I noticed was a large number of page faults:

	For show command:
        Page faults:                                 30440
	  and took 3 minutes to complete
    
	For defragment vol command:
	Page faults:                                 48709

	For show command:
        Page faults:                                 30419
	  and took 1.33 mintes to complete (not too bad)

  I think that this large amount of page faults is caused by the
  large number of files on the disk and a small working set.  I
  would like to see if we could decrease the pages faults and
  possible time by increasing the processes WSDEFAULT, WSEXTENT,
  WSQUOTA.  When a defragmentation process is started, it looks
  at the three quotas,

  A hard coded value of
    WSDEFAULT       512
    WSEXTENT        16000
    WSQUOTA         2048

  Or PQL sysgen quotas
    PQL_DWSDEFAULT
    PQL_MWSDEFAULT
    PQL_DWSQUOTA                  
    PQL_MWSQUOTA                 
    PQL_DWSEXTENT               
    PQL_MWSEXTENT               

  or the following logicals

  DFG$WSDEFAULT       
  DFG$WSEXTENT        
  DFG$WSQUOTA         

  Which ever quota is the largest, the defragmentor process sets
  its quotas to that value.

  The DFG$ logicals are set in SYS$STARTUP:DFG$STARTUP.COM.

  You will see that they are commented out.

  If you uncomment them and increase these quotas and then
  execute DFG$STARTUP.COM it will define these logicals for the
  the defragmentation processes that start up.  You do not need
  to stop the schedulers before executing DFG$STARTUP.COM.  It
  will say that the scheduler is already running.

  thanks
   jim amend