Conference bulova::decw_jan-89_to_nov-90

re: < Note 645.0 by UKCSSE::LESLIE "Send lawyers, guns & money" >
                            -< Defragment and fly >-

So.  Uhh.... How do you do it?  I need it bad!

re :.1

If you have a local tape then just go get your handy standalone backup tape
and backup the disk /image and then restore it back /image and everything will
be very contiguous.

If you don't have a tape then I do this by booting into a LAVC and backup my
disk to a BIG disk and then restore it back, with the /image switch again.


Sean

    I used the second method in .2.

        re .4
        
        JUst don't come bleating to us if you do, and it doesn't work
        afterwards....
        
        
        q
        

    re .4,.5
    
    I know this isn't a Performance Conference, but is Rabbit-7 available
    internally (I.e have we got a Company license?!?)

        That doesn't mean that VMS sanctions or supports its use. The
        internal license was arranged by some people that obviously though
        they new more about VMS disk support than the file system
        developers....
        
        q

    RE: .0
    Andy,
    What percentage of the System Disk should be free ideally to really
    fly?
    I have a schyzophrenic II/GPX with etier DECwindows or VWS available
    on demand to cater for the people who still use VWS/Sight in EDU
    at EHQ. (We have an LCG01!) I have oredered a 3rd RD54 and I was
    going to used it for image backup/restore operations. I am just
    wondering if, depending how long the migration will take from UIS
    to DECW, of moving the VWS/HCUIS side of the game on the 3rd disk,
    to have more than 43'000-odd block on sys$sysdevice:
    What thinkest thou?
    Nick

Hi Nick! How's Geneva?

Anyhow, I run with 60k free but think that's largely irrelevant as what was 
slowing LESLIE:: down was probably physical frag mentation of my page file.

I run with 6mb and a 55000 block page file currently.

Andy

There's a nifty application floating around somewhere that runs under VMS and
displays a "picture" of the free space your disk. Really great for tracking 
fragmentation. It's called Bitmap or  Diskmap, I forget which. Perhaps someone 
else can provide a pointer.

-Kelly

DISKMAP is a product of Dave Porter (of NMAIL fame).  It was originally a VWS
display, called BITMAP (which explains Kelly's confusion over the name).  Dave
renamed it to DISKMAP a while after porting it to DECwindows, since there's
already a DECwindows BITMAP program.  It's easy for me to remember where to get
this fine product, since he named the directory after me (that's a joke):

	SMAUG::DAVES_BITSHOP:DISKMAP.EXE

It's not one of the more USEFUL applications, actually, since it doesn't account
for caching, etc. (I usually keep around 100,000 blocks free on my RD54 so I
can do a full backup on a wimpy little TK50, but DISKMAP always shows on the
order of 16,000 blocks free).  On the other hand, I still run it once in a while
just because it LOOKS neat.  It's got a flashy little icon, too!

	/dave (one of the other Daves)

I'm aware of the fact that DISKMAP doesn't account for cached blocks.

One workaround is to start a ANAL/DISK on the target disk before
running DISKMAP.  You can ^C almost immediately.   This apparently
causes a cache-flush to occur.

I've heard that I could reproduce the same effect by temporarily
opening the disk allocation bitmap file for write access within DISKMAP.
I haven't dared to try it yet!

    If you do an ANAL/DISK before running DISKMAP all your free blocks will
    show up.
    

    Have you tried REVIEW from the Toolshed - that gives you a good
    idea of the state of your disk.
    

	The current version of rabbit 7 is for pre vms-5.0.  The newer
	version v3.0+ is for vms5.0+ and is being tested now to be
	availiable in May

	Paul H.

       RE: .15
       
       
       	Don't go submitting any crashes from a system running one of these
       	"de-fraggers"..  VMS Development won't be very interested.  You're
       	mixing fire and gas..
       
       						mike
       					   VMS Development

    I agree with .16.  As we say in Indy, "Thats like mixing Mario Andretti
    and any driver under the age of 40".
    
    Dick

    
    Perhaps someone could ask Keith Walls whether a memo he wrote a while
    ago about the dangers of these things should be circulated (again ? - I
    can't remember whether I saw it in SYSNOTES or somewhere else...).
    
    Trevor

re .18
                IS this what you were talking about?
        
               <<< BULOVA::DOCD$:[NOTES$LIBRARY]VAX-RMS.NOTE;2 >>>
                            -< VAX RMS Conference >-
================================================================================
Note 300.1                 Disk/File de-fragmentation                    1 of 12
STAR::WALLS "Eppur si muove"                        115 lines  18-AUG-1987 23:22
                               -< Rushed reply. >-
--------------------------------------------------------------------------------
    In the UK RABBIT and DISKIT disk de-fragmentation tools now claim
    to run on-line and securely. This has two effects
    1	people ask why we can't
    2	there has been a publicity blitz and customers think fragmentation
    	is a problem not just a characteristic that needs monitoring.

Clarification:  People ask why we can't <what>.  Please fill in the blank.


Fragmentation is  not  a  problem unless it is immoderate. That is, the file
system is designed to cope very well with a certain degree of fragmentation.
The  times  that  fragmentation becomes a problem are when large chunks of a
file  must be reached, whether in random or sequential order. The definition
of  "large"  in  the  previous  sentence can only be defined in terms of the
state  of  fragmentation  of the disk. A good example is BACKUP/IMAGE, which
must  traverse  the every file on the volume (set). However, note that there
is  little  you  can  do to help a file that is large and accessed randomly.
Even  if it's contiguous, there is much head movement involved in getting to
its data.

    So does anyone know these products and if so can they really
    de-fragment a file whilst it is being accessed by application software?

Most of  the packages recommend that you do a BACKUP/IMAGE before you start.
I  am  not  familiar  with  the  packages  (and  don't want to be, for legal
reasons).  However, it is most unlikely that they can do it properly. Unless
they  provide  copious  patches  to the file system and to the exec. This is
because  these  parts of VMS hold in-memory records that are relevant to the
position of the file.

I have  been asked many (too many) times to restore a disk that one of these
packages had destroyed. Their shortcomings take numerous forms. Among these,
I have heard:

1) Fails to recognise open files.
2) Fails to recognise PAGEFILE.SYS, SWAPFILE.SYS, SYSDUMP.DMP.
3) Fails to recognise spool files, spooled files, INSTALLed files, etc.
4) Completely destroys a disk on ^Y, STOP/ID, crashes and powerfails.
5) Gives some people parts of other people's files.
6) Shortens files (makes a 200-block, _full_ file 20 blocks long).
7) Fragments free disk space worse than it was before.
8) Places  directory files so as to maximize head movement from them to the
   index file.
9) Cannot cope with volume sets or files split across volumes.

Please don't  let  our  customers  take  these  risks.  How  to  answer  the
questions:

1) Recommend  to  the  customer  that  (s)he take a BACKUP/IMAGE of the disk
before  they  run  the package, habitually. The astute customer will ask why
they  then  bother  with the package. Don't answer this question - it has no
answer!
2) Deliver the above list of things the customer should be wary of.
3)  Recommend  that the customer get the package on approval and then try it
on  scratch disks (BACKUP/PHYSICAL of real disks, __with activity__).
4)  Tell  the  customer  that  the  VMS file system is designed to cope with
a  certain  degree  of  fragmentation and that they should make some careful
measurement  on  what they hope to achieve, then measure the results against
the risks, the value of their data, the integrity of their system, the saved
time  (if  any).  [One  complaint  came to me that the package slowed things
down.]
5) Learn to read the output of MONITOR, and SPM. Use these tools to prove to
a  customer  that  there  is/is not benefit from defragmentation, or whether
it's a problem for that customer.
6)   Tell   the   customer   that   BACKUP/IMAGE   is  the  safest  form  of
defragmentation. Suggest that they do a BACKUP/IMAGE and restore and measure
the improved performance.
7) Help the customer determine whether fragmentation is really a problem for
their  site.  If  it is, look for a cheaper and more reliable solution. Like
migrating   files   to  different  disks,  monthly  backups  with  restores,
pre-extending  files,  setting  RMS defaults, etc. In other words, show that
you  are  more  concerned  with  the  performance and _reliability_ of their
system than the vendors of the de-fragmentation package.
8) Let it be known that we (Digital) understand that fragmentation can be a
problem.  However, also let them know that it is not as big a problem as the
people  wanting to make money from de-fragmentation packages would like them
to believe. Show them alternatives, educate them and let them make their own
decision.  (But tell them _not_ to call me with mangled disks!)
9)  Suggest  other alternatives (larger cluster size, larger caches, larger
extension defaults, etc).
10)  There  are  genuine  cases  where  our current offerings are inadequate.
BACKUP/IMAGE  just  doesn't  cut it under some circumstances. Make sure your
customer  isn't  one  of  these (rare) instances before declaring they don't
need it.

[I know I have left repetitions in there - please excuse the pressing nature
of V5]

    Can they defragment files if the disk does not contain a single
    big enough space to put it in??

Yes. But  unless  they  are using space on a spare disk, or are patching the
file system, or worse, the package cannot be robust to powerfail.

As an  aside,  I am currently designing a disk utility that will de-fragment
disks  and  solve  some  other  common  disk problems. At Nashville DECUS, I
mentioned  this  fact  in  "public".  I  mentioned  it  under the heading of
"absolutely  no  committment,  no time scale offered, no work necessarily in
progress,  these  represent  our current intentions and are designed only to
help  customers with very long-term planning" (etc). I see no reason why you
shouldn't  do  this, _BUT_, do not suggest that we will have the product out
before  any  problem  may  become  critical for them. Do not suggest that we
intend  put  the  competitors out of business. Simply use it to show that we
acknowledge  that  defragmentation  is  a problem and that we are addressing
that  problem. If anything, the statement that we are doing it should convey
two things:

1) We  recognise  the need, and therefore that the 3rd party packages have a
market.
2)  We will do it properly (no BACKUP/IMAGE needed, robust to power failure,
etc).


Keith
(VMS file system project leader)
        

Way back in the very, very early days of RSX and Files-11, there was a disk
compression utility that could be used online.  Unfortunately, DCU stood for
Disk Corruption Utility.  Unfortunately, DEC beat Rabbit and the rest to the
disk corruption market by about 15 years, but we pulled out.

    In summary: backup/image and then a restore is a simple and supported,
    although time-consuming, method of improving system performance when
    the fragmentation of your system disk has become extreme and affects
    the pagefile.
    
    - Andy

Um, excuse me, I'm sorry if I've missed something.  While I'm sure the
information in the base note and its replies is certainly useful and is
worth learning, I'm not sure I see what it has to do with DECWindows :-)

- Ken

The original note I think was to stress the point that after you do an upgrade
from a previous version of VMS to some later one on a non-HUGE disk, those
files on the system disk are most likely to be fragmented and that doing one
of the metioned operations can get some performance back.


Sean

re: .22

The author of the base note is running DECwindows on a 6mb VS2000.  I'm sure
that anyone with that configuration would be delighted to hear about *anything*
that would provide a substantial performance improvement.

-Lance- who_is_enjoying_his_recent_upgrade_to_16mb_immensely

    re: .22 et al, 
    
    .24 has my point exactly. As the topic has run its course, I'll write
    lock the note (not as Moderator this time, as its my own I can SET
    NOTE/NOWRITE).
    
    Andy