|
Karl is right. You want to keep the program as clean as possible
and get file tuning translated to file attributes. As such this
is more an RMS question than a BASIC question and perhaps better
placed in the RMS conference VMSZOO::RMS_OPENVMS. The BASICness
about the question is really how to access certain RMS options from
BASIC but first you'd want to identify the desirable options right?
> doing query on a large file (over 100,000 blocks)
That's a tiny file by most standards.
> global buffer count: 0 <- someone say increase this will help, how ?
Depends entirely on the access pattern. It _tends_ to help a lot for
two main reasons:
- It can multiple concurrently accessing programs shared the
same buffers and same a lot of (read) IOs that way.
- It is an easy way to give a single program access to a lot
of buffers without chaning the program code. If you can change the
code, you may well be better of with a large number of LOCAL BUFFERS.
If you can change the process context you can do this with
$SET RMS/IND/BUF=nnn
Global buffer do not help at all for a program reading the file from
the beginning to the end.
> bucket size: 5
That is lowish. As Karl wrote, try EDIT/FDL/NOINTER to see whether
it perhpas suggests something large, or just try say 20 and watch
the results. Then again, if the file has 500,000 record with a
typical key size of say 10 bytes, then it the resulting root level
will be just 2 and the short IOs may be optimal for quick random access
> GET #FILE%, "condition" REGARDLESS \ UNLOCK #FILE%
>
>and OPEN file with ALLOW MODIFY and ACCESS READ.
I wouldn't bother with the unlock. It's just extra CPU time for
the extra trip to RMS, as your access mode it READ, you will not lock.
SOmetimes, if locking is truly excessive and only read to local buffers
are used, then making a private copy and using is ACCESS READ ALLOW
READ is the more efficient way to do business, voind any and all locks.
> VAX BASIC has the following features mentioned for Opening file which
I suspect BUFFERS is the only usefull, justifyiable clause for you.
(Try 100 :-).
> (if you need more info on this file, I can list out all the info
> as shown by analyze/rms_file if necessary)
Yeah, you really shoudda. Why wait to be asked? Also check out $SET
FILE /RMS and the (wonderful :-) RMS_STATS tool on the VMS freeware CD.
hth,
Hein.
|
|
Karl is right. You want to keep the program as clean as possible
and get file tuning translated to file attributes. As such this
is more an RMS question than a BASIC question and perhaps better
placed in the RMS conference VMSZOO::RMS_OPENVMS. The BASICness
about the question is really how to access certain RMS options from
BASIC but first you'd want to identify the desirable options right?
+++>> yes, how to access RMS options from BASIC that help performance
will probably the direction to go, please advise further.
> doing query on a large file (over 100,000 blocks)
That's a tiny file by most standards.
+++>> if the file is not considered to be large, then it may be good to
aware that the entire file will be searched for matching records
and this file will be shared by many applications for concurrent
read/update. The thing is when this program is not run, the locking
rate in the system is basically at a steady rate but when this
program is run to do query search, the locking rate jumps and may
shoot up to 10 times the rate before. So we are looking ways to
reduce this rate by reviewing the BASIC application.
> global buffer count: 0 <- someone say increase this will help, how ?
Depends entirely on the access pattern. It _tends_ to help a lot for
two main reasons:
- It can multiple concurrently accessing programs shared the
same buffers and same a lot of (read) IOs that way.
- It is an easy way to give a single program access to a lot
of buffers without chaning the program code. If you can change the
code, you may well be better of with a large number of LOCAL BUFFERS.
If you can change the process context you can do this with
$SET RMS/IND/BUF=nnn
Global buffer do not help at all for a program reading the file from
the beginning to the end.
+++>> the application will search the entire file for matching records
so you say Global Buffer will not help at all ?
> bucket size: 5
That is lowish. As Karl wrote, try EDIT/FDL/NOINTER to see whether
it perhpas suggests something large, or just try say 20 and watch
the results. Then again, if the file has 500,000 record with a
typical key size of say 10 bytes, then it the resulting root level
will be just 2 and the short IOs may be optimal for quick random access
+++>> I will post the complete output of ANALYZE/RMS_FILE.
yes, please review the file and see if larger bucket seems feasible.
> GET #FILE%, "condition" REGARDLESS \ UNLOCK #FILE%
>
>and OPEN file with ALLOW MODIFY and ACCESS READ.
I wouldn't bother with the unlock. It's just extra CPU time for
the extra trip to RMS, as your access mode it READ, you will not lock.
SOmetimes, if locking is truly excessive and only read to local buffers
are used, then making a private copy and using is ACCESS READ ALLOW
READ is the more efficient way to do business, voind any and all locks.
+++>> someone says if you do a READ, the file will be locked for a short
burst, and unlocking the file will speed up the release of the
resource for other applications to access the same file.
is this non-sense then ?
A experiment has been done to remove the UNLOCK from the GET.
But monitor lock or dlock still indicates excessive high rate
of locking.
> VAX BASIC has the following features mentioned for Opening file which
I suspect BUFFERS is the only usefull, justifyiable clause for you.
(Try 100 :-).
+++>> the BASIC manual says BUFFERS clause rarely need to be specified,
the default is normally sufficient. Will setting of BUFFERS will
directly reduce the locking rate ?
> (if you need more info on this file, I can list out all the info
> as shown by analyze/rms_file if necessary)
Yeah, you really shoudda. Why wait to be asked? Also check out $SET
FILE /RMS and the (wonderful :-) RMS_STATS tool on the VMS freeware CD.
hth,
Hein.
+++>> I will post the complete output of ANALYZE/RMS_FILE.
please advise further.
FILE HEADER
File Spec: ABC.MAS
File ID: (312,1,0)
Owner UIC: [100,100]
Protection: System: RWED, Owner: RWED, Group: RWED, World: R
Creation Date: 2-APR-1997 09:54:55.20
Revision Date: 26-APR-1997 09:56:29.54, Number: 22
Expiration Date: none specified
Backup Date: none posted
Contiguity Options: CONTIGUOUS-BEST-TRY
Performance Options: none
Reliability Options: none
Journaling Enabled: none
RMS FILE ATTRIBUTES
File Organization: indexed
Record Format: fixed
Record Attributes: carriage-return
Maximum Record Size: 50
Longest Record: 50
Blocks Allocated: 122432, Default Extend Size: 5500
Bucket Size: 5
File Monitoring: disabled
Global Buffer Count: 0
FIXED PROLOG
Number of Areas: 2, VBN of First Descriptor: 2
Prolog Version: 3
AREA DESCRIPTOR #0 (VBN 2, offset %X'0000')
Bucket Size: 5
Reclaimed Bucket VBN: 0
Current Extent Start: 116933, Blocks: 5500, Used: 645, Next: 117578
Default Extend Quantity: 5500
Total Allocation: 121000
AREA DESCRIPTOR #1 (VBN 2, offset %X'0040')
Bucket Size: 5
Reclaimed Bucket VBN: 0
Current Extent Start: 110001, Blocks: 1432, Used: 1415, Next: 111416
Default Extend Quantity: 200
Total Allocation: 1432
KEY DESCRIPTOR #0 (VBN 1, offset %X'0000')
Index Area: 1, Level 1 Index Area: 1, Data Area: 0
Root Level: 3
Index Bucket Size: 5, Data Bucket Size: 5
Root VBN: 110256
Key Flags:
(0) KEY$V_DUPKEYS 0
(3) KEY$V_IDX_COMPR 1
(4) KEY$V_INITIDX 0
(6) KEY$V_KEY_COMPR 1
(7) KEY$V_REC_COMPR 0
Key Segments: 1
Key Size: 48
Minimum Record Size: 48
Index Fill Quantity: 2432, Data Fill Quantity: 2432
Segment Positions: 0
Segment Sizes: 48
Data Type: string
Name: "ABC INDEX"
First Data Bucket VBN: 3
The analysis uncovered NO errors.
|
| >> if the file is not considered to be large, then it may be good to
>> aware that the entire file will be searched for matching records
If the entire file is searched, then the bucket size should be
bumped up a lot. It's simple really... with a bucket size of 5
a file of 100,000 block will require 20,000 IOs when using a
bucket size of 25 it will use 4,000 IOs. No magic. Simple math.
>> program is run to do query search, the locking rate jumps and may
>> shoot up to 10 times the rate before. So we are looking ways to
That appears correct. As you read through the file RMS will repeat
the following approximate sequence for each record:
- [unlock previous if needed]
- lock bucket
- lock record
- [unlock current if ROP=NLK or FAC=GET]
- unlock bucket
There is NOTHING you can do about this. This WILL happen for each record.
The total number of lock transition is pre-defined by the number of
records in the file. The lock rate is simply a matter of how fast you
read through the file. Faster = better = more lock/second
They only way to have a real effect is to figure out a way not to read
the whole file. Use the key, add an extra key on a sub-field if needed.
>> the application will search the entire file for matching records
>> so you say Global Buffer will not help at all ?
Correct. Global buffer may well help those 'other' users of the file,
but a program scanning the file will be a 'bad citizen' and thrash
the global buffer space with buckets it will not look back at and
for which no other stream has an interest. Such program should
probaly use a USEROPEN to clear FAB$W_GBC before calling SYS$CONNECT
to have less of an impact on the world.
>> someone says if you do a READ, the file will be locked for a short
>> burst, and unlocking the file will speed up the release of the
That person is correct for WRITE ACCESS, not for read access.
If you do not believe it, then why not spend 5 minutes typing in
a sample program? read a record, ANAL/SYS... SET PROC... SHOW PROC/LOCK
'unlock' record record SHOW PROC/LOCK.
>> the BASIC manual says BUFFERS clause rarely need to be specified,
>> the default is normally sufficient. Will setting of BUFFERS will
>> directly reduce the locking rate ?
As per above... the locking rate is not important it is just a
reflection off the work being done. More buffers do not cause
more work to be doen but may help to do that work faster.
It will not help in a sequential read for an indexed file though
for that, 1 buffer is only really being used (Pitty that :-( ).
For keyed read the default number of buffers will do just fine.
For repeated random inserts inserts in multy key files you will
find that the more buffers the more performance. Hundreds of
buffers may be appropriate for such applications.
>> I will post the complete output of ANALYZE/RMS_FILE.
Well, you did NOT include the statistics area (record count,
compression rates,....) too bad.
>> Root Level: 3
Boooh. With a bigger bucket you can reduce this to 2 and thus
reduce the lock rate for keyed access by 25%.
These records are all key huh?!
hth,
Hein.
|