Conference azur::mcc

I'm sorry I didn't see this earlier.

The question has to do with ILV, but the answers have to do with
memory management in MCC.  I am going to answer in two sections, one
for VMS and one for Ultrix, as their memory management necessarily
differs.  I will use ILV for examples, but the "protocols" apply to other
mcc routines.  Please bear with me if I tell you a lot that you already know!
Also, please correct me if you think I'm wrong, or if I missed answering
a question.

VMS:
    Dynamic descriptors are descriptors with the mcc_b_class field
set to Class D (DSC_K_CLASS_D).  This means that the memory they
point to was allocated with the VMS dynamic string routines, such as 
LIB$SGET1_DD, etc. and their memory wil be freed with the corresponding 
LIB$SFREE..._DD routines. (Described in the VMS Run Time Library volumes)
Since ILV and other mcc routines are assured that they know the source
of this memory, they can manage it.  This means that ILV can allocate
a new buffer to replace one that is too small.  If you plan on
using Class D descriptors and the VMS dynamic string routines for
memory management, you MAY set mcc_w_maxstrlen and mcc_w_curlen to 0, and
mcc_a_pointer to MCC_K_NULL_PTR and ILV will use the dynamic string routines
to "extend" your buffer.  It will modify the maxstrlen, curlen, and a_pointer
fields of your descriptor appropriately.  If you have allocated a buffer
with the dynamic string routines already, pass it to ILV with the
pointer, class, and lengths set up and ILV will adjust the
buffer and descriptor only if necessary.

*Your code must still deallocate the memory correctly when you are done 
with it.*

If you use mcc_malloc or mcc_calloc and mcc_free (these are the preferred 
memory management routines) you are allocating memory from a different 
source than the dynamic string routines do. Stack memory is another source 
yet.  The use of these kinds of memory is indicated in the descriptor by 
making it "static" -- the mcc_b_class field of the descriptor is 
DSC_K_CLASS_S.  If ILV or another mcc routine sees a descriptor has
class S, it will not be sure of the source of the memory, and so cannot
free and reallocate buffers for you.  Instead, if a buffer is too small,
an mcc routine will attempt to set curlen to the necessary length and 
return a warning or error CVR.  In quite a few cases, however, an 
mcc routine can't guess how long the buffer ought to be, and so won't guess.  
For instance, if you are building an ILV encoding, ILV could tell you (with 
an over-estimate) how much memory you need to finish the current "put", but 
cannot tell you how much memory you need to finish your current encoding.  
Consequently, it doesn't try.  But, for a get, since the data type is known, 
ILV can and does tell you exactly how much buffer it needs.

In the CLASS S case, ILV will only modify the curlen field of your
descriptor, (and the contents of Your buffer).  You must do ALL allocation
and deallocation of buffers.


ULTRIX

Ultrix does not have "dynamic string routines".  For kernel portability,
ultrix equivalents of the lib$ string routines were made based on 
mcc_malloc and mcc_free. 

*The use of these equivalents will remain undocumented and is discouraged.* 

This means that for development on ultrix, you should always use Class S
descriptors and do your own memory management.


Other considerations:

Since dynamic string routines are VMS specific, their use is
discouraged even on VMS. (See last section).

Mcc descriptors should always be initialized.  All of the fields are
used someplace!  mcc_w_maxstrlen should always be greater than or
equal to mcc_w_curlen, and both should be 0 or greater.  If the buffer
pointer mcc_a_pointer points to something, both maxstrlen and curlen
should be set.  If the pointer is NULL, then maxstrlen and curlen should
be 0.

maxstrlen is always in bytes.  curlen is USUALLY in bytes.  The current
exceptions: 
	BITSET - curlen is in bits ( the number used)
	HEXSTRING - curlen is in "nibbles"  (also the number used)

I understand that there is an SRM ECO being written to describe mcc
descriptors.


Other questions raised: 

As the  use of the VMS-specific dynamic string routines is discouraged,
MMs should use Static (DSC_K_CLASS_S)  descriptors, and the INSUF_BUF CVR with 
curlen= size needed protocol if Out_P is too small.  

FCL uses mcc_malloc and mcc_calloc to allocate buffers, so a descriptor
with a maxstrlen and a buffer pointer but 0 curlen is simply empty of
contents.  The descriptor is describing an unused buffer.

A CURRENT HOT BUTTON:

We could use documented and consistently used  memory management techniques to
clean up the hodge-podge that helps make memory leaks in mcc!

In .1, I said that BITSET curlen are in bits used, but maxstrlen is in bytes.
I was mistaken.  For BITSET, maxstrlen is in BITS, and must be a multiple of
8 bits, up to 32 bits long.  (The 4 byte limit on bitsets is a C limitation.)

Also, the mcc_b_dtype field of BITSET is DSC_K_DTYPE_V, and
the mcc_b_dtype field of a hexstring is DSC_K_DTYPE_HHBS.

in summary, for BITSETs:
	mcc_w_maxstrlen is 0,8,16,24 or 32 (bits).
	mcc_b_dtype is DSC_K_DTYPE_V,
	mcc_w_curlen is 0-32 (number of bits actually used)
	mcc_l_dt is MCC_K_DT_BITSET

for HEX STRINGS:
	mcc_w_maxstrlen is in bytes
	mcc_b_dtype is DSC_K_DTYPE_HHBS,
	mcc_w_curlen is  in NIBBLES 
	mcc_l_dt is MCC_K_DT_HEXSTRING

and we prefer mcc_b_class to be DSC_K_CLASS_S.

Ruth

Re .1:

I think DYNAMIC descriptors on Ultrix is an obscure point.

>ULTRIX
>
>Ultrix does not have "dynamic string routines".  For kernel portability,
>ultrix equivalents of the lib$ string routines were made based on 
>mcc_malloc and mcc_free. 
>
>*The use of these equivalents will remain undocumented and is discouraged.* 
>

Hum hum...

So why FCL PM is (now X1.2.17) passing DYNAMIC out_p descriptor ?

Isn't it a problem ? What will happen if I use this MCC descr. with ILV
routines and I do not have enough room to put the value in ?

>
>I understand that there is an SRM ECO being written to describe mcc
>descriptors.
>

Great ! I like this one ! Better late than never ! :-)
 
Regards.

Pierre.

                      <<< Note 2431.3 by TAEC::LAVILLAT >>>
             -< So why are you passing DYNAMIC desc. to the users >-

>>Re .1:
>>
>>I think DYNAMIC descriptors on Ultrix is an obscure point.
>>
>>ULTRIX
>>
>>Ultrix does not have "dynamic string routines".  For kernel portability,
>>ultrix equivalents of the lib$ string routines were made based on 
>>mcc_malloc and mcc_free. 
>>
>>*The use of these equivalents will remain undocumented and is discouraged.* 
>>
>>
>>Hum hum...
>>
>>So why FCL PM is (now X1.2.17) passing DYNAMIC out_p descriptor ?
>>
>>Isn't it a problem ? What will happen if I use this MCC descr. with ILV
>>routines and I do not have enough room to put the value in ?
>>
The use of the mcc equivalents to the VMS dynamic string routines (LIB$SGET1_DD
and so on) is discouraged.  The use of dynamic memory on VMS is discouraged,
and if you port an application from Ultrix to VMS which relies on class
D memory management in the kernel but uses plain mcc_malloc/calloc and mcc_free,
be aware that on VMS you will have to switch over to the correct dynamic
string LIB$ routines or screw up your memory management.

The mcc equivalents were written so that the framework could support minimum 
change to the code for the same function on Ultrix and VMS.  ILV and other 
routines which support Class D on VMS will support Class D on ultrix.  I.E. 
they will switch buffers for you, and free the old buffer.

I, personally, used to think that the mcc equivalents of the VMS string routines
*ought* to be documented, BUT now I think it would be preferrable to have 
*one* set of memory allocation, extension, and freeing routines to be ported 
to all underlying OSes, and phase out the use of both Class D and Class
S for MCC.

As for FCL, if an MM or a routine does not work with Class D protocols,
you will get back an exception or CVR with some form of "Insufficient Buffer",
whatever the Class of the Out_P buffer descriptor.  This means that Class 
S descriptor behaviour is the required fallback protocol for all routines
and MMs.

If I've missed answering something, please send mail as well as posting
a note if you require a quick reply.

Ruth K.

>
>If I've missed answering something, please send mail as well as posting
>a note if you require a quick reply.
>

No.

Thanks.

Pierre.

Ruth,

Thank you very much for your very detail answers.

The only problem here, it is not what I have expected, and judging from your
answers, I think you have the same opinion.

But first, let me explain the origin of original question. I have been taught
that for compatibility between VMS and ULTRIX don't use anything platform
specific in MCC. So, when Matt Guertin in note 2439.6 have said:

>       Read the routine description of the mcc_call -- Page 508 on Out_P. 
>    For Class D only, the Service Provider should call lib$sget1_dd() to
>    re-size the buffer.

I have been gratefull for that referal, because I even din't think to look into
SRM for that kind of answers.

What I did expected as the answer is that there are some routines like
mcc_malloc and mcc_free (BTW, these two are undocumented too) which I may use
when dealing with dynamic descriptors.

You have mentioned

> The question has to do with ILV, but the answers have to do with
> memory management in MCC.

Sorry, but the name of this note is "Dynamic descriptors?". ILV has been taken
only as an example. But there is the reason why I have been trying to
concentrate only on one data type: MCC developers provide the different routine
sets for different data types - for example you have mcc_time_create,
mcc_time_delete and mcc_aes_create, mcc_aes_delete. And until recently I haven't
had the slightest idea that the descriptor for time is dynamic one. And I don't
need to know that as long is I have mcc_time_create and mcc_time_delete
routines. With such approach the MCC developers are free to implement them as
necessary on different platforms and the users of the mcc_... routines won't
have any problem with the portability issues: the same code will be able to run
on any platform after recompiling and relinking. You even don't need to insert
any conditional #if statement into your code, so it will be compiled differently
on different platforms.

> Ultrix does not have "dynamic string routines".  For kernel portability,
> ultrix equivalents of the lib$ string routines were made based on 
> mcc_malloc and mcc_free. 
>
> *The use of these equivalents will remain undocumented and is discouraged.* 
>
> This means that for development on ultrix, you should always use Class S
> descriptors and do your own memory management.
>
> Since dynamic string routines are VMS specific, their use is
> discouraged even on VMS. (See last section).

I am not happy about this. The ability to use dynamic descriptors will greatly
simplify the code. You don't need to worry about reallocation of memory or
allocate more than needed to avoid reallocation. Plus, the most important thing,
different developers don't need to reinvent the weel and do their own memory
handling, quite often in more than one place in their code.

What I whould like to see, either 

> BUT now I think it would be preferrable to have 
>*one* set of memory allocation, extension, and freeing routines to be ported 
>to all underlying OSes, and phase out the use of both Class D and Class
>S for MCC.

but with dynamic descriptors handling, or sets of routines similar to
mcc_time_create, mcc_time_delete for all data types which provide their own
dynamic memory handling.

Two more things.

First, you have mentioned

> If you plan on
> using Class D descriptors and the VMS dynamic string routines for
> memory management, you MAY set mcc_w_maxstrlen and mcc_w_curlen to 0, and
> mcc_a_pointer to MCC_K_NULL_PTR and ILV will use the dynamic string routines
> to "extend" your buffer.

There is one problem. ILV routines will do fine with such descriptors, but some
modules cannot handle that, and not because they are not ready to deal with
dynamic descriptors (otherwise you won't be able to call such module from FCL),
but because they are not able to handle dynamic descriptors with 0 maxstrlen.

The problem, I think, has to do with the fact, that such modules in their
validation of the mcc_call parameters test that mcc_a_pointer is not
MCC_K_NULL_PTR and mcc_w_maxstrlen is not 0 without any regard what kind of
descriptor it is, and return MCC_S_INV_OUT_P for dynamic descriptor with
maxstrlen equal 0.

And the last one:

>I understand that there is an SRM ECO being written to describe mcc
>descriptors.
>
>
> Great ! I like this one !

I like this one too. When?

Best regards,
Gene

>I am not happy about this. The ability to use dynamic descriptors will greatly
>simplify the code. You don't need to worry about reallocation of memory or
>allocate more than needed to avoid reallocation. Plus, the most important thing,
>different developers don't need to reinvent the weel and do their own memory
>handling, quite often in more than one place in their code.

We are pretty much agreed, except that heap memory is easier to deal with
on VMS--there are tools to find memory leaks with, for one thing.  Another
consideration is porting from Ultrix TO VMS-- you have to carefully look
at each use of the dynamic memory to make sure it is correct as ported.

>> BUT now I think it would be preferrable to have 
>>*one* set of memory allocation, extension, and freeing routines to be ported 
>>to all underlying OSes, and phase out the use of both Class D and Class
>>S for MCC.
>
>but with dynamic descriptors handling, or sets of routines similar to
>mcc_time_create, mcc_time_delete for all data types which provide their own
>dynamic memory handling.

There are always backward compatibilty issues.  Any new common memory
management routines would have to be introduced before the old stuff
can be phased out.  Hopefully, multiplying the mess for a known short
term would ease transition...  Besides, it ain't gonna happen for v1.2.

>Two more things.
>
>First, you have mentioned
>
>> If you plan on
>> using Class D descriptors and the VMS dynamic string routines for
>> memory management, you MAY set mcc_w_maxstrlen and mcc_w_curlen to 0, and
>> mcc_a_pointer to MCC_K_NULL_PTR and ILV will use the dynamic string routines
>> to "extend" your buffer.
>
>There is one problem. ILV routines will do fine with such descriptors, but some
>modules cannot handle that, and not because they are not ready to deal with
>dynamic descriptors (otherwise you won't be able to call such module from FCL),
>but because they are not able to handle dynamic descriptors with 0 maxstrlen.

There is a Class S descriptor protocol and a Class D descriptor protocol.
If a module cannot deal with Class D, it must fall back onto the Class S
protocol, and return, in the mcc_w_curlen field of the erring descriptor the 
amount of buffer it requires.  (With an error message.)  In some cases,
this can't be estimated -- in ILV, for instance, if the buffer to contain
the encoding is too small at some point, only the caller (if anyone) can 
really tell how much more there is to encode. 

In any case, if a module can't deal with class D and falls back onto
the "insuff_buf" and curlen protocol, you _can_ extend it in your module,
whether you prefer class S or D.

>The problem, I think, has to do with the fact, that such modules in their
>validation of the mcc_call parameters test that mcc_a_pointer is not
>MCC_K_NULL_PTR and mcc_w_maxstrlen is not 0 without any regard what kind of
>descriptor it is, and return MCC_S_INV_OUT_P for dynamic descriptor with
>maxstrlen equal 0.
>
Have you experimented?  In these cases, is there a value returned in 
the curlen?
>
>And the last one:
>
>>I understand that there is an SRM ECO being written to describe mcc
>>descriptors.
>>
>>
>> Great ! I like this one !
>
>I like this one too. When?
>

I think I heard that it was in progress, and I expect the finished version
the next version of the SRM.  I'm sure someone will correct me if I'm
wrong!

Ruth

>>The problem, I think, has to do with the fact, that such modules in their
>>validation of the mcc_call parameters test that mcc_a_pointer is not
>>MCC_K_NULL_PTR and mcc_w_maxstrlen is not 0 without any regard what kind of
>>descriptor it is, and return MCC_S_INV_OUT_P for dynamic descriptor with
>>maxstrlen equal 0.
>>
>
>Have you experimented?  In these cases, is there a value returned in 
>the curlen?

Yes, I did. There is no value in the curlen, as it is supposed to be for such
status return - MCC_S_INV_OUT_P.

I even have seen source code for some modules where they do the kind of testing
which I have mentioned. I am not saying that this code still the same for all
such modules, but probably in some it is not have been changed.

BTW, I think it may be a know bug, and it is the reason why FCL call modules
with out_p whith maxstrlen set to 1024, as I have mentioned in the original
note. Otherwise, why to bother to allocate this memory?

Best regards,
Gene

    Returning MCC_S_INV_OUT_P because a Class D descriptor has a maxstrlen
    of 0 is a bug.  There is nothing invalid about such a descriptor setup.
    A return of MCC_S_INV_OUT_P implies that the Out_P descriptor structure
    itself is invalid (for example, if the mcc_b_ver field is anything other
    than 1).  We have already fixed a couple of these bugs in V1.2.
    
    -Matt.
    

RE: .9

> A return of MCC_S_INV_OUT_P implies that the Out_P descriptor structure
> itself is invalid (for example, if the mcc_b_ver field is anything other
> than 1).

  Instead of use the 1 (above) - please use the proper literal symbols:

	MCC_K_VER_DESCRIPTOR

  /keith

    Yeah, I forgot the spelling of it and didn't want to look it up.  All
    good engineers should use constants.  Thanks for keeping my hat on
    straight. 
    
    -Matt.