| T.R | Title | User | Personal
Name | Date | Lines |
|---|
| 70.1 | Versioning Proposal | NNTPD::"[email protected]" | Michelle | Tue Apr 22 1997 15:26 | 264 |
| Here is the initial proposal. It can also be found online at
http://www.zk3.dec.com/~mn/ofst/version.html
Object File Format Versioning
Randy Lowell and Michelle Neisser,
DIGITAL UNIX Development Environment
Overview
This proposal outlines a multi-pronged versioning scheme for the object file
and symbol table formats.
Version information will be available in several forms:
1.Object file version (exclusive of the symbol table)
2.Symbol table version
3.Tool-specific version information
The object file and symbol table versions will be mandatory. They will provide
an immediate
"yes-or-no" answer to the question of whether a tool can successfully process
a given object file.
These version numbers will use a major.minor format.
Tool-specific version information will be optional. It will provide the
opportunity for producers to
communicate version information targeted at particular consumers, which can
then make
better-informed decisions about how to proceed and print more helpful warnings
to the user.
The OF/STWG is the owner of the object file and symbol table formats and will
decide when to bump
these version numbers. The OF/STWG will also work with tool owners to
determine when tool-specific
version numbers should be bumped by object file producers.
Justification
Whenever possible, binary-incompatible changes to the object file format
should be avoided.
However, compatible (minor version) and even incompatible (major version)
changes may occur over
time. Versioning is a safety mechanism that should be robust enough be relied
upon if changes to the
format or interpretation of data are deemed necessary.
The driving factor in this discussion is the need to improve robustness. Old
tools frequently receive
new data, but their behavior in this situation is unpredictable and varies by
tool. Most tools make
educated guesses about how to handle unfamiliar data based on examining object
file contents. We
would like to eliminate the guesswork, by using version numbers to identify
new data and suggesting a
consistent response.
Current Status
Fields designated for version numbers exist in the object file today. Both the
a.out (optional) header
and the symbol table header have vstamp fields that store major and minor
version numbers.
Currently, the linker is the only tool we know of that uses these version
numbers (other than for simply
printing them out, as file and odump do, for example). The linker checks that
the two vstamp fields
are identical and reports an error if they are not.
There is no tool-specific version information in the current object file
format. We plan to create a new
system-defined .comment subsection for the purpose of storing this
information.
Object File and Symbol Table Version Fields
These fields exist but are largely ignored. Therefore the problem is how to
attach semantics to them.
We confront three main questions when defining usage of the version stamps:
1.Do the two vstamp fields need to match in a single object (as the linker
assumes today), and
what does it mean if they do not?
2.What does a tool do upon receiving a version number higher than it knows
how to handle?
3.How do we establish a starting point for meaningful use of version
numbers?
An additional question arises for the linker:
4.What happens when input object files contain different version numbers?
The proposed answers to these questions are:
1.The object file and symbol table vstamp fields can differ. The Third Eye
format is designed for a
separable symbol table, which we can exploit in this case. The only
connection between the
symbol table and the rest of the object file is the file header. The
fields f_symptr and f_nsyms
announce the size and location of the symbolic header, and they are set
to zero if the symbol
table has been stripped. Provided that the use of these fields do not
change (if they do, the
a.out header vstamp should be bumped), the symbol table can be easily
accessed and then
treated as a separate entity for versioning purposes. The advantage of an
independent version
number for the symbol table is that tools like debuggers that are
primarily interested in the
symbol table can then use new features without worrying about changes to
unrelated object file
components such as relocations or raw data.
Some changes might reach across the symbol table/rest-of-object-file
divide. For instance,
external relocations are recorded using external symbol indexes. A change
could be made to the
external symbol table or relocations format that would cause both vstamp
fields to be
incremented.
2.How to respond to a newer object is ultimately the tool developer's
choice. However, we
recommend that all tools use the following guidelines:
If the major version changes, at a minimum a warning should be
printed. Possibly the
version change should cause the tool to exit.
If the minor version changes, at a maximum a warning should be
printed. Possibly the tool
should silently proceed.
Generally, object file producers and modifiers should adhere to
stricter standards than
read-only consumers.
The warning should state the equivalent of: "This object file (or symbol
table) is version 4.1 and I
can handle only 3.1." This warning might be helpfully augmented by
referencing the tool-specific
version information: "Please obtain overify 2.0 or later."
3.To implement versioning support, starting version numbers have to be
agreed upon. Then, all
initial producers must fill in the vstamp fields appropriately. All
consumers must begin checking
these fields. The assumption would be that all tools are built against a
major version, and all are
robust enough to handle potential minor version changes as well as all
previous major versions.
We plan to start with version 3.12 (for both fields), which is the
current value for objects
produced by tools in the DE space. The header file /usr/include/stamp.h
will be updated if
the version changes.
4.The linker cannot link objects whose major version numbers exceed what it
knows how to
handle. However, it can link objects that all have version numbers it
recognizes, even if they do
not all match. In this case, the linker will generate an output object
with a version at least as high
as that of the highest input version. The output version could be higher
than any of the input
objects' versions if the linker adds information belonging to a later
version.
Tool-Specific Version Information
We are introducing a flexible new mechanism for storing version information on
a per-tool basis. The
aim is to give tools enough information to report which newer version is
required to process the object.
Possible uses include indicating that a minor format change has more drastic
implications for one
particular tool, or conversely that a major change may have minimal impact on
this tool. Tool-specific
version information should override the object file and symbol table format
versions.
The tool-specific information will be located in the comment section. The
.comment subsection tag for
this data will be ST_CM_TOOLVERSION, and its attributes will be ST_CMF_APPEND
| ST_CMF_DELETE |
ST_CMF_REPLACE.
The CM_TOOLVERSION subsection will contain a list of (tool_id, version) pairs.
Each consumer will
be identified by name (or other identification string it chooses). The version
number associated with
the tool will indicate the earliest version of the tool that can successfully
process the object.
The format of the subsection is an array of 16-byte entries, divided between a
a 12-byte character
string and a 4-byte unsigned integer. The string stores the tool ID and the
integer stores the version
number.
The tool ID should be unique. If a tool had reason to include more than one
entry, it would pick a
distinct name for each entry. Tool IDs will be registered with the OF/STWG to
help prevent conflicts.
The version information can be encoded however a tool wishes, subject to two
constraints. One
constraint that it must allow for numeric sorting. This is because the linker
may have to select one
TOOLVERSION entry for an output object from among several input entries with
the same tool ID.
Another constraint that the encoding should be consistent over time. Old tools
should reliably report
which new version of the tool is required to process an object.
The concept of tool-based versioning is not completely novel: it has been used
successfully with the
loader. There is an rld version in the dynamic header of shared objects that
tells the loader whether
or not it can load the object. This version number has been changed several
times and the
mechanism has functioned as designed.
Scope
Implementing version numbers on the level proposed here is a broad but shallow
effort. Once we
agree upon starting version field values, tools simply have to begin checking
for them. For example,
ostrip might handle symbol tables up to major version 4. If this version of
ostrip then receives an
object containing a symbol table vstamp of 5, it can quit with an appropriate
message. The only
danger is that a tool neglecting to check version numbers may encounter
trouble if the object file
contents change and the tool fails to detect the change before errors result.
Producers are responsible for filling in the correct version numbers. In
addition to coordinating with
consumers, they must ensure that the versions are appropriate for the system
being used. If a
producer is newer than its host system, it should emit an object at the
version level of the system. An
additional aspect is the cluster-based changes that are new in Steel. The
cluster-version API should
be used (see Al's comments) to determine the applicable OS version.
Minimal infrastructure is required for tool-specific versioning. The linker
must merge TOOLVERSION
subsections correctly. Also, an API should be provided to allow a tool to
request its own information.
For instance, we can provide an interface of the form:
unsigned int
get_version_byname("objname", "toolname")
Such a routine would return only the version data appropriate to that tool, if
it is present. The tool is
then responsible for interpreting that data and deciding whether and how to
proceed.
The data in the stamp.h file also must be supplemented. Definitions for the
symbol table major and
minor versions should be added.
[Posted by WWW Notes gateway]
|
| 70.1 | versioning proposal | SMURF::LOWELL | | Wed Apr 23 1997 10:24 | 212 |
| Here is the initial proposal. It can also be found online at:
http://www.zk3.dec.com/~mn/ofst/version.html
Object File Format Versioning
Randy Lowell and Michelle Neisser,
DIGITAL UNIX Development Environment
Overview
This proposal outlines a multi-pronged versioning scheme for the object file
and symbol table formats. Version information will be available in several
forms:
1. Object file version (exclusive of the symbol table)
2. Symbol table version
3. Tool-specific version information
The object file and symbol table versions will be mandatory. They will
provide an immediate "yes-or-no" answer to the question of whether a tool
can successfully process a given object file. These version numbers will use
a major.minor format.
Tool-specific version information will be optional. It will provide the
opportunity for producers to communicate version information targeted at
particular consumers, which can then make better-informed decisions about
how to proceed and print more helpful warnings to the user.
The OF/STWG is the owner of the object file and symbol table formats and
will decide when to bump these version numbers. The OF/STWG will also work
with tool owners to determine when tool-specific version numbers should be
bumped by object file producers.
Justification
Whenever possible, binary-incompatible changes to the object file format
should be avoided. However, compatible (minor version) and even incompatible
(major version) changes may occur over time. Versioning is a safety
mechanism that should be robust enough be relied upon if changes to the
format or interpretation of data are deemed necessary.
The driving factor in this discussion is the need to improve robustness. Old
tools frequently receive new data, but their behavior in this situation is
unpredictable and varies by tool. Most tools make educated guesses about how
to handle unfamiliar data based on examining object file contents. We would
like to eliminate the guesswork, by using version numbers to identify new
data and suggesting a consistent response.
Current Status
Fields designated for version numbers exist in the object file today. Both
the a.out (optional) header and the symbol table header have vstamp fields
that store major and minor version numbers. Currently, the linker is the
only tool we know of that uses these version numbers (other than for simply
printing them out, as file and odump do, for example). The linker checks
that the two vstamp fields are identical and reports an error if they are
not.
There is no tool-specific version information in the current object file
format. We plan to create a new system-defined .comment subsection for the
purpose of storing this information.
Object File and Symbol Table Version Fields
These fields exist but are largely ignored. Therefore the problem is how to
attach semantics to them. We confront three main questions when defining
usage of the version stamps:
1. Do the two vstamp fields need to match in a single object (as the
linker assumes today), and what does it mean if they do not?
2. What does a tool do upon receiving a version number higher than it
knows how to handle?
3. How do we establish a starting point for meaningful use of version
numbers?
An additional question arises for the linker:
4. What happens when input object files contain different version numbers?
The proposed answers to these questions are:
1. The object file and symbol table vstamp fields can differ. The Third
Eye format is designed for a separable symbol table, which we can
exploit in this case. The only connection between the symbol table and
the rest of the object file is the file header. The fields f_symptr and
f_nsyms announce the size and location of the symbolic header, and they
are set to zero if the symbol table has been stripped. Provided that
the use of these fields do not change (if they do, the a.out header
vstamp should be bumped), the symbol table can be easily accessed and
then treated as a separate entity for versioning purposes. The
advantage of an independent version number for the symbol table is that
tools like debuggers that are primarily interested in the symbol table
can then use new features without worrying about changes to unrelated
object file components such as relocations or raw data.
Some changes might reach across the symbol table/rest-of-object-file
divide. For instance, external relocations are recorded using external
symbol indexes. A change could be made to the external symbol table or
relocations format that would cause both vstamp fields to be
incremented.
2. How to respond to a newer object is ultimately the tool developer's
choice. However, we recommend that all tools use the following
guidelines:
o If the major version changes, at a minimum a warning should be
printed. Possibly the version change should cause the tool to
exit.
o If the minor version changes, at a maximum a warning should be
printed. Possibly the tool should silently proceed.
o Generally, object file producers and modifiers should adhere to
stricter standards than read-only consumers.
The warning should state the equivalent of: "This object file (or
symbol table) is version 4.1 and I can handle only 3.1." This warning
might be helpfully augmented by referencing the tool-specific version
information: "Please obtain overify 2.0 or later."
3. To implement versioning support, starting version numbers have to be
agreed upon. Then, all initial producers must fill in the vstamp fields
appropriately. All consumers must begin checking these fields. The
assumption would be that all tools are built against a major version,
and all are robust enough to handle potential minor version changes as
well as all previous major versions.
We plan to start with version 3.12 (for both fields), which is the
current value for objects produced by tools in the DE space. The header
file /usr/include/stamp.h will be updated if the version changes.
4. The linker cannot link objects whose major version numbers exceed what
it knows how to handle. However, it can link objects that all have
version numbers it recognizes, even if they do not all match. In this
case, the linker will generate an output object with a version at least
as high as that of the highest input version. The output version could
be higher than any of the input objects' versions if the linker adds
information belonging to a later version.
Tool-Specific Version Information
We are introducing a flexible new mechanism for storing version information
on a per-tool basis. The aim is to give tools enough information to report
which newer version is required to process the object. Possible uses include
indicating that a minor format change has more drastic implications for one
particular tool, or conversely that a major change may have minimal impact
on this tool. Tool-specific version information should override the object
file and symbol table format versions.
The tool-specific information will be located in the comment section. The
.comment subsection tag for this data will be ST_CM_TOOLVERSION, and its
attributes will be ST_CMF_APPEND | ST_CMF_DELETE | ST_CMF_REPLACE.
The CM_TOOLVERSION subsection will contain a list of (tool_id, version)
pairs. Each consumer will be identified by name (or other identification
string it chooses). The version number associated with the tool will
indicate the earliest version of the tool that can successfully process the
object.
The format of the subsection is an array of 16-byte entries, divided between
a a 12-byte character string and a 4-byte unsigned integer. The string
stores the tool ID and the integer stores the version number.
The tool ID should be unique. If a tool had reason to include more than one
entry, it would pick a distinct name for each entry. Tool IDs will be
registered with the OF/STWG to help prevent conflicts.
The version information can be encoded however a tool wishes, subject to two
constraints. One constraint that it must allow for numeric sorting. This is
because the linker may have to select one TOOLVERSION entry for an output
object from among several input entries with the same tool ID. Another
constraint that the encoding should be consistent over time. Old tools
should reliably report which new version of the tool is required to process
an object.
The concept of tool-based versioning is not completely novel: it has been
used successfully with the loader. There is an rld version in the dynamic
header of shared objects that tells the loader whether or not it can load
the object. This version number has been changed several times and the
mechanism has functioned as designed.
Scope
Implementing version numbers on the level proposed here is a broad but
shallow effort. Once we agree upon starting version field values, tools
simply have to begin checking for them. For example, ostrip might handle
symbol tables up to major version 4. If this version of ostrip then receives
an object containing a symbol table vstamp of 5, it can quit with an
appropriate message. The only danger is that a tool neglecting to check
version numbers may encounter trouble if the object file contents change and
the tool fails to detect the change before errors result.
Producers are responsible for filling in the correct version numbers. In
addition to coordinating with consumers, they must ensure that the versions
are appropriate for the system being used. If a producer is newer than its
host system, it should emit an object at the version level of the system. An
additional aspect is the cluster-based changes that are new in Steel. The
cluster-version API should be used (see Al's comments) to determine the
applicable OS version.
Minimal infrastructure is required for tool-specific versioning. The linker
must merge TOOLVERSION subsections correctly. Also, an API should be
provided to allow a tool to request its own information. For instance, we
can provide an interface of the form:
unsigned int
get_version_byname("objname", "toolname")
Such a routine would return only the version data appropriate to that tool,
if it is present. The tool is then responsible for interpreting that data
and deciding whether and how to proceed.
The data in the stamp.h file also must be supplemented. Definitions for the
symbol table major and minor versions should be added.
|
| 70.2 | Comments | FLYBA::BRENDER | Ron Brender | Thu Apr 24 1997 12:45 | 52 |
| Just a couple comments regarding .1:
>The version information can be encoded however a tool wishes, subject to two
>constraints. One constraint that it must allow for numeric sorting. This is
>because the linker may have to select one TOOLVERSION entry for an output
>object from among several input entries with the same tool ID. Another
>constraint that the encoding should be consistent over time. Old tools
>should reliably report which new version of the tool is required to process
>an object.
Is this excess generality? That is, is there good reason not to
specify that a version number is a major-minor 16-bit pair?
>The concept of tool-based versioning is not completely novel: it has been
>used successfully with the loader. There is an rld version in the dynamic
>header of shared objects that tells the loader whether or not it can load
>the object. This version number has been changed several times and the
>mechanism has functioned as designed.
There are guidelines for version number usage on the previous page
that apply (just?) to the object and symbol vstamp version fields.
Shouldn't there be similar, if not identical, guidelines for tool
versions?
>Producers are responsible for filling in the correct version numbers. In
>addition to coordinating with consumers, they must ensure that the versions
>are appropriate for the system being used. If a producer is newer than its
>host system, it should emit an object at the version level of the system. An
WOW! Here is zinger hidden in the hay stack! Does this mean that a
compiler (for example) must not emit (at least by default) any new
feature support if that support "goes with" an increased version
number (even if only a higher minor version)? What if you upgrade
all of the related tools in one fell swoop, so that all tools are
in fact perfectly capable of supporting the new features -- since
the "system"/OS version is unchanged, is (default) use of such new
features still a no-no?
Seems like this rule is either mistated/confusing (to me) or it
is so restrictive as to make most of this versioning stuff
superfluous. Please tell me I'm confused...
> unsigned int
> get_version_byname("objname", "toolname")
>
>Such a routine would return only the version data appropriate to that tool,
Is there already an analogous routine for obtaining the object and/or
symbol vstamp version data? If not, shouldn't there be?
|
| 70.3 | | NNTPD::"[email protected]" | Michelle | Fri Apr 25 1997 10:04 | 19 |
| Ron,
> Is this excess generality? That is, is there good reason not to
> specify that a version number is a major-minor 16-bit pair?
We kind of looked at it the other way-- what reason is there to specify
it? Some tools might already have version numbers they use that don't
precisely fit that model. Ladebug, for instance, looks different to
me, and I can't say how exactly they would represent their numbers.
The format versions are specified by the existing format, so of course
those have more constraints.
> Is there already an analogous routine for obtaining the object and/or
> symbol vstamp version data? If not, shouldn't there be?
You're right. There isn't, and there should be. I wrote one when
prototyping, but didn't bother adding it to the proposal. I'll do that.
-Michelle
[Posted by WWW Notes gateway]
|
| 70.4 | | SMURF::LOWELL | | Wed Apr 30 1997 16:58 | 24 |
| > WOW! Here is zinger hidden in the hay stack! Does this mean that a
> compiler (for example) must not emit (at least by default) any new
> feature support if that support "goes with" an increased version
> number (even if only a higher minor version)? What if you upgrade
> all of the related tools in one fell swoop, so that all tools are
> in fact perfectly capable of supporting the new features -- since
> the "system"/OS version is unchanged, is (default) use of such new
> features still a no-no?
>
> Seems like this rule is either mistated/confusing (to me) or it
> is so restrictive as to make most of this versioning stuff
> superfluous. Please tell me I'm confused...
This zinger springs from my interpretation of the rolling cluster upgrade
requirement. I believe we are only talking about major version changes in
the object and symbol table versions, but I agree that there's still a
problem here. Your scenario of replacing all the tools in one fell swoop,
will become more of a reality once the DE pool has split from the base
system.
I've asked Al to come to the review meeting for this proposal, so we
can discuss there what impact the rolling cluster upgrade requirements
should have on object versioning.
|
| 70.5 | Updated Proposal | NNTPD::"[email protected]" | Michelle | Fri May 09 1997 11:38 | 10 |
| The proposal has been updated. The URL is unchanged at
http://www.zk3.dec.com/~mn/ofst/version.html
Underlining is used in place of change bars.
Please review the changes by 5/19/97, which is the date for online approval.
Thanks,
Michelle and Randy
[Posted by WWW Notes gateway]
|
| 70.6 | YES | GEMGRP::BRENDER | Ron Brender | Thu May 15 1997 16:43 | 1 |
| I vote YES.
|
| 70.7 | Approved | SMURF::LOWELL | | Tue May 20 1997 16:58 | 3 |
| Approved online on 5/19/97
OF/STWG ECO ID = 16
|
| 70.8 | toolversion revisited | NNTPD::"[email protected]" | lowell | Wed Jun 04 1997 16:50 | 62 |
| Michelle and I ran into difficulty trying to implement the
toolversion representation. We'd like to reconsider the 3-string
approach for two reasons.
1. The major and minor version strings are unlikely to
be both sortable and meaningful. It seems possible
when we're dealing strictly with numbers:
Maj=4 Min=0 < Maj=4 Min=2
But length throws off a strict comparison
Maj=10 Min=0 < Maj=4 Min=2
So we'd have to add leading 0's, or some other invisible
place-holder that is less than any character that appears
in a version stamp.
Characters, symbols and dates in version stamps make sorting
completely inpractical, but they provide a better match for
user-visible version identification.
With the three string representation we can create either
sortable tool versions or meaningful tool versions but not
both.
2. The way we intend to use toolversion in general does not rely
on the ability to distinguish major or minor versions of a tool.
We simply need to know that a toolversion is greater than or
equal to the recorded value.
Furthermore, the major/minor version of a tool presumably
identifies a kit for that tool. There is no direct corellation
between major version changes in the tool and major version
changes in the symbol table format.
We'd like to back up to the toolversion representation that I believe Al
first suggested. Each toolversion entry is still a triple, but rather
than distinguish major and minor version strings we would record both
an integer value and a string.
char toolname[], unsigned long toolver, char toolstr[]
The integer value can be split anyway the individual tool would like,
but the bottom line is that toolver is always numerically greater for
newer versions of a tool.
toolstr is the user-visible string which we report in warning or error
messages.
toolver is unaligned. Since there's no way to overlay a structure on
the toolversion entries alignment would be wasteful.
[Posted by WWW Notes gateway]
|