[Search for users] [Overall Top Noters] [List of all Conferences] [Download this site]

Conference smurf::unix_objsym

Title:Digital UNIX Object File/Symbol Table Notes Conference
Moderator:SMURF::LOWELL
Created:Mon Nov 25 1996
Last Modified:Thu Jun 05 1997
Last Successful Update:Fri Jun 06 1997
Number of topics:71
Total number of notes:314

68.0. "Extended Source Location Information" by SMURF::LOWELL () Wed Apr 16 1997 11:03

     	A Symbol Table Proposal to Extend Source Location Information
    
    	Authors: Ron Brender, Dennis Murphy, Peter Thompson
    	
    
    
                       Extended Source Location Information.
                       
       
    1.1 Introduction
    
    This document describes how Extended Source Location information
    is to be contained in the Third Eye Symbol table and is intended
    to serve as a specification as to how that information is
    constructed and accessed by the compilers and tools.
    
    Extended Source Location information is intended to provide more
    information than is contained in the existing line number table
    including PC deltas, line number deltas, file transitions,
    columns, and line/column ranges.
    
    It is also intended that the Extended Source Location information
    be a compact and extensible representation. Those issues will be
    addressed in the detailed description.
    
    While it was initially intended that Extended Source Location
    information be defined to allow ATOM to accurately describe "split
    routines" it seems likely that it will also be used be the
    compilers to describe "out-of-line" code where functions are
    segregated into hot and cold regions.
    
    The existing line number table is unable to describe either split
    routines or out-of-line code, resulting in difficult to debug
    code.
    
    This document makes use of a Tag/Length/Value (TLV)
    representation, a self-describing data format. This particular TLV
    representation is described in UNIX_OBJSYM note 33.3 and will not
    be explained further in this document.
    
    1.2 Basic rules
    
    o If a compilation is to build Extended Source Location information 
      that information is to be provided for the entire compilation. It is 
      NOT a pick-and-choose mechanism. 
    
    o Extended Source Location information is to be contained in the 
      Optimization Table.
    
    o Extended Source Location information will be produced in addition 
      to the existing line number table, at least initially. This is
    intended
      to provide backward compatibility with existing tools. Issues
      of object file size and compact line number representation are
      to be deferred.
    
    o Each procedure descriptor will describe its Optimization Table
      contribution via the pdr iopt element as detailed below.
    
    o File descriptor entries describing the Optimization Table are
      defined in a following section.
    
    1.3 Symbol Table Placement
    
    The Extended Source Location information will be located in the
    currently unused Optimization Table.
    
    Compilers will construct Extended Source Location information in
    the Optimization Table in addition to any other information to be
    placed in that table.
    
    All entries placed in the Optimization Table are in the form of
    self-describing TLV entries.
    
    The compilation of a module will produce an object file with a
    single Optimization Table that is constructed and accessed using
    the existing file and procedure mechanisms.
    
    The linker will concatenate the Optimization Tables of all modules
    contributing to an image. Note in particular that the linker need
    not parse or analyze the Optimization Table information in any
    way.  The resulting image Optimization Table will be accessed by
    post link tools such as OM and debuggers
    
    1.4 Form
    
    The Extended Source Location information is managed as a 
    Tag/Length/Value (TLV) element of the Optimization Table.
        
    The fdr copt element is a count of the number of bytes a file
    contributed to the Optimization table.
    
    The fdr ioptBase is used to locate the files first TLV entry.
    The files first TLV entry must be the same as the first TVL entry
    of the first procedure (via PD table) of the file.
    
    Each procedure descriptor iopt member will locate the Optimization
    Table contribution using the standard Third Eye rules for indexing
    via the procedure descriptor.
    
    The iopt value will indicate the first TLV entry for a given routine.
    
    Tools, including the debuggers, seeking to process the Extended Source 
    Location information will scan a procedure's Optimization table TVL
    elements until the Extended Source Location element is found or the
    procedure's terminal TVL entry is encountered.
    
    It is an error for a module containing Extended Source Location 
    information to omit producing an Extended Source Location TLV entry
    for any routine. However, the error is always nonfatal.    
    
    1.5 Issues
    
    The Tag value for Extended Source Location information will need to
    be specified.
        
    
    
    2.1 Detailed Extended Source Location Format
        
        
    The current line number information has a very compact and very
    restrictive format.  We propose some changes to the current format
    in order to gain some flexibility and allow for future expansion.
        
    The current format is basically a byte stream, with each byte
    being divided into two 4 bit halves.  The high ordered bits are
    used to indicate the line number delta, with a range of -7 to 7
    lines.  A bit string indicating -8, or '1000'b is used to indicate
    that the line number delta is too large, and is represented in the
    next 2 bytes.
        
    The low 4 bits of each byte, except in the case above, are used to
    represent an instruction delta, from 1 to 16 instructions (the
    range covers from 0 to 15, but you always add 1, as each pc delta
    represents at least one instruction).
        
        
    2.2 New Proposal
        
    In order to give us greater flexibilty in dealing with line
    numbers we want to separate the byte stream into two different
    types of formats, which we will call data mode and command mode.
    The basic format described above will now be known as data mode 1.
    Since we know the initial state of things from the procedure and
    file descriptors before we start looking at the line number info,
    it is intended that we will always presume to start in data mode
    1.  There will also be a data mode 2, which will be used to
    represent line, pc, and column information for programs with more
    than one statement on a line.  Data mode 0 is currently not
    defined, and would be seen as an error.
        
    The current 'escape' of '1000'b in the line delta information
    would now become a escape to the command mode.
        
    Once an escape is encountered, the line number algorithm will
    switch to interpreting command mode, and subsequent bytes will be
    treated as commands, paired with LEB128 data.  LEB(Little Endian
    Byte)128 is a compact way of expressing integer data and does not
    need to be used only on little endian machines.  Each command will
    be 1 byte in length, with the 2 high order bits as flags for mark
    and resume, respectively.  The mark bit will be used to indicate
    the last command in a sequence that takes us to the next line. The
    resume flag would be an indication to return to the previous data
    mode.  The low 6 bits then, will be used to distinguish the
    various commands.
        
    The LEB128 is a byte stream way of encoding data taking advantage
    that most of the integers will be fairly small.  Integers are
    split into 7 bit segments, '1111111'b, and the segments are put
    into the 7 lower bits of a byte. The high order eighth bit is used
    as a flag to indicate whether there are any more bytes in the
    number. A 1 indicates a continuation and a 0 indicates completion.
    Large numbers would thus be formed by concatenating together the 7
    bit segments.
        
    2.3 Commands
        
    We've identified a small group of commands that we intend to start
    with.  These may grow as new needs are assessed in the future.
    The commands, with an explanation of how the LEB data should be
    interpreted follow below.  No particular bit pattern has been
    established yet for each command.
        
    add_pc, LEB 
    	        signed delta, Value is added to current pc 'value' (as
        	determined by previous line number machinations) to 	
        	end up with a pc value that is greater or less than the
        	current value.
        
    add_line, LEB	
    	        signed delta, added to current line number
        
    set_col, LEB	
        	absolute value, used to associate pc with a particular
        	section of a line, rather than just the beginning of 
        	a line.
        
    set_file, LEB	
        	absolute value, used to set the current information to
        	a particular source file.  Needed to handle include 
        	files, which we have no way of currently representing 
        	in the present symbol table.
        
    set_data_mode, LEB
        	absolute value.  Only a 1 or a 2 would be acceptable 
        	here, for data mode 1 or data mode 2.  Anything
        	else would be an error.
        
    add_pc_line, LEB, LEB	
        	Sort of a larger, more flexible representation of the
        	current delta/pc pair.  Both signed deltas.
        
    add_pc_line_col, LEB, LEB, LEB
        	To handle pc, line and column data all at once.  
        	The first 2 are signed delta's.  The third is an
        	absolute value.
    
T.RTitleUserPersonal
Name
DateLines
68.1Is effect of backing up the PC defined?WIBBIN::NOYCEPulling weeds, pickin' stonesWed Apr 16 1997 19:025
It's relatively easy for a debugger to expand the current line-number table
into a sorted PC->line number map that is easy to search.  It's not so clear
how to do that with the new proposed table, given the ADD_PC function.  What
is supposed to happen if the same PC is described as part of several lines?
In any case, building the map seems to require a sort operation in general. 
68.2Does this answer the question, or did you have something else in mind?QUARRY::petertrigidly defined areas of doubt and uncertaintyThu Apr 17 1997 17:5626
>                    Is the effect of back up the PC defined?
> It's relatively easy for a debugger to expand the current line-number table
> into a sorted PC->line number map that is easy to search.  It's not so clear
> how to do that with the new proposed table, given the ADD_PC function.  What
> is supposed to happen if the same PC is described as part of several lines?

I think I know what you're questioning here, but let me just go over a 
few things to make sure.  

This proposal started out as a way to represent split routines in the symbol
table, but it was felt my suggestions were a bit awkward and didn't address
all the possibilities, specifically problems with line numbers.  So we 
transitioned into this proposal which deals only with line numbers.  While
there is a many PC to line number mapping, there is never (well, not considering
alternate entry points) a many lines to one PC here.  The only time the 
PC would actually be backed up is when a section of code is split off 
and placed in a "hot region".  This could be a good distance away from
its previous position, though it may just be moved to the top of the 
routine.  Once the end of this section of code had been reached, the PC
would be adjusted again, likely forward, to where it had been before it
had been adjusted backwards.  I don't forsee a case where several lines 
would point to the same PC.  I would consider that an error on the part
of the producer of the extended source info, but maybe I'm not thinking 
object orientated enough...

PeterT
68.3TLE::DMURPHYFri Apr 18 1997 12:4129
> It's relatively easy for a debugger to expand the current line-number table
> into a sorted PC->line number map that is easy to search.  It's not so clear
> how to do that with the new proposed table, given the ADD_PC function. 

  I'll be the first person to admit that there are additional problems with
  trying to represent what the compilers are doing than what is detailed
  in this note. Having said that let explain how I thought we'd use the table.

  The procedure descriptors, in addition to the elements of the expanded
  line number table are sorted in address assending order. (Well the linker
  only partially sorts them but that's another issue.)

  Locate the function for a given pc taking into account lexically enclosed
  functions. Use the new representation to run a machine that transitions       
  pc/file/line until we get the desired PC. Set the debugger location state
  given the file/line/column we arrived at by running the machine. 

> What is supposed to happen if the same PC is described as part of several     
  lines?

  This is certainly not something that can be done with the existing mechanism.
  Is that your point?  The compilers wound not issue such a description until
  we defined what was suppoed to happen.

 >In any case, building the map seems to require a sort operation in general. 

  I guess I don't see it as a simple map. It does require more work of 
  the debuggers but it more accurately captures the program state at debug
  time. 
68.4Does it meet the goals?WIBBIN::NOYCEPulling weeds, pickin' stonesFri Apr 18 1997 13:1015
>  Locate the function for a given pc taking into account lexically enclosed
>  functions. Use the new representation to run a machine that transitions       
>  pc/file/line until we get the desired PC.

Two problems:  Worst, this assumes that function fragments aren't interleaved
with each other, so that you can find the function before interpreting the
line-number info.  I thought one reason for the proposal was so that rarely-
used code could be pushed far away from the "hot" code of an image.  And
second, this requires a linear search to map PC to line number,  The latter
doesn't sound too bad, until you realize that debuggers like to do this
operation for every line of machine code they display.

Maybe I'm misunderstanding.  To find the function, do you use the debugger
info, or the Runtime Procedure Descriptor and Code Range Descriptor (which
can describe discontiguous, interleaved pieces of functions)?
68.5This only addresses a subset of the problem...SMURF::PETERTrigidly defined areas of doubt and uncertaintyFri Apr 18 1997 14:2349
    Let's be discontiguous:
    
    > Maybe I'm misunderstanding.  To find the function, do you use the
    > debugger info, or the Runtime Procedure Descriptor and Code Range 
    > Descriptor (which can describe discontiguous, interleaved pieces of 
    > functions)?
    
    Neither dbx or ladebug (to my knowledge) use the RPD's and code range
    descriptors.  While the potential seems great, they are not currently
    part of non-shared objects, and there is no easy mapping from a 
    procedure descriptor, which contains all sorts of useful info, to 
    the RPD /CRD pair which contain info useful for exception handling.
    You still need to go through the symbol table to find the function.
    
    > Two problems: Worst, this assumes that function fragments aren't
    > interleaved with each other, so that you can find the function
    > before interpreting the line-number info.  I thought one reason
    > for the proposal was so that rarely- used code could be pushed
    > far away from the "hot" code of an image.
    
    This started out as a split routine proposal, so that the hot and 
    cold regions could be described accurately, but that was seen as 
    missing some basic assumptions, such as how to deal with the line
    numbers.  OM can already split routines and executables and rearrange
    them into hot and cold regions.  But it does nothing to the line number
    info. So once the routine is split, the line numbers no longer
    represent the code accurately.  This proposal is intended for producers
    and consumers alike to define a way to map the line numbers in 
    cases where odd things have been done to the code, and to deal with
    deficencies like how to handle include files which contain code,
    something we've ignored for a long time.  The function will still
    be found by start address + size of function for a particular
    pc.  Obviously this will not map well with split routines, but
    it does allow a way of getting there by laboriously going 
    through the line number info if no other match is found.
    I'd like to see us eventually come up with an easy way of mapping
    which code ranges apply to which function, but, no that is no a 
    goal of THIS proposal.
    
    >  And second, this requires a linear search to map PC to line
    > number, The latter doesn't sound too bad, until you realize that
    > debuggers like to do this operation for every line of machine
    > code they display.
    
    Yes it can be very compute intensive, but at least it's a way to deal
    with it until we come up with a more complete answer.
    
    PeterT
    
68.6TLE::DMURPHYFri Apr 18 1997 16:2467
    > Maybe I'm misunderstanding.  To find the function, do you use the
    > debugger info, or the Runtime Procedure Descriptor and Code Range 
    > Descriptor (which can describe discontiguous, interleaved pieces of 
    > functions)?

    I believe that ladebug makes limited use of the prd & crd data. 

    One reason not to use it is that I believe code linked -static (the            
 kernel??) does not/did not retain the .xdata/.pdata sections containing
    the prd/crd information.motie>

    The following is a 3.2 mvunix.
 odump -h /vmunix


                        ***SECTION HEADER***
        Name             Paddr               Vaddr               Size
                         Scnptr              Relptr              Lnnoptr
                         Nreloc              Nlnno               Flags

/vmunix:
        .text     0xfffffc0000230000  0xfffffc0000230000  0x00000000002ef930
                  0x0000000000000270  0x0000000000000000  0x0000000000000000
                             0                   0            0x00000020
        .data     0xfffffc0000537340  0xfffffc0000537340  0x000000000006c480
                  0x00000000003075b0  0x0000000000000000  0x0000000000000000
                             0                   0            0x00000040
        .rdata    0xfffffc00005352a0  0xfffffc00005352a0  0x00000000000020a0
                  0x0000000000305510  0x0000000000000000  0x0000000000000000
                             0                   0            0x00000100
        .rconst   0xfffffc000051f930  0xfffffc000051f930  0x0000000000015970
                  0x00000000002efba0  0x0000000000000000  0x0000000000000000
                             0                   0            0x02200000
        .lit8     0xfffffc00005a37c0  0xfffffc00005a37c0  0x0000000000000880
                  0x0000000000373a30  0x0000000000000000  0x0000000000000000
                             0                   0            0x08000000
        .sdata    0xfffffc00005a4040  0xfffffc00005a4040  0x0000000000003330
                  0x00000000003742b0  0x0000000000000000  0x00000000003775e0
                             0                   0            0x00000200
        .sbss     0xfffffc00005a7370  0xfffffc00005a7370  0x0000000000000960
                  0x0000000000000000  0x0000000000000000  0x00000000003775e0
                             0                   0            0x00000400
        .bss      0xfffffc00005a7cd0  0xfffffc00005a7cd0  0x00000000000c3650
                  0x0000000000000000  0x0000000000000000  0x0000000000000000
                             0                   0            0x00000080

    
    > Two problems: Worst, this assumes that function fragments aren't
    > interleaved with each other, so that you can find the function
    > before interpreting the line-number info.  I thought one reason
 
    I think I've already agreed that the current symbol table format is a pretty
    poor vessel for containing the information the compilers would use
    to convey scheduling/optimization/other information.

    This proposal seeks to cause minimal disturbance of the symbol table and
    breakage of the tools. 
 
    >  And second, this requires a linear search to map PC to line
    > number, The latter doesn't sound too bad, until you realize that
    > debuggers like to do this operation for every line of machine
    > code they display.
 
    I'd be willing to that this is not as bad as you make it sound. Caching 
    search results and running the machine from known cached states would
    most probably speed up succeeding mappings.
68.7approvedSMURF::LOWELLMon Apr 21 1997 13:185
Approved on 4/21/97.

OF/STWG ECO ID = 13

68.8Various commentsFLYBA::BRENDERRon BrenderWed Apr 23 1997 11:0441
I think there are a couple different issues that are getting mixed up together
-- maybe these comments will help clarify, at least what *I* had in mind...


Re .3: > Locate the function...
Re .4: > ...interleaved function fragments...

The source locator table does not, and is not intended, to address how to do
that. Additional symbol table information is needed (and I have a proposal in
mind that will support even interleaved function fragments). So, this is not
a complete solution to the "split routine problem" if you will. But it is a
necessary part, which also provides many other desirable features.


Re .1: > ...the same PC is described as part of several lines...

Ah, good question! This representation is intended to allow such descriptions,
in order to support the full glory :-) of debugging optimized code in the
future -- in particular, in order to deal with the likes of cross-jumping
optimizations (and others -- but not inlining, which is different however).

It is TBD when we will want to support this full generality, however. In the
short term, as Peter suggests, we may possibly limit ourselves to emitting
only a 1-to-many PC-to-line mapping. However, it would please me greatly if
ladebug at least would take on the challenge of supporting the full generality
of the representation.

There is a half-way point here, which is a reasonable fall back for dbx and/or
ladebug. Allow a many-to-many mapping to be represented in the symbol table
but ignore duplicate PC information. This would allow GEM to emit the full
description in advance of when the debugger and other tools choose to deal
with it.

(GEM already knows how to do this, given a suitable object symbol
table representation! GEM needs to be a tad careful to arrange that the more/
most useful mapping comes first, but this is no different than being careful
to arrange that the only mapping is the more/most useful one in the current
scheme.)

It is a weakness of the proposal that we did not point out and clarify this
issue.
68.9Fly in the ointment ?GEMGRP::MONTELEONEThu May 22 1997 15:2521
    
    
    Regarding this point:
    
    
        The linker will concatenate the Optimization Tables of all modules
        contributing to an image. Note in particular that the linker need
        not parse or analyze the Optimization Table information in any
        way.  The resulting image Optimization Table will be accessed by
        post link tools such as OM and debuggers
    
    
    Consider the set_file command, which references a file descriptor
    index. Wouldn't the linker need to detect such commands and possibly
    add to the IFT table for this case. How else would the postlink tools 
    know how to map the files referenced in the set_file command if there
    were no such mapping ?
    
    
    Bob
    
68.10-.1 not a problem...GEMGRP::MONTELEONEThu May 22 1997 17:2515
    
    
    This is not a problem afterall, since the linker produces a complete
    FIT for each file descriptor which has a mapping for every file 
    descriptor described in its .o file. I was under the impression that
    the FIT for a file descriptor contained only entries that were 
    *referenced* in the file descriptor, but it doesn't - it produces
    entries for all fellow files which come from the same .o file. 
    
    If there linker were to ever optimize this behavior, then there would
    be a problem, but there is no problem now...
    
    
    Bob
     
68.11 Alternate entry point clarification...GEMGRP::MONTELEONEMon Jun 02 1997 15:3533
    
    This note clarifies the specification of alternate entry points
    and extended source line information.
    
    An entry point and all of its corresponding alternates will
    reference the same set of PPOD structures.  The main entry point
    will reference the PPOD structures through its related procedure
    descriptor via the iopt field as usual.  The iopt field associated
    with the alternate entry point procedure descriptors will be zero.
    The PPOD structures associated with the alternate entry points must
    be accessed via the procedure descriptor associated with the main
    entry point.
    
    To determine the initial source location state for an alternate
    entry point, the initial state of the related main entry point
    is retrieved (e.g. the lower line number in the procedure descriptor,
    the source file associated with the main entry point etc.).  The
    PPOD structures associated with the main entry point are found
    via the iopt field of the procedure descriptor.  The PPOD structures
    are interpreted and the source state is updated until the current
    address is equal to the address specified in the alternate entry
    point's procedure descriptor.  At this point the initial source
    location is known for the alternate entry point.
    
    Note that this process is especially meaningful when there exist
    file boundaries amongst entry points.
    
    
    
    Bob
    
    
        
68.12SMURF::LOWELLTue Jun 03 1997 10:3717
By making alternate entries share their PPOD structures with the
main entrypoint we are establishing a precedent that we will
probably want to follow for anything else that is implemented
as optimization symbols.

(Ron, will this work for split-lifetimes etc.?  Or can we expect at
some point that alternate entries will have some of their own opt
symbols, but they always have to look to the main entry for extended
source line information?)

One small correction on your clarification, Bob.  The PDR iopt field
should be set to ioptNil, not zero.

Also, what did we decide to do with the lnLow field in alt entry PDR's?
We were planning on setting it to -1 right?  Will this be a problem
for debuggers that don't know about the new line number info?

68.13GEMGRP::MONTELEONETue Jun 03 1997 11:0315
    
    
    >>Also, what did we decide to do with the lnLow field in alt entry PDR's?
    >>We were planning on setting it to -1 right?  Will this be a problem
    >>for debuggers that don't know about the new line number info?
    
    
    No, we cannot set the lnLow field to -1, since the tools which use
    the old line number table information depend on the low line number
    value being accurate. The high line number will be -1, per usual.
    If we decide to remove the old line number table information at
    some point in the future, then we could set the lnLow field to -1.
    
    
    Bob
68.14No iopt's for alternate entries are forseenFLYBA::BRENDERRon BrenderThu Jun 05 1997 17:1420
Re .12:

>(Ron, will this work for split-lifetimes etc.?  Or can we expect at
>some point that alternate entries will have some of their own opt
>symbols, but they always have to look to the main entry for extended
>source line information?)

Good question! In fact, the split lifetime information by it's nature applies
to a complete routine. Where the entry points are and how many is not relevant.
Trying to subdivide the information in some way related to entries would only
complicate matters.

Similarly for the semantic stepping information; while that probably *could*
be subdivided at alternate entry point boundaries, there is no reason to
want to do so.

So, at present, I don't see any reason for an alternate entry point to want/
need its own iopt information. There is certainly a possiblility of doing
that in the future, I 'spose, but it doesn't appear on the horizon yet as
far as I can see...