|
I have attached a list of the trace flags and have someone looking into
the master password question. We will let you know soon.
Jeff
Technote: 10398
Subject: Trace flags, configuration flags, and general debugging info
This technote describes many of the trace and configuration flags useful
in debugging various problems and how to use them. The information
presented is relevant for release 2.0 and 3.0 although some is only
relevant for 3.0. At the end are several sections with other useful
debugging information.
Section Headers in this technote:
*** Trace Flags ***
*** Configuration Flags ***
*** Performance Flags ***
*** Environment Variables ***
*** Signaling a process to produce a backtrace ***
*** Distributed Hang Diagnosis ***
*** Translating Backtraces ***
*****************************************************************************
*** Trace Flags ***
*****************************************************************************
Trace flags can be set either through the FORTE_LOGGER_SETUP environment
variable, through the ModifyFlags method of the LogMgr object, or through
the Modify Log Flags command of the workspace workshop or the econsole
command. Trace flags always have 4 parts as in:
trc:os:1:1
The first part is the message which for trace flags is trc. This identifies
this flag as a trace flag as opposed to other types of flags (e.g.
configuration flags). Note that when using the Modify Log Flags commands
from the workshops the message type is Debug as opposed to trc. They both
mean the same thing.
The second part is the service for which to trace messages. This can be
any one of (this is not an exhaustive list):
c4 The 4GL system (TOOL compiler)
cf The partitioning system
cm The communication system
db The database system
do The distributed object system
ds The display system
em The environment management system
ex The Forte Express system
lo The object system
os The operating system
rp The repository system
rt The runtime system
sh The Forte runtime shell
sp The Forte runtime system
td The test driver
th The distributed test harness
ui The user interface system
User1 -> User10
Available for users.
The third and fourth parts are the group and level, respectively. Within
each service generally the various events or operations that can be traced
are specified by a group number between 1 and 63. Levels are generally used
to specify how much information is given, i.e. how verbose the trace is.
Level 1 gives the least amount of information (sometimes none), and level
255 gives the most. Usually not all levels are used.
Wildcards can be used for either group or level. For example trc:sh:6:*
produces all levels of transaction tracing, and trc:do:*:5 produces all
distributes trace messages at level 5. You can combine the functions and
levels to get more specific information.
Following are effects of individual trace flag settings. They are listed
in alphanumeric order:
trc:c4:2 Causes messages to be printed when c4 objects are logged to
the repository for changes. Valid levels are 1, 10 and 20.
trc:c4:3 Causes IR to be dumped during compiles. Valid levels are:
trc:c4:3:10 dump IR.
trc:c4:3:15 shows a block table.
trc:c4:3:20 shows debugger line information.
trc:c4:5 Traces IR and other information for a mapped class:
trc:c4:5:5 prints out informational messages
trc:c4:5:10 prints IR for access code in mapped classes
trc:c4:5:50-60 prints out tag information
trc:c4:6 Prints out information about SQL statements as
as they are parsed. The most interesting piece of
information is the SQL text that will be sent to the
database. Other stuff, less interesting to the user,
is also printed for the more complex statements.
trc:c4:7 Traces the compiler to print out sql statements are they are
compiled.
trc:c4:9 Traces messages about dependency managers notifications:
trc:c4:9:1 Traces messages about run-time compiles
trc:c4:9:10 Traces all notifications sent
trc:c4:9:50 Traces full dumps of the DM to be done at various times.
This should only be turned on before a showproj and then
turned off otherwise you will generate tons of output.
trc:c4:11 Traces changes to the extended properties for a component.
trc:c4:12 Traces changes to the relative ids of components.
trc:c4:40 Trace anything to do with repository versioning.
trc:c4:41 Trace updatework space operations:
1 - prints out name of class being updated and whether
a merge or a replace is being done.
25 - Prints out the components that are sending
invalidations during the update and whether they
are deletes or worst case.
50 - Prints out individual components changed during
the update and whether the change is from mainline
or the workspace.
50 - Prints out the dependency manager and the user
data for each project during the update.
trc:c4:50 Trace EOS compilation.
trc:cf:5 Logs information about all partitioning activities.
trc:cf:7 Logs information about partition related dependency management
activities.
trc:cf:10 Shows the 4GL code generated for each partition's startup.
trc:cf:15 Logs information about the activities related to making of
distributions.
trc:cf:20 Adds tracing to the partition startup code which shows when
service objects are being created and registered.
trc:cg:1:2 Traces names of methods being processed by codegen.
trc:cm Traces high level communications activities.
trc:cm:*:1 Very light trace about high level events
trc:cm:*:4 Traces packet level communications. Most useful to see
majority of CM activity, opens, closes, sends, recvs, etc.
trc:cm:*:8 Traces packet and serialization communications activities.
trc:cm:*:10 In addition to above, provides packet dumps in hex. Very
verbose, shows full blow by blow trace of serialization and
dumps formatted packet contents.
trc:cm:30:2 Can see the internet location (node, port) trying to connect
to. For example, set for a client that's having trouble
finding the name server.
trc:db:1 Cursor operations:
trc:db:1:1 Session level state transitions (e.g. OpenCursor)
trc:db:1:100 Vendor specific cursor transitions
trc:db:1:200 Checking cursor queues, fetching rows, cursor count
trc:db:2 Trace SQL commands:
trc:db:2:1 Shows all SQL statements processed by the database session.
trc:db:2:50 Shows input values for SQL statements.
trc:db:2:100 Vendor specific SQL statement processing.
trc:db:3 Trace SQL prepare statements:
trc:db:3:1 Trace statement text in Prepare and parse.
trc:db:3:100 Vendor specific statement text transformations.
trc:db:4:1 Trace SQL error translations using ANSI SQL state. Note all
rdbms vendors use SQL state.
trc:db:5 Transaction tracing:
trc:db:5:1 Session level transaction events: Begin, Commit, Abort.
trc:db:5:100 Vendor specific transaction events.
trc:db:6 DBResourceMgr events:
trc:db:6:1 Trace sessions and their agents being created in a
partition. Also trace parition lost events.
trc:db:6:100 Trace sessions being de-registered (going away) in a partition.
trc:db:6:200 Trace transaction state transitions (i.e. Commit, Abort).
Also trace sessions being registered in a partition.
trc:db:7 Trace locking done in DBSession. These are not DBMS locks but
are locks that internal code uses to maintain consistency of
internal state and data structures.
trc:db:8 Method tracing:
trc:db:8:1 DBSession methods.
trc:db:8:50 Other non-vendor specific methods.
trc:db:8:100 Vendor subclass methods
trc:db:9 Dataset memory use. The level indicates a threshold number
(1-100) of K bytes for tracing.
trc:db:9:1 Traces each block of memory allocated by a DB dataset over 1K
trc:db:9:50 Traces each item of memory allocated by a DB dataset over 50K
trc:db:9:N Traces each item of memory allocated by a DB dataset over NK
trc:db:10 Dataset events.
trc:db:10:1 SetMaxSize, SetMaxRows methods.
trc:db:10:200 Allocation requests
trc:db:11 Connection events for
trc:db:11:1 Connection events.
trc:db:11:100 Option list processing.
trc:db:12 Serialization tracing:
trc:db:12:1 Begin/end banner for dataset with summary info.
trc:db:12:100 Begin banner for column.
trc:db:12:200 Begin/end banner for column with summary info.
trc:db:31 Rdb Connection events.
trc:db:32 Rdb SQL events.
trc:db:32:1 commits and aborts.
trc:db:32:50 major sql operation
trc:db:32:100 progress reports on sql operations.
trc:db:32:200 errors and other annoyances, such as blobs.
trc:db:33 Rdb specific locking
trc:db:35 Rdb data handling.
trc:db:35:1 Special handling of input/output data.
trc:db:35:50 Normal data handling.
trc:db:35:100 Rdbms-specific bind and set operations.
trc:db:51 Sybase connection errors.
trc:db:63 Try block tracking - shows entry into TRY blocks.
trc:do Distributed object tracing. Helpful to figure out how the
client and server partitions are talking to each other.
trc:do:*:1 Non message summary information (partition/proxy
creation/destruction, exception events, etc).
trc:do:*:2 The same events as level 1, but more detail (the full details
of proxies print, for example). Particularly useful when
debugging exceptional events and you don't want tracing on
a per-message basis.
trc:do:*:5 Message activity summary. Most per-message message traffic is
printed
trc:do:*:7 Full per-message traffic is printed (like details of allocation
of message objects, etc).
trc:do:*:8 Serialization. The full details of message serialization are
printed. This is the most detailed tracing.
trc:do:1 Basic one-line trace per sigificant event (messages, partition
going away, etc).
trc:do:2 Tracing at the DOM interface.
trc:do:10-11 Partition manager. Show partitions and locations coming and
going. (11 prints the partition information, 10 prints only
the header).
trc:do:15 Proxy manager. Shows proxies coming and going. A proxy is a
control block that records a remote reference to an object.
This is useful (especially with function 3 above) when
resolving problems relating to binding and accessing remote
objects.
trc:do:17 Distributed Event Manager. Shows events raised across
partitions. See also trc:lo:1-2.
trc:do:20/21 Message tracing. Shows all activity related to specific
messages (sending, receiving, failing, etc). 21 shows the full
message dump, 20 does not.
trc:do:25 Message manager. Shows message allocation, freeing, and
message failure handling related to partition/task failures.
trc:do:30 OLS tracing. Shows the binding of names to object identifiers.
trc:do:35 Name Service. Shows client access to the name server.
trc:do:40 Serialization. The details of message serialization. A lot
of output. This must be combined with level 8.
trc:do:45 Load balanced router tracing. Shows all per-message router
processing in detail.
trc:do:51 Prints DOM dumps on partition destruction.
trc:em:12 Name Server tracing. The name server (part of the environment
manager) is in the em service.
trc:ex Traces for Express generated applications. Note: Express
objects initialize their trace flags when they are
instantiated with the trace settings currently in effect at
that time. Subsequent modifications of the trace flags for
the partition wont affect objects already in existance.
trc:ex:1 Traces text of SQL Select statements run by Express BusinessMgr
objects.
trc:ex:1:1 Traces SQL statement text only.
trc:ex:1:2 Traces SQL statement text and input values.
trc:ex:2 Traces text of SQL Insert/Update/Delete statements run by
Express BusinessMgr objects.
trc:ex:2:1 Traces SQL statement text only.
trc:ex:2:2 Traces SQL statement text and input values.
trc:ex:3 Traces Concurrency events by Express BusinessMgr objects.
trc:ex:3:1 Traces new clients connecting to the ConcurrencyMgr.
trc:ex:3:20 Traces rollback requests.
trc:ex:3:30 Traces commit requests.
trc:ex:3:40 Traces ready-to-commit requests.
trc:ex:3:50 Traces load requests (requests to add records to a transaction).
trc:ex:3:60 Traces start requests.
trc:ex:4 Traces Concurrency events by Express ConcurrencyMgr objects.
trc:ex:4:1 Traces new clients connecting to or disconnecting from the
ConcurrencyMgr.
trc:ex:4:20 Traces rollback requests.
trc:ex:4:30 Traces commit requests.
trc:ex:4:40 Traces ready-to-commit requests.
trc:ex:4:50 Traces load requests (requests to add records to a transaction).
trc:ex:4:60 Traces start requests.
trc:ex:5:1 Traces Select requests by Express BusinessClinet objects.
trc:ex:5:50 Traces BusinessClasses returned by Express BusinessClient
Select requests.
trc:ex:6 Traces BusinessClasses modified by Express BusinessClient
Update requests.
trc:ex:6:1 Traces BusinessClasses after modification.
trc:ex:6:100 Traces BusinessClasses before and after modification.
trc:ex:8 Traces Revert requests by Express BusinessClient objects.
trc:ex:9 Traces LogAttr requests by Express BusinessClient objects.
trc:ex:10 Traces Delete requests by Express BusinessClient objects.
trc:ex:11 Traces StartTrans requests by Express BusinessClient objects.
trc:ex:12 Traces EndTrans requests by Express BusinessClient objects.
trc:ex:13 Traces SaveForRevert requests by Express BusinessClient objects.
trc:ex:16 Traces new BusinessMgr or ConcurrencyMgr instantiations.
trc:in:1 Prints out information about the interpreter state object.
1 - Forces the interpreter to do tracing. If this
is not set when the state is created, the interpreter
will not trace for this state.
2 - Print IR blocks when they are setup.
5 - Print starts and exits of the interpreter.
10 - Print method entry/exit.
50 - Print method IR on each entry.
trc:in:2 Trace the interpretation of method invocations and
exception handling.
1 - Forces the interpreter to do tracing. If this
is not set when the state is created, the interpreter
will not trace for this state.
2 - Prints entry/exit and exceptions.
5 - Prints more details of exception handling.
10 - Prints method calls, event waits.
15 - Prints details of method calls.
20 - Prints a stack trace at all points above.
50 - Prints the offset of each IR statement executed.
trc:in:51:1 Traces method entry/exit by task
trc:in:52:1 Traces call tree by task
trc:in:53:1 Traces method entry/exit by application
trc:in:54:1 Traces call tree by application
trc:lo:5 Traces event postings that are ignored. The most common reason
being that no one is registered for them or hasn't registered
for them in time. This sometimes points to a timing window.
trc:lo:25 Displays all exceptions.
trc:os:1:1 Logs garbage collection activities.
trc:os:5:5 Used in conjunction with trc:os:1:1 will show all collected
types and sizes on stable collects.
trc:os:6:1 Trace alarm activity - Dangerous on the Mac and VMS, not
recommended.
trc:os:7:1 Trace deferred interupt activity - Dangerous on some ports,
not recommended.
trc:os:8:1 Trace all signals that occurred.
trc:os:9:1 Backtrace all exceptions. Useful for tracing down how an
exception occurred.
trc:os:10:1 Print out information about getenv, setenv calls.
trc:os:11:1 Print out tracing to see what the final formatted command will
be after calling
trc:os: OperatingSystem.RunCommand
trc:os:12:1 OS Wait/activity management trace. Used for communications
activity.
trc:os:13:1 Light thread tracing.
trc:os:14 Traces dynamic loading of shared libraries.
trc:os:15:1 Loop after a segv rather than exiting. Useful for attaching
with a debugger.
trc:os:18 Shows which directories are searched when doing an install and which
files it attempts to copy, if any (set on both the node manager with the
distribution and the target of the install).
trc:rp Logs all activities on the Repository.
trc:rp:2 Enables tracing of user actions that affect the repository.
Messages will be generated for the local client only.
Messages are the same format as those written to the
central repository operation (.ROP) file. There are two
useful "volume" levels: 50 and 75. 50 is normal and should
suffice for most occasions. Level 75 is quite verbose.
trc:rp:21 Starting with release 1.1.J.0, this trace logs performance
characterstics of the repository cache (known as the LOS, the
Local Object Space manager). See technical note FSIaa09343
for more details.
trc:rp:39 Trace operations at the storage manager access level.
In particular, trc:rp:39:1 will display messages about recovery
from Ctree and Btree before image logs.
trc:rp:41 Traces dependency manager notification operation.
1 - Funny situations that might or might not be
expected.
25 - Start of notificiations.
50 - Which dependents are notified.
150 - Each dependent as it is considered.
trc:rp:42 Traces dependency manager management operation (e.g. addition
and deletions from DM).
50 - Entry of new records, deletion of records, bit
setting
75 - Gotten bit values
100 - method entries
trc:rp:45-47 Traces Btree cache, before image logging, and logical I/O
activity
trc:rt:1 Traces dependency notifications to run-time classes
trc:rt:2 Traces dependency notifications to IR blocks
trc:rt:3 Causes an interpreted partition to print out the list of
projects which it contains. For each project, it prints the
project's name, runtime id, and absolute id. This trace flag
is useful in helping to decode trace output which contains
runtime project ids (such as is produced by most of the DO
tracing).
trc:sh:1:* Shell tracing, useful during startup and shutdown.
trc:sh:3:* Activity manager tracing. Shows OS/CM/SH interaction.
trc:sh:5:* task tracing. Fairly verbose, but gives a complete picture of
the task activity.
trc:sh:6:* High level transaction manager tracing.
trc:sh:7:* Transaction log manager tracing.
trc:sh:8:* Generic session operations.
trc:sh:9:* Transaction lock manager tracing.
trc:sh:10:* Traces dynamic loading of scopes.
trc:sh:11:* Durable lock tracing.
trc:sh:12:* Native language manager (2.0)
trc:sh:13:* Local descriptor tracing (2.0)
trc:sh:14:* Message catalog tracing (2.0)
trc:sh:15:* Codeset conversion operations. (2.0)
trc:sh:25:1 mutex tracing. Very verbose. Shows all mutex locks and
unlocks, condition variable activity is included in R2.
trc:ui:30:* Code generation tracing.
*****************************************************************************
*** Configuration Flags ***
*****************************************************************************
cfg:c4:2 Allows you to access to private components. In R2 allows you
access to INTERNAL components as well.
cfg:c4:9 Causes the showproject command and other commands to print all
scope ids as 0. This is useful for tests so that the results
are diff'able from run to run.
cfg:c4:10 Flags SQL statements that may not be ANSI 89 compatible.
cfg:c4:13 Causes only those components whose version state is NEW,
USER_BRANCH or CHECKOUT to be written to an export file during
an ExportProj command. Read-only components are not exported.
cfg:c4:14:1 Causes read-only components to be checked-out before they are
changed (useful on import for instance).
cfg:c4:14:2 Causes read-only components to be branched before they are
changed (again useful on imports).
cfg:c4:23 (R3) Allows you access to INTERNAL components
cfg:cf:21:1 Suppresses the status messages that are normally generated
during application partitioning.
cfg:db:1 Backward compatibility switch
cfg:db:1:1 Release 1.0 compatibility: For Oracle all numeric types other
than integer will be returned as DoubleData objects. In
release 2.0 we got smarter and returned DecimalData or
DoubleData as appropriate.
cfg:db:2 Adds more debugging info to some traces.
cfg:db:6:255 Allow commit/rollback executeImmediate statements (Oracle only)
cfg:db:8 Don't free the ODBC environment.
cfg:db:27 Issue generic ODBC messages for regression tests.
cfg:db:42 If set, the session runs with the 'SQLV40' dialect setting,
rather than the default 'SQL92' setting. Rdb only.
cfg:db:43 If set, Rdb transaction setting is "READ WRITE WAIT" rather
than the default "READ WRITE NOWAIT". Rdb only.
cfg:do:4 If set, events may be raised on message duration dialogs.
cfg:do:5 If set, if the initial connection to the name service fails,
the process is started anyway so that subsequent access to name
service is possible.
cfg:ds:1 Use expanded set of "sleeping" mouse cursors (showing garbage
collection and remote async waits as separate cursors).
cfg:em:1 Disables creation of icons during application installation.
Should be set on the client prior to running econsole, escript,
or iscript.
cfg:ex:1 Causes express exception messages to include a stack trace.
Should only be used when running interpreted code. Should
not be used for compiled code.
cfg:ex:2 Disables use of SQL statement caching by express BusinessMgr
objects.
cfg:in:3 If set to 1 then no quantums will be applied in the
interpreter. This limits the amount of multi-tasking.
cfg:os:5:5 Causes a detailed dump of all memory to be printed in the log
file when an out of memory exception is raised.
cfg:rp:17:255 Disables Ctree before image logging. (not recommended)
cfg:rp:17:1 Forces Ctree before image logging on, regardless of default
cfg:rp:18 Disables Ctree check that verifies that the data file, index
file, and before file all have the same timestamp and id.
This can be used to force a recovery that would otherwise
be aborted, or open a set of repository files that do not
appear to match (not recommended).
cfg:rp:19-23 DO NOT set these. They are used to test before image logging
cfg:rp:28 by aborting during various Btree and Ctree operations.
cfg:rp:26 Turns off printing of element ids in DM tracing.
cfg:rp:27:xx Set the number of 4k pages in the Ctree before image logging
cache to xx. The default is 20. Values below 10 may result in
a severe performance degradation. Values above 20 may yield 5%
performance increase if adequate memory is available. Note
that this and other Ctree config flags cannot be changed after
a repository has been opened; you must restart to modify them.
cfg:rp:31 Disables before image logging in Btree. Not recommended.
cfg:rp:32:xx Set the number of 4k pages in the Btree cache. Works just like
cfg:rp:27 above.
cfg:rp:37 (3.0.A.0 and later) Allows use of Btree repositories on
remotely mounted volumes (e.g. NFS). Not recommended or
supported.
cfg:rp:38 (3.0.C.0 and later) Use standard Btree format for rpclean
temporary repositories, instead of special "data inline" format.
Not recommended.
cfg:rt:1 Causes any scope ids in IR blocks to be printed with the value
0. This is useful for tests.
cfg:sm:1 Disables redirection of stdout and stderr. Normally the
nodemgr directs trace information into various files. With
this flag, all messages will go into one file.
cfg:sp:1:1 Causes system to run so regression tests don't diff
cfg:ui:3 Allows modification of Express generated TOOL code and disables
use of compiled templates during code generation.
cfg:ui:20 Enables the "Export as Template" command from the Repository
Workshop
*****************************************************************************
*** Performance Flags ***
*****************************************************************************
prf:c4:1 Causes the parser to print a few performance numbers.
prf:in:1 Causes the interpreter to print gross performance information.
*****************************************************************************
*** Environment Variables ***
*****************************************************************************
FORTE_NOHANDLER
When this is set, Forte exception and signal handling is disabled. On
windows, you will get a dialog with the address of the fault which can
be used to find the offensive line of code. On Unix, this will dump core,
which is useful when you can't get a good backtrace out of the system.
FORTE_XM_SINGLE_THREADED_CLIENT (release 3.0)
"true" causes the display system to be single threaded, "false" causes
the display system to be multi-threaded. The default is platform
specific, but usually "false".
*****************************************************************************
*** Signaling a process to produce a backtrace ***
*****************************************************************************
On UNIX, you can send a process a signal to get it to print a backtrace.
This is good to diagnose a hang or a spin. This is done with SIGUSR2 in the
following manner:
kill -USR2 process_id
*****************************************************************************
*** Distributed Hang Diagnosis ***
*****************************************************************************
Release 1.0 - Nscript
Start nscript and cd to the partitions directory. Do a directory listing
(dir or ls). Run debug <partition_id> for each partition number displayed
by the listing. This will send a message to each partition telling it to
dump its state to the log files in $FORTE_ROOT/log. More specifically, you
can cd into a partition directory where the location attributes (hostname,
ip address or decnet node, port number or object name) can be listed with a
"dir" command. This information can be used to correlated against information
from operating system utilities such as netstat.
Release 2.0 - escript
Start escript and type "dumpstatus". This will send system management agent
commands to each managed partition in the environment to dump the current
status. This information will go to the logs in $FORTE_ROOT/log.
Information dumped includes task manager, operating system, a task/thread,
event manager, Dom and CM dumps.
*****************************************************************************
*** Translating Backtraces ***
*****************************************************************************
OSF, SunOS, RS6000, and DG stack backtraces can be translated using a shell
script - /forte1/d/tools/bin/cmn/xlateTrc. Solaris and sequent support will
soon be added. HP dumps are translated by the OS at the time they occur.
Use xlateTrc in the following manner.
xlateTrc executable file_containing_dump
The exact same version of the binary and libraries must be used for the
translation as the ones that produced the trace.
|