Conference npss::gigaswitch

        Have you reported this through the normal channels?  (IPMT)
    
        Although you may get a response in a reasonable timeframe from
    noters, the traditional way of reporting problems will yield a more
    predictable response.
    
    I.E. When I have five problems sitting in front of me, the notes
    conference is at the bottom of my list.
    
    MDL
    GIGAswitch/FDDI product support

>n 14-Jan-1997, GS#1 couldn't reach out into GS#2, GS#3, and GS#4.
>But between FGL-2 ports within the GS#1 were able to communicate each other.
>It leads me both AGL-2 line cards has troubled at same time?.
    
      Which AGL-2 ports are connected to which GIGAswitch/FDDI systems?  Are
    the connections between GS#1-GS#2 and GS#1-GS#3 both on the same AGL-2
    card?   Please be a little more specific as to how the GIGAswitch/FDDI
    systems are connected together.
       
    
>The Firmware of AGL-2 and Clock module is 3.00, but the others are all 3.01
>Do you think it can cause the problem?

        No.
    
>Our GIGAswitch/FDDI's are core backbone, and under GIGAswitch/FDDI bunch of
>Cisco routers have been connected.  Reliability of GIGAswitch/FDDI is still
>very good to the customer(there was no problem happened so far almost 6
>monthes), so that they have schedule to replace Cisco into Digital products.
>Could you please understand its significance?  

        I'm not certain that I understand what you're asking here.
    
        The SCP error log entries are not particularly interesting in this
    case.  The FGL-2 in slot 12 appears to have been power cycled or crashed 
    a couple of days before the reboot - it may be related, although I have 
    no reason to expect it to be related.
    
        Since the GIGAswitch/FDDI system was rebooted, it's unlikely that
    we'll figure out what went wrong.  However, since it was a linecard
    which was having the problem, the linecard error logs should be
    examined.
    
        Linecard error logs (FGLs and AGLs) all may be obtained in the same
    general way (see below).  Each linecard has its own error log, which is
    separate from the other linecards.  FGL and AGL error logs are not
    cleared on reboots or firmware installations.
    
    
    ..to get the error log for the card in slot 9 from OBM, do the following:
    
    ...at the CLK> prompt, type control-e  (^e)
    
    ..and gather the following information:
    
CLK>Connect to linecard in what slot num ?9        !this may take a few minutes
LINECARD_DVT>

* FGL DVT Monitor V3.00  DEFAULTS - /HALT/PASS=1/INHIBIT_PASS/BELL/SKIP

Fgl_s09_p2> dump

No Errors...

Fgl_s09_p2> 

 ^d (to return to the CLK prompt)


    

Hello Michael Lyons,

I have escalated this prblem through IPMT today. But I think that it didn't
reach your group yet due to my escalation manager was not noticed yet.
I beleive that its problem will be informed your group very soon.  As it's
urgent case, I'd like to hear from you through this note instead. Would you
please help with reading below IPMT template attached?

Regards.
SJ.
---------------------------------------------------------------------------
    1. Detailed Problem Information.
       (Include: Problem Frequency, Symptoms, When did it Start?, Any changes
                 made before problem started, Can it be reproduced at will?)
       -----------------------------                 

       Symptom: There are four(4) GIGAswitch/FDDI spred over different
                buildings.  Each GIGAswitch/FDDI has six(6) FGL-2 and
                two(2) AGL-2 linecards.  Under those FGL-2 linecards,
                Cisco's brouters and DEChub 900 are hooked up, and throughout
                AGL-2 line cards four GIGAswitch/FDDI has connected as loop
                topology. More detail network diagram will be attached at last.

                Very intermittently and randomly GIGAswitch/FDDI can not
	        reach out to other GIGAswitch/FDDI's throughout AGL-2 link.
	        It means that FGL-2 line cards can talk each other within
                the GIGAswitch/FDDI, but can communicate with the other
                GIGAswitch/FDDI's when the problem happens. Linecard error log
                of AGL-2 will be attached at last.

                Once the problem happens, the problem lasts until the
                GIGAswitch/FDDI is rebooted.

       Problem frequency;
       When did happen?;

                These GIGAswitch/FDDI's have been installed last year around
                May,1996. ANd no this kinds of problem happend until last
                November, 1996. In other words, the problem has started
                from last november, but the problem interval was very
                long. Since February, 1997 the problem used to occure more
	        often, and recently it happens three or four time a week. 

       Any change made before the problem started?
       Can it be reproduced at will;

                They say that there is no any hardware and software changed
                at around of the first problem happened.
	        On May 13th, 1997, I upgraded the full firmware version into
                V310 from V300, but the problem still exists.


    2. Severity of impact to Customer.
       (Include: Current Customer situation and any workarounds in place.)
       -----------------------------

       GIGAswitch/FDDIs are the core network backbone to them. Under those
       FGL-2 linecards bunch of Digital Alpha systems and RAM manufacturing
       systems are running. Accordingly the reliability of GIGAswitch/FDDI
       should be very highly required. It goes without saying that their
       productivity of RAM manufacturing lines depends on these GIGAswitch.

       As its being very critical situation, temporarily I put FGL-2 links
       together with AGL-2 links which are currently operating. In other words,
       besides of AGL-2 links for connecting 4 GIGAswitch/FDDIs, FGL-2 links
       are also connected, and hope it to be a temporary workaround. 
                                                       	

    3. Hardware/Software configuration. 
       (Include: Revisions where appropriate. Also list any patches applied
                 and any third party software/hardware installed.)
       -----------------------------

	H/W CONFIGURATION       :

               -----------                   ---------
              | 6LINE-GSW |-----------------| RND-GSW |
              |           |-----------------|         |
               -----------                   ---------
                   | |                           | |
                   | |        ATM links          | |
                   | |                           | |
               ---------                     -----------
              | RND-GSW |-------------------| 4LINE-GSW |
              |         |-------------------|           |
               ---------                     -----------

              Four GIGAswitch/FDDI have connected via DUAL AGL-2 linecards.
              For example, in the RND-GSW has two AGL-2 linecards, two ATM
              links go to 6LINE-GSW, and the other two ATM links on the other
              AGL-2 linecard go to 4LINE-GSW. It's designed for redundancy
              purpose incase of one ATM link is failed. The Hunt Group and
              Reliability group are not applied.

              About the problem/symptom exampled on RND-GSW, FDDI communication
              throughout FGL-2 linecards works well only *WITHIN* the RND-GSW,
              but can not talk to other 4LINE-GSW, ENG-GSW, and 6LINE-GSW
              sometimes.


	S/W CONFIGURATION       :     

              All GIGAswitch/FDDI has the same configuration and same firmware
              as below;

                  GIGAswitch Slot Configuration
Slot Card Type        State         HW  FW                    Status
   1 -                -
   2 FGL-2            Powered Up    F    3.10                 Initialized
   3 FGL-2            Powered Up    F    3.10                 Initialized
   4 FGL-2            Powered Up    F    3.10                 Initialized
   5 AGL-2            Powered Up    E    3.10                 Initialized
   6 Switch Engine    Powered Up    H    3.10 DL1.00 BB3.10   Elected SCP
   7 Clock            Powered Up    D    3.00
   8 CBS              Powered Up
   9 Switch Engine    Powered Up    H    3.10 DL1.00 BB3.10   Backup SCP
  10 AGL-2            Powered Up    E    3.10                 Initialized
  11 FGL-2            Powered Up    F    3.10                 Initialized
  12 FGL-2            Powered Up    F    3.10                 Initialized
  13 FGL-2            Powered Up    F    3.10                 Initialized

 Debug Options Menu       1. Error log menu
                          2. Expanded SCP firmware versions
                          3. Expanded AGL firmware versions
                          4. Management memory dump
                          5. Linecard options menu
                          6. Monitor messages
                          7. Return to previous menu
 Choice: 2
SCP diagnostics (310):
Mon Jan 20 15:10:36 EST 1997 /scp-fw2/src/bootimage_sources.bb_rev310/src/dbgmo

SCP network download (100):
Wed Aug 4 17:58:23 EDT 1993 /two/scpdump/bootimage_sources.rev100/src/rom segoo

SCP operational (310):
Thu Jan 16 13:18:41 EST 1997 /scp-fw/src/bootimage_sources.op_rev310/src/snmp h

 Debug Options Menu       1. Error log menu
                          2. Expanded SCP firmware versions
                          3. Expanded AGL firmware versions
                          4. Management memory dump
                          5. Linecard options menu
                          6. Monitor messages
                          7. Return to previous menu
 Choice: 3
      
AGL extended firmware version
Slot    Card Type     FW
5       AGL-2         OP  3.10 BK 3.00 BB 1.00
10      AGL-2         OP  3.10 BK 3.00 BB 1.00

Currently a BK image can NOT be loaded if running BK firmware.

 AGL Firmware Options    1. Force load AGL BK image
 Menu                    2. Unforce load AGL BK image
                         3. Return to previous menu


	VERSION/PATCH APPLIED   :
	LAYERED PRODUCT &       :
	ANY OTHER VENDOR'S      :
	CONFIGURARTION          :


    4. Location of further available Information.
       (Include: Location of Crash Dumps, Network Topology, Log files etc.)
       -----------------------------  

Slot 5 AGL-2 linecards errolog in the 4LN-GSW;
==============================================
dump
Dump of Error Log:
Entry #      = 28
Entry status = 0  [0=valid, 1=wrt_err, 2=invalid, 3=empty, 4=crc_err]
Entry CRC    = BC245C72
ID/Rev       = 0531
Reset Count  = 0000
Timestamp    = 0000 0007 0D75
Write Count  = 0001
FRU Mask     = 0000
Test ID      = F310  -  Operational Firmware Error
Test Data    =
                0:00000118  1:00000000  2:00011D68  3:00002600
                4:0000808E  5:00000010  6:0000808E  7:00002100
                8:000FFF01  9:00840001 10:00840002 11:00842000
               12:008320A4 13:0009F79E 14:0009F79E 15:0009F59A
               16:BFE7BD21 17:0009F79E 18:000FFDF6 19:000FFD92
               20:0000808E 21:00000001 22:FFFFFFFF 23:FFFFFFFF
               24:FFFFFFFF 25:FFFFFFFF 26:FFFFFFFF 27:FFFFFFFF
               28:FFFFFFFF 29:FFFFFFFF 30:FFFFFFFF 31:FFFFFFFF
               32:FFFFFFFF 33:FFFFFFFF 34:FFFFFFFF 35:FFFFFFFF
               36:FFFFFFFF 37:FFFFFFFF 38:FFFFFFFF 39:3202000F

Entry #      = 27
Entry status = 0  [0=valid, 1=wrt_err, 2=invalid, 3=empty, 4=crc_err]
Entry CRC    = A72D6399
ID/Rev       = 0530
Reset Count  = 0000
Timestamp    = 0000 0029 D74A
Write Count  = 0001
FRU Mask     = 0000
Test ID      = F300  -  Operational Firmware Error
Test Data    =
                0:00000118  1:00000000  2:0001198E  3:00002600
                4:0000808E  5:00000010  6:0000808E  7:00002100
                8:000FFF01  9:00847004 10:00845000 11:00842000
               12:008320A4 13:0009F57E 14:0009F57E 15:00099D04
               16:0009F57E 17:D194FFFE 18:000FFE52 19:000FFDEE
               20:0000808E 21:00000001 22:FFFFFFFF 23:FFFFFFFF
               24:FFFFFFFF 25:FFFFFFFF 26:FFFFFFFF 27:FFFFFFFF
               28:FFFFFFFF 29:FFFFFFFF 30:FFFFFFFF 31:FFFFFFFF
               32:FFFFFFFF 33:FFFFFFFF 34:FFFFFFFF 35:FFFFFFFF
               36:FFFFFFFF 37:FFFFFFFF 38:FFFFFFFF 39:3202000F

Slot 10 AGL-2 linecard error log in the 4LN-GSW;
===============================================
Connect to linecard in what slot num ?10
LINECARD_DVT>
* AGL DVT Monitor V3.10DEFAULTS - /HALT/PASS=1/INHIBIT_PASS/BELL/SKIP
Agl_s10_p1> dump

  Dump of Error Log:

Entry #      = 31
Entry status = 0  [0=valid, 1=wrt_err, 2=invalid, 3=empty, 4=crc_err]
Entry CRC    = D1B62A49
ID/Rev       = 0510
Reset Count  = 0000
Timestamp    = 0000 0001 C3D8
Write Count  = 0001
FRU Mask     = 0000
Test ID      = F100  -  Operational Firmware Error
Test Data    =  
                0:00000118  1:00000001  2:0001189E  3:00002600
                4:000080CE  5:00000010  6:000080CE  7:00002100
                8:000FFF00  9:00847004 10:00845001 11:00842000
               12:008320A4 13:0009D8DE 14:0009D8DE 15:000FFE8B
               16:BF57F74A 17:0009D8DE 18:000FFDAE 19:000FFD4A
               20:000080CE 21:00000003 22:FFFFFFFF 23:FFFFFFFF
               24:FFFFFFFF 25:FFFFFFFF 26:FFFFFFFF 27:FFFFFFFF
               28:FFFFFFFF 29:FFFFFFFF 30:FFFFFFFF 31:FFFFFFFF
               32:FFFFFFFF 33:FFFFFFFF 34:FFFFFFFF 35:FFFFFFFF
               36:FFFFFFFF 37:FFFFFFFF 38:FFFFFFFF 39:0002000F

Entry #      = 30
Entry status = 0  [0=valid, 1=wrt_err, 2=invalid, 3=empty, 4=crc_err]
Entry CRC    = 1343C855
ID/Rev       = 0510
Reset Count  = 0000
Timestamp    = 0000 0001 C3D5
Write Count  = 0001
FRU Mask     = 0000
Test ID      = F100  -  Operational Firmware Error
Test Data    =  
                0:00000118  1:00000001  2:0001189E  3:00002600
                4:0000804E  5:00000010  6:0000804E  7:00002100
                8:000FFF00  9:00847004 10:00845001 11:00842000
               12:008320A4 13:0009D8DE 14:0009D8DE 15:00098000
               16:BF57F74A 17:0009D8DE 18:000FFDCE 19:000FFD6A
               20:0000804E 21:00000002 22:FFFFFFFF 23:FFFFFFFF
               24:FFFFFFFF 25:FFFFFFFF 26:FFFFFFFF 27:FFFFFFFF
               28:FFFFFFFF 29:FFFFFFFF 30:FFFFFFFF 31:FFFFFFFF
               32:FFFFFFFF 33:FFFFFFFF 34:FFFFFFFF 35:FFFFFFFF
               36:FFFFFFFF 37:FFFFFFFF 38:FFFFFFFF 39:0002000F

Entry #      = 29
Entry status = 0  [0=valid, 1=wrt_err, 2=invalid, 3=empty, 4=crc_err]
Entry CRC    = BCFAF7D1
ID/Rev       = 0510
Reset Count  = 0000
Timestamp    = 0000 0001 C3D2
Write Count  = 0001
FRU Mask     = 0000
Test ID      = F100  -  Operational Firmware Error
Test Data    =  
                0:00000118  1:00000001  2:0001189E  3:00002600
                4:000080CE  5:00000010  6:000080CE  7:00002100
                8:000FFF00  9:00847004 10:00845001 11:00842000
               12:008320A4 13:0009D8DE 14:0009D8DE 15:00098000
               16:BF57F74A 17:0009D8DE 18:000FFDD6 19:000FFD72
               20:000080CE 21:00000001 22:FFFFFFFF 23:FFFFFFFF
               24:FFFFFFFF 25:FFFFFFFF 26:FFFFFFFF 27:FFFFFFFF
               28:FFFFFFFF 29:FFFFFFFF 30:FFFFFFFF 31:FFFFFFFF
               32:FFFFFFFF 33:FFFFFFFF 34:FFFFFFFF 35:FFFFFFFF
               36:FFFFFFFF 37:FFFFFFFF 38:FFFFFFFF 39:0002000F

SCP error log of 4LN-GSW
========================
Log#    Date      Time     TimeSinceRestart   Reason

19    03/21/97  15:04:08    1:01:24:07.017   21     errorLog_gpi_Rq_DIC_Count_ZeroEACH 52ED5A551:59    0:00:00:11.890   70     ioctl ENETUNREACH 52ED5A5CPbackup
5     03/13/97  18:38:47    0:00:00:11.267   70     ioctl ENETUNREACH 52ED5A5:38:47    0:00:00:11.265   70     ioctl ENETUNREACH 2E2ED5A5

        Please provide the complete error logs for all cards, including:
    
    Error logs from every FGL
    Error logs from every AGL
    Complete SCP error log summary
    
        Do not edit out entries which appear to be uninteresting or
    unrelated.
    
        Monitor the following MIB objects and determine if there is a
    correspondence between changes in these MIB objects and the outages
    observed by the customers:
    
    dot1dStpTopChanges.0               (spanning tree topology changes)
    
    dot1dStpPortForwardTransitions.*   (* for all ports, including those
                                        which don't appear relevant to you.
                                        This is the number of times the
                                        port has entered forwarding from
                                        the learning state)
    
    ifLastChange.*                     (* for all ports
                                        This is the system time when the
                                        interface last started)
    
    MDL