| T.R | Title | User | Personal
Name | Date | Lines |
|---|
| 4576.1 | | MOLAR::YAHEY::BOSE | | Mon Mar 15 1993 10:52 | 7 |
|
This was due to a problem in interaction between Alarms FM and
SNMP AM. We are aware of the problem, however, the fix could not
make it into the SSB submission kit. The current plan is to have
it go into any V1.3 updates we might have in the future.
Rahul.
|
| 4576.2 | Can't wait until 'updates we might have in the future' | CSOADM::ROTH | ELVIS:: is alive... and reachable!! | Mon Mar 15 1993 11:56 | 11 |
| > The current plan is to have it go into any V1.3 updates we might have
> in the future.
I don't want to sound like a crybaby, but can I get a copy of the fix now?
Getting 15 rule firings each time I get a trap sent to me from an
unregistered SNMP box is giving me fits. Very soon I need to add even
more domains thus the problem will multiply.
Thanks,
Lee
|
| 4576.3 | Register it before configuring traps? | TOOK::R_SPENCE | Nets don't fail me now... | Mon Mar 15 1993 12:54 | 7 |
| Doesn't the box have to be explisitly told to send traps to the DECmcc
system?
If so, then register it before you configure the trap table for now.
s/rob
|
| 4576.4 | traps from box not always same IP source | CSOADM::ROTH | ELVIS:: is alive... and reachable!! | Mon Mar 15 1993 20:33 | 10 |
| I have Cisco boxes with multiple interfaces. Each interface has its own
IP address... some boxes have 10+ IP addresses.
At certain times I receive traps that have an IP address other that what
I was expecting. I don't want to have to register each and every
IP interface/address in the Cisco within DECmcc just to get around a
fluke in the code.
Lee (tired of trick workarounds and 'make do' with SNMP hacks)
|
| 4576.5 | don't need trick workarounds or SNMP hacks | MOLAR::PERRY | | Tue Mar 16 1993 13:58 | 51 |
| Lee,
I have included below a description, written by William Kwak, of how
wildcarded OCCURS alarm rules differ in V1.3 from V1.2. This
description was taken from QAR # 159 from the external mcc qar
database.
According to this description, you should not be seeing all your alarm
rules firing in all your domains when an unregistered entity generates
a trap (assuming you included the "for domain" clause in your alarm
rule definition). Only the rules in the domains in which the entity is
a member should be firing, therefore, unregistered entities shouldn't
cause any alarm rules to fire. If this is not the case, then let us
know since this would indicate that there is a bug in the ALARMS FM not
the SNMP AM. The SNMP AM filters on entity instance (and event type)
irregardless of domain membership. The SNMP AM does not care whether
an entity is registered or not when it gets a wildcarded getevent
request. For wildcarded getevent request, as long as the event passes
the event filter test, the SNMP AM will simply passes the event along.
Jim
*************************************************************************
>There has been a change in wildcarded OCCURS rule's behaviour in T1.3
>(V1.3).
>
>In DECmcc V1.2, the wildcarded OCCURS rules fire regardless of the
>domain membership of the entity.
>
>Suppose you have a rule expression:
> OCCURS( snmp * any event) for domain .foo
>
>When the Trap happens on entity called .ipbar, the Alarms rule did not
>check if the entity .ipbar is a member of domain .foo, and fires
>alarms. And all the domains to which ".ipbar" belongs received the
>rule firing notification.
>This was a BUG.
>
>In DECmcc V1.3 (T1.3.0), the Alarms FM makes sure that the entity
>belongs to the domain (i.e. the ".ipbar" has to be a member of the
>domain .foo in order for the above rule to fire).
>
>I'm closing this QAR.
>
>William 2/15/93
|
| 4576.6 | Still occuring... lots of details | CSOADM::ROTH | ELVIS:: is alive... and reachable!! | Mon Apr 05 1993 18:51 | 286 |
| > According to this description, you should not be seeing all your alarm
> rules firing in all your domains when an unregistered entity generates
> a trap (assuming you included the "for domain" clause in your alarm
> rule definition). Only the rules in the domains in which the entity is
> a member should be firing, therefore, unregistered entities shouldn't
> cause any alarm rules to fire. If this is not the case, then let us
> know since this would indicate that there is a bug in the ALARMS FM not
> the SNMP AM.
This is to 'let you know'.
- Attached are details from the notification log and getevent directive.
- Customer has pager interfaced to rule firings; gets flooded with pages
when this occurs.
- Customer's impression is that DECmcc is a piece of crap. Every version I
have installed in the past year has some new problem that is SNMP related.
This current problem as outlined in .0 started with T1.3.
Forgive me for the long posting... am trying to provide as much detail here
as possible.
Thanks-
Lee Roth
Okay, here is what I currently have:
SSB V1.3 BMS (Virgin install, no upgrade)
VMS V5.5
VS3100/76 -w- 32Mb (yes, a bigger platform is on order)
Local MIR
Batch job running so as to keep notifications going on a 24 hr basis.
Batch job looks like this:
$!batch job for rule enabling
$! sits in batch queue and runs all the time
$ set ver
$ set out=00:00:01
$ submit:= submit/noprint/user=mccuser/que=mcc$general/restart
$ submit/after="+8:" mcc_common:alarms_batch_enable !run 8 hrs later
$ purge/keep=30 alarms_batch_enable.log
$ manage/enterprise
set mcc 0 tcpip_am udp timeout = 15
set mcc 0 tcpip_am udp retries = 3
enable Domain * Rule *
!drop out after 8hr 5 mins
show domain * rule * all count, at start = +8:05
MIR organization:
Top level domain: 'State'
Next level domains: 6 'city' domains
3rd level domains: multiple 'building' domains. Two of the city domains
have buildings, other cities are empty
In the 'building' Some are empty, some have 1 SNMP entity, a few
domains: have multiple SNMP entities
Each 'building' domain has a series of rules
named using the 'domain name'_'rule name'
convention. The rules use the following expressions:
Expression = (OCCURS (SNMP * coldStart))
Expression = (OCCURS (SNMP * IP reachability DOWN))
Expression = (OCCURS (SNMP * linkDown), in domain xxx)
Expression = (OCCURS (SNMP * linkUp), in domain xxx)
Expression = (OCCURS (SNMP * warmStart))
!the 'in domain' part was added as an attempted
!workaround for this problem
Here is what I did to create/demonstrate the problem:
<<<<<<< NORMAL TRAP HANDLING >>>>>>>
1) Via the iconic interface, register an SNMP entity known as 'TEST' in one of
the 'building' domains. IP address is 156.63.64.63. Address is also placed
in UCX name database.
2) Issue GETEVENT directive for 'SNMP * ANY CONFIG EVENTS' and capture output
to a file
3) Create a new notification log file
4) Cause Cisco box 'TEST' to generate an SNMP LINKDOWN event and then an
SNMP LINKUP event.
This is (expected) result:
>>> GETEVENT INFO <<<
SNMP LOCAL_NS:.test
AT 5-APR-1993 15:38:17 CONFIGURATION EVENTS
Successfully received event:
Event: linkDown
A linkDown trap was received:
enterprise = "1.3.6.1.4.1.9.1.1"
agent-addr = 156.63.64.63
generic-trap = linkDown
specific-trap = 0
time-stamp = 862671
ifIndex = 4
SNMP LOCAL_NS:.test
AT 5-APR-1993 15:39:17 CONFIGURATION EVENTS
Successfully received event:
Event: linkUp
A linkUp trap was received:
enterprise = "1.3.6.1.4.1.9.1.1"
agent-addr = 156.63.64.63
generic-trap = linkUp
specific-trap = 0
time-stamp = 868681
ifIndex = 4
>>> NOTIFICATION LOG <<<
Event: critical SNMP test linkDown 5-APR-1993
15:38:20.42 Domain LOCAL_NS:.soc [47,208]
Alarm: critical SNMP test Rule soc_SNMP_LINKDOWN has fired
5-APR-1993 15:38:23.26 Domain LOCAL_NS:.soc [67,209]
Info1: The last event detected: SNMP LOCAL_NS:.test linkDown 5-APR-1993
15:38:17.62
Info2: Event: linkDown A linkDown trap was received: enterprise =
""1.3.6.1.4.1.9.1.1"" agent-addr = 156.63.64.63 generic-trap = linkDown
specific-trap = 0 time-stamp = 862671 ifIndex = 4
Info3: (OCCURS (SNMP * linkDown), in domain soc)
Event: clear SNMP test linkUp 5-APR-1993 15:39:19.34 Domain
LOCAL_NS:.soc [46,210]
Alarm: clear SNMP test Rule soc_SNMP_LINKUP has fired
5-APR-1993 15:39:21.73 Domain LOCAL_NS:.soc [67,211]
Info1: The last event detected: SNMP LOCAL_NS:.test linkUp 5-APR-1993
15:39:17.76
Info2: Event: linkUp A linkUp trap was received: enterprise =
""1.3.6.1.4.1.9.1.1"" agent-addr = 156.63.64.63 generic-trap = linkUp
specific-trap = 0 time-stamp = 868681 ifIndex = 4
Info3: (OCCURS (SNMP * linkUp), in domain soc)
<<<<<<< THIS IS THE FAILURE MODE >>>>>>>
1) Delete the SNMP entity 'TEST' using the 'double slash' icon in the toolbox.
2) Cause the Cisco to generate the same 'link down' and 'link up' traps
as done previously.
This is the (unexpected, undesired) result:
>>> GETEVENT INFO <<<
!same traps as before
SNMP LOCAL_NS:.test
AT 5-APR-1993 15:47:18 CONFIGURATION EVENTS
Successfully received event:
Event: linkDown
A linkDown trap was received:
enterprise = "1.3.6.1.4.1.9.1.1"
agent-addr = 156.63.64.63
generic-trap = linkDown
specific-trap = 0
time-stamp = 916709
ifIndex = 4
SNMP LOCAL_NS:.test
AT 5-APR-1993 15:48:28 CONFIGURATION EVENTS
Successfully received event:
Event: linkUp
A linkUp trap was received:
enterprise = "1.3.6.1.4.1.9.1.1"
agent-addr = 156.63.64.63
generic-trap = linkUp
specific-trap = 0
time-stamp = 923719
ifIndex = 4
>>> NOTIFICATION LOG <<< !note that no 'EVENT' was reported
Alarm: critical SNMP LOCAL_NS:.test Rule vitalink_SNMP_LINKDOWN has
fired 5-APR-1993 15:47:22.32 Domain LOCAL_NS:.vitalink [67,212]
Info1: The last event detected: SNMP LOCAL_NS:.test linkDown 5-APR-1993
15:47:18.02
Info2: Event: linkDown A linkDown trap was received: enterprise =
""1.3.6.1.4.1.9.1.1"" agent-addr = 156.63.64.63 generic-trap = linkDown
specific-trap = 0 time-stamp = 916709 ifIndex = 4
Info3: (OCCURS (SNMP * linkDown), in domain vitalink)
!this is repeated multiple times, except that domain name differs...
!excerpts from log detail data:
Alarm: critical SNMP LOCAL_NS:.test Rule vitalink_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule cin_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule sur_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule rif_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule col_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule soc_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule bmvx_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule tolct_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule tol_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule akr_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule noeca_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule rho_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule odb_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule obs_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule sys_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule str_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule nwoca_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule arl_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule mrd_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule cle_SNMP_LINKDOWN has fired
! Same thing happens with LINK Up trap... a plethora of rule firings from
! other domains with SNMP entities in them.
=================================
Then I tried this:
1) Delete batch job that keeps rules alive
2) Enable rules in domain that did have the entitiy 'TEST' in it via the
iconic interface
3) Enable rules in two other domains via the IMPM
4) Created the linkdown & linkup traps
Result: Linkdown & linkup rule firings from all 3 domains.
=================================
Then I tried this:
1) DISable rules in domain that did have the entitiy 'TEST' in it via the
iconic interface
2) Leave rules enabled in two other domains
3) Created the linkdown & linkup traps
Result: Linkdown & linkup rule firings from the domains with the rules left
enabled.
=================================
Then I tried this:
1) Shutdown icon interface altogether.
2) Restart iconic interface !no rule keepalive batch job running
3) $ MAN/ENT GETEVENT SNMP * ANY CONFIG EVENT, FOR DURATION xxx
4) Force SNMP traps to occur
Result: GETEVENT directive 'sees' traps but nothing happens with notifications
5) Now enable rules a single domain via icon interface
6) Force SNMP traps to occur
7) Get rule firing notifications from that domain
|
| 4576.7 | Domain name is specified in rule name hierarchy | TOOK::KWAK | | Tue Apr 06 1993 17:36 | 31 |
|
RE: .6
In reply .5, the domain membership relationship is between
the domain name in the rule specification and entity name in
the expression.
Suppose you create a rule:
MCC> create domain .soc rule rule1 -
expression = (occurs(snmp * linkDown)), .....
The domain membership means that the SNMP entity is
(in your case ".test") in the domain ".soc".
In your expression
Expression = (OCCURS (SNMP * linkDown), in domain xxxx)
the "in domain xxxx" is ignored.
The domain is specified in your rule name hierarchy or
"in domain" qualifier as following:
MCC> create mcc 0 alarms rule myrule -
expression = (...............), -
descrip = "..........",
in domain .soc
^^^^^^^^^^^^^^
In your test case, when the linkDown and linkUp events will cause
all the rules in the domain containing the SNMP entity
"LOCAL_NS:.test" to fire.
William
|
| 4576.8 | | MOLAR::PERRY | | Thu Apr 08 1993 10:50 | 18 |
|
> - Customer's impression is that DECmcc is a piece of crap. Every version I
> have installed in the past year has some new problem that is SNMP
> related.
Lee, please try to keep your criticism constructive. Most of us,
myself included, take offense to something we've worked very hard
on being called "a piece of crap". Besides, you probably find that
we are a lot more cooperative if you try to work with us to
improve the DECmcc product.
Also, I'll try and restate this in terms that hopefully you can
understand ... This is *NOT* an SNMP related problem. You haven't
defined your alarm rules properly (RE: -1).
Jim
|
| 4576.9 | source of confusion located!! | CSOADM::ROTH | you just KEEP ME hangin' on... | Tue Apr 13 1993 11:25 | 97 |
| .8>Lee, please try to keep your criticism constructive. Most of us,
.8>myself included, take offense to something we've worked very hard
.8>on being called "a piece of crap". Besides, you probably find that
.8>we are a lot more cooperative if you try to work with us to
.8>improve the DECmcc product.
We're on the same team. Really. I'll respond to this shortly in a
separate reply.
.8>Also, I'll try and restate this in terms that hopefully you can
.8>understand ... This is *NOT* an SNMP related problem. You haven't
.8>defined your alarm rules properly (RE: -1).
.8>
.8>Jim
Well, in my mind I have agreed with that all along. I have gone *crazy*
trying to find out what has happened along the way... and I have finally
figured it out...
Originally, I posted the text of .0 as QAR161. A reply to QAR161 made the
following suggestion:
QAR161>> To prevent this rude behavior, do I have to have the expression
QAR161>> contain the domain name, e.g.
QAR161>>
QAR161>> Expression = OCCURS (DOMAIN ovakr SNMP * linkUp))?
QAR161>>
QAR161>
QAR161> Yes, but the syntax is actually:
QAR161>
QAR161> Expression = OCCURS( snmp * linkup) for domain ovakr
Well I tried that and it did not work... "missing right paren" was the
result. So I posted a note here stating my problem (note 4620.x). Reply
4620.2 fixed my "missing paren" problem but subtly 'switched my train'
onto a different track. I did not see the difference until today:
================================================================================
Note 4620.2 Missing right paren (again) 2 of 3
HADRES::KRAUSE "European NewProductEngineer for MCC" 10 lines 2-MAR-1993 12:08
-< Try correct syntax... >-
--------------------------------------------------------------------------------
> Create Domain LOCAL_NS:.ovakr Rule ovakr_SNMP_LINKUP -
> Expression = (OCCURS (SNMP * linkUp) for domain ovakr), -
How about this one?
Create Domain LOCAL_NS:.ovakr Rule ovakr_SNMP_LINKUP -
Expression = (OCCURS (SNMP * linkUp), in domain ovakr)
------------------------------------------------^^^^
*Robert
A windows cut/paste of the text above and viola!, my "missing paren"
problem had been cleared up. At that point I believed that I was
implementing the suggestion of QAR161 to the letter. When I still had the
problem I continued to reply to this note... still not realising the
change that 4620.2 had made... "in domain" instead of "for domain".
Reading reply .7 got me even more confused- I could swear I was already
doing that!!
So I sat down this morning and carefully studied this note string, note
4620.x and QAR161 and found out what went wrong.
So now, I am going to try the rule format suggested in reply .7, e.g.
.7> The domain is specified in your rule name hierarchy or
.7> "in domain" qualifier as following:
.7> MCC> create mcc 0 alarms rule myrule -
.7> expression = (...............), -
.7> descrip = "..........",
.7> in domain .soc
.7> ^^^^^^^^^^^^^^
But this brings up a question: Are there TWO correct ways to implement this??
WAY #1:
The rule format that I have now (as posted in .0) and implement the
'for domain' as suggested in QAR161 (and posted in note 4620.0)
WAY #2:
The format suggested in .7, i.e. the "CREATE MCC 0" and add the "IN
DOMAIN" qualifier
At this point I am going to implement whatever works, period, but I
want to know if the suggestion in QAR161 was valid or not.
Thanks and many apologies to all,
Lee
|
| 4576.10 | Unfortunately there are two ways :( | MOLAR::DFLAT::PLOUFFE | Jerry | Tue Apr 13 1993 15:25 | 19 |
| Lee:
> But this brings up a question: Are there TWO correct ways to implement this??
Yes, unfortunately there are two ways to specify alarms rules in DECmcc.
It is regrettable that this situation has occurred. It only serves to verify
the fact that DECmcc's user interface is hard to use.
Solutions to this problem have been proposed for future releases of DECmcc
(i.e., Relationships).
Please send mail to Jim Swist, Erik Mintz, Jim Lemmon and Dan Carr to relate
your frustration and confusion. With people like you lobbying maybe this
problem can be resolved in future releases of the product.
- Jerry
P.S. All these people are on TOOK::
|
| 4576.11 | still floundering | CSOADM::ROTH | you just KEEP ME hangin' on... | Wed Apr 14 1993 17:51 | 49 |
| .8>Also, I'll try and restate this in terms that hopefully you can
.8>understand ... This is *NOT* an SNMP related problem. You haven't
.8>defined your alarm rules properly (RE: -1).
I'm trying... really. But snippets of suggested formats aren't working
for me, I'm afraid. I need some WORKING examples.
Challenge/request #1:
Post WORKING EXAMPLES of (or proper pointers to):
A rule in the "CREATE MCC 0 ALARMS RULE xxx" format that implements
the "IN DOMAIN xxx" suggestion in reply .7 of this note.
I've tried again and again and again to implement a rule using the
"CREATE MCC 0 ALARMS RULE xxx" syntax, but it must require secret
knowledge or wizardry... it simply keeps blowing up over and over because
it cannot parse what I am giving it. The only reference I can find to
this format is in the examples located in appendix A in the Alarms
manual, but the example must be BOGUS. Also, the example has no provision
for defining SEVERITY or the BATCH QUEUE to use.
I've even looked in the 'sample alarms' .COM that comes with DECmcc...
nothing uses this "CREATE MCC 0 ALARMS RULE xxx" construct.
The alarms manual (pg. 5-32) would lead you to believe that the
MCC_ALAMRS_EXTRACT_RULES.COM will do the trick ("... It reconstructs all
the CREATE MCC 0 ALARMS RLULE directives...") but, it outputs
in the "DOMAIN <domain_name> RULE <rule_name>" format.
Challenge/request #2:
Post WORKING EXAMPLES of (or proper pointers to):
A rule in the "CREATE DOMAIN <domain_name> RULE <rule_name>" that
implements the suggested "FOR DOMAIN XXX" addition (from QAR161).
I've tried various insertions of commas and parens to no avail. I've
looked through the Alarms manual but can find no reference to the
"FOR DOMAIN" business.
All in all, I've had lots of pointers saying "your rules are screwed up"
and some offhand suggestions of how to fix them; but no clear, working
examples.
Lee
|
| 4576.12 | #1 is easy. | MCDOUG::doug | Dead or Canadian? | Wed Apr 14 1993 18:19 | 129 |
| >I'm trying... really. But snippets of suggested formats aren't working
>for me, I'm afraid. I need some WORKING examples.
>
>Challenge/request #1:
>
>Post WORKING EXAMPLES of (or proper pointers to):
>
> A rule in the "CREATE MCC 0 ALARMS RULE xxx" format that implements
> the "IN DOMAIN xxx" suggestion in reply .7 of this note.
>
Guess I don't understand the problem... works for me.
I have no problems with the syntax specified by William.
re: challenge #2
I'll try that later.
(logfile of real bona fide MCC FCL session attached)
!
!
create mcc 0 alarms rule foo1 expression = (occurs (collector * any event)), descri="blah blah", in domain .consoles
!
!MCC 0 ALARMS RULE foo1
!AT 1993-04-14-17:06:44.906
!
!Entity created successfully.
!
sho mcc 0 alarm rule * all attr
!
!MCC 0 ALARMS RULE *
!AT 1993-04-14-17:06:57.691 All Attributes
!
!No rules have been created.
!
sho mcc 0 alarm rule * all attr, in domain .consoles
!
!MCC 0 ALARMS RULE foo1
!AT 1993-04-14-17:07:18.629 All Attributes
!
! NAME = foo1
! State = Disabled
!The rule MCC 0 ALARMS RULE foo1 was not enabled, and therefore has no counters.
!
! Expression = (occurs (collector * any event))
! Description = "blah blah"
! Perceived Severity = Indeterminate
! Probable Cause = Unknown
!
!
ena mcc 0 alarm rule foo1
!
!MCC 0 ALARMS RULE foo1
!AT 1993-04-14-17:07:51.094
!
!No such entity: MCC 0 ALARMS RULE foo1
! Unknown Entity = MCC 0 ALARMS RULE foo1
!
ena mcc 0 alarm rule foo1, in domain = consoles
!
!MCC 0 ALARMS RULE foo1
!AT 1993-04-14-17:08:00.141
!
!Normal operation has begun.
!
enable mcc 0 collec sink udpip
!
!MCC 0 COLLECTION_AM SINK UDPIP
!AT 1993-04-14-17:08:22.344
!
!Enable completed successfully.
!
register collector bogus
!
!Collector LOCAL_NS:.bogus
!AT 1993-04-14-17:08:31.934
!
!Registration successful.
!
sho mcc 0 alarm rule * all attr, in domain .consoles
!
!MCC 0 ALARMS RULE foo1
!AT 1993-04-14-17:11:57.199 All Attributes
!
! NAME = foo1
! State = Enabled
! Substate = Running
! Time of Last Evaluation = 1993-04-14-17:11:44.000
! Result of Last Evaluation = True
! Current Severity = Indeterminate
! Creation Timestamp = 1993-04-14-17:08:00.156
! Evaluation Error = 0
! Evaluation True = 2
! Evaluation False = 0
! Expression = (occurs (collector * any event))
! Description = "blah blah"
! Perceived Severity = Indeterminate
! Probable Cause = Unknown
!
notify domain .consoles
!%MCC-S-NOTIFSTART, Notify request 1 started
!
!!!!!!!!!!!!!!! Alarm, 1993-04-14-17:12:14. !!!!!!!!!!!!!! [1]
!Domain: LOCAL_NS:.consoles Severity: Indeterminate
!Notification Entity: Collector LOCAL_NS:.system_1
!Event Source: Domain LOCAL_NS:.consoles Rule foo1
!Event: OSI Rule Fired
!
! Event Type = QualityofServiceAlarm
! Event Time = 1993-04-14-17:12:14.164
! Probable Cause = Unknown
! Additional Info = { (
! significance = True,
! information = "The last event detected: Collector LOCAL_NS:.system_1 General Event
! 1993-04-14-17:12:13.422" ),
! (
! significance = True,
! information = "Event: General Event ""test"" Event Text = ""1-2-3""" ),
! (
! significance = True,
! information = "(occurs (collector * any event))" ) }
! Managed Object = Collector LOCAL_NS:.system_1
! Perceived Severity = Minor
! Additional Text = "blah blah
! "
!
!
exit
!
|
| 4576.13 | LIVE examples | CTHQ::WOODCOCK | | Wed Apr 14 1993 18:23 | 39 |
| Examples that work for us:
Create Domain .systems Rule lh_reach -
Expression = (node4 * buffer size<500,at every 0:10:0), -
Severity = Minor, -
Alarm Fired Procedure = DISK$MCC:[MCC.COM]NODE_DOWN.COM;1, -
Alarm Exception Procedure = DISK$MCC:[MCC.COM]NODE_DOWN.COM;1, -
Batch Queue = "mcc$batch"
Domain .systems has 24 microvax's in it which we simply poll for reachability.
.............................................................................
Create Domain .MKO-3 Rule poll_MKO-3 -
Expression = (node4 * circuit syn-* substate <> none,at every 0:10:0), -
Severity = Critical, -
Alarm Fired Procedure = DISK$MCC:[MCC.COM]CKT_DOWN.COM;1, -
Alarm Exception Procedure = DISK$MCC:[MCC.COM]NODE_DOWN.COM;1, -
Batch Queue = "mcc$batch"
Domain .mko-3 contains all routers we manage in it and this rule ensures all
routers and circuits are up.
.............................................................................
create mcc 0 alarms rule poll_MKO-3 -
expression=(node4 * circuit syn-* substate <> none,-
at every 0:10:0),-
procedure=[mcc.com]ckt_down.com;,-
exception handler=[mcc.com]node_down.com;,-
queue=mcc$batch,-
perceived severity=critical,in domain .MKO-3
This is the same rule but in the former syntax. Note that all these rules
all polling FORWARD in time. We've never used FOR DOMAIN in any rule to date.
best regards,
brad...
|
| 4576.14 | Don't believe everything you read | TOOK::GUERTIN | MCC: Legend or Nightmare? | Thu Apr 15 1993 09:15 | 14 |
| In have NEVER EVER heard of FOR DOMAIN -- IN MY LIFE!!! until now.
And I have worked on MCC virtually since its inception almost 7 years
ago. I would assume that whoever replied to that qar had too much
coffee or has a metal plate in their head and was sitting too close to
a power supply -- or both. It is invalid syntax anyway, the FOR
qualifier is only used for Time specifications. Do not take
everything you read as gospel. Especially if the person who is
responding to you doesn't know how to help you (they may be simply
trying to move that qar off of their desk). Several people replied to
you, stating, "this works." I would believe them. Only one human
being in the entire universe stated that FOR DOMAIN works and he/she
has been very very quiet.
-Matt.
|
| 4576.15 | | TRM::KWAK | | Thu Apr 15 1993 10:16 | 22 |
|
I guess that I was assuming too much knowledge of readers on the
rule expression syntax when I answered the QAR #159 (MCC013_EXT).
In the QAR, the rule name was "domain LOCAL_NS:.ovohio Rule
SNMP_LINKDOWN", and I assumed that the "for domain" clause in the QAR's
answer was interpreted, by the readers, as the domain name in the rule name.
(
The part of text in the QAR's answer contains the following text:
>Suppose you have a rule expression:
> OCCURS( snmp * any event) for domain .foo
)
However, the "for domain .foo" seemed to be literally taken as a part of
rule expression syntax:
I guess I should have written the answer as following:
Suppose you have a rule expression:
"OCCURS( snmp * any event)" for the domain .foo
|
| 4576.16 | clarification | CSOADM::ROTH | you just KEEP ME hangin' on... | Fri Apr 16 1993 10:07 | 67 |
| Well, .12, you get partial credit. Or maybe I didn't specify things well
enough. I need to specify ALL of the various 'rule trappings' in the
"CREATE MCC 0 ALARMS RULE xxx" format...
This should give you an idea of what I mean...
$ man/ent
DECmcc (V1.3.0)
!
!
!Okay, let's see if we can successfully use the example in A-10
!of the DECMCC V1.3 Alarms manual, 'Alternate Format'. This is (nearly)
!verbatim...
!
CREATE MCC 0 ALARMS RULE BRIDGE_RESETS -
expression = (CHANGE OF(BRIDGE DPW UNSOLICITED RESETS ,-
*, *), AT EVERY 00:15:00) ,-
procedure = MCC_COMMON:MCC_ALARMS_BROADCAST_ALARM.COM ,-
exception handler = MCC_COMMON:MCC_ALARMS_BROADCAST_EXCEPTION.COM ,-
category = "Bridge Reset Problem" ,-
description = "blah blah blah" ,-
alarm fired parameter = "USER=ROTH" ,-
IN DOMAIN SOT
%MCC-I-NOPARCMD, alarm fired parameter = "USER=ROTH"
^
%MCC-I-SYNTAXERR, Syntax error -- unable to interpret remainder of line
%MCC-W-QUALUNKNOWN, qualifier is unknown: ALARM
!Well, so much for the alarm parameter. Let's try to specify the batch
!queue instead...
!
CREATE MCC 0 ALARMS RULE BRIDGE_RESETS -
expression = (CHANGE OF(BRIDGE DPW UNSOLICITED RESETS ,-
*, *), AT EVERY 00:15:00) ,-
procedure = MCC_COMMON:MCC_ALARMS_BROADCAST_ALARM.COM ,-
exception handler = MCC_COMMON:MCC_ALARMS_BROADCAST_EXCEPTION.COM ,-
category = "Bridge Reset Problem" ,-
description = "blah blah blah" ,-
batch queue = ALARMS$BATCH ,-
IN DOMAIN SOT
%MCC-I-NOPARCMD, batch queue = ALARMS$BATCH
^
%MCC-I-SYNTAXERR, Syntax error -- unable to interpret remainder of line
%MCC-W-QUALUNKNOWN, qualifier is unknown: BATCH
!Didn't like the batch queue syntax eh? How about severity level then...
!
!
CREATE MCC 0 ALARMS RULE BRIDGE_RESETS -
expression = (CHANGE OF(BRIDGE DPW UNSOLICITED RESETS ,-
*, *), AT EVERY 00:15:00) ,-
procedure = MCC_COMMON:MCC_ALARMS_BROADCAST_ALARM.COM ,-
exception handler = MCC_COMMON:MCC_ALARMS_BROADCAST_EXCEPTION.COM ,-
category = "Bridge Reset Problem" ,-
description = "blah blah blah" ,-
severity = CRITICAL ,-
IN DOMAIN SOT
%MCC-I-NOPARCMD, severity = CRITICAL
^
%MCC-I-SYNTAXERR, Syntax error -- unable to interpret remainder of line
%MCC-W-QUALUNKNOWN, qualifier is unknown: SEVERITY
!give up
exit
$ exit
|
| 4576.17 | This example is a hybird of the two Alarms Interfaces | MOLAR::ROBERTS | Keith Roberts - Network Management Applications | Fri Apr 16 1993 10:44 | 46 |
|
RE: .16
Lee,
The example you have entered is a Hybird of the two Alarms
Interface .. yuck .. Also, the Change Of function has an
underscore in it: CHANGE_OF.
So...
>CREATE MCC 0 ALARMS RULE BRIDGE_RESETS -
> expression = (CHANGE OF(BRIDGE DPW UNSOLICITED RESETS ,-
> *, *), AT EVERY 00:15:00) ,-
> procedure = MCC_COMMON:MCC_ALARMS_BROADCAST_ALARM.COM ,-
> exception handler = MCC_COMMON:MCC_ALARMS_BROADCAST_EXCEPTION.COM ,-
> category = "Bridge Reset Problem" ,-
> description = "blah blah blah" ,-
> alarm fired parameter = "USER=ROTH" ,-
> IN DOMAIN SOT
Should be:
create mcc 0 alarms rule bridge_resets -
expr = (change_of(bridge dpw unsolicited resets,*,*)), -
proc = mcc_common:mcc_alarms_broadcast_alarm.com, -
excep hand = mcc_common:mcc_alarms_broadcast_exception.com, -
category = "Bridge Reset Problem", -
description = "blah blah blah", -
parameter = "user=roth", -
in domain sot
The 'mcc 0 alarms rule' interface uses 'parameter', while the
'domain rule' interface uses 'alarm fired parameter'
I don't like the differences, and the people that introduced them
should apologize ... while you're waiting for that, could you please
file a QAR on the Alarms Manual describing the problems in the
examples.
Thanks /keith
|
| 4576.18 | Treasure hunt for non-existant treasure | CSOADM::ROTH | you just KEEP ME hangin' on... | Fri Apr 16 1993 16:15 | 67 |
|
I finally got the syntax problems with the "MCC 0" format solved.
This is what the "MCC 0 ALARMS" format rule looks like:
!includes the 'in domain' clause
create mcc 0 alarms rule soc_SNMP_LINKUP -
expr = (OCCURS (SNMP * linkUp)), -
proc = _DKA700:[ALERT]ALARM_PARSE.COM, -
excep hand = "_DKA700:[MCC]MCC_ALARMS_LOG_EXCEPTION.LOG", -
category = "SNMP_LNKUP",-
description = "A link on a Vitalink or Cisco box has come back up", -
queue = alarms$batch,-
perceived severity = clear,-
in domain .soc
Now here is the rule format I have been using all along (i.e. what I
had to start with when I began this topic):
!
Create Domain LOCAL_NS:.soc Rule soc_SNMP_LINKUP -
Expression = (OCCURS (SNMP * linkUp), in domain soc), -
Severity = Clear, -
Category = "SNMP_LNKUP", -
Description = "A link on a Vitalink or Cisco box has come back up", -
Alarm Fired Procedure = _DKA700:[ALERT]ALARM_PARSE.COM, -
Alarm Exception Procedure = _DKA700:[MCC]MCC_ALARMS_LOG_EXCEPTION.COM, -
Alarm Fired Parameters = "_DKA700:[MCC]MCC_ALARMS_LOG_EXCEPTION.LOG", -
Batch Queue = "ALARMS$BATCH"
Don't these two do the same thing?
MOLAR::PERRY writes:
.5>According to this description, you should not be seeing all your alarm
.5>rules firing in all your domains when an unregistered entity generates
.5>a trap (assuming you included the "for domain" clause in your alarm
.5>rule definition).
TRM::KWAK writes:
.15>However, the "for domain .foo" seemed to be literally taken as a part of
.15>rule expression syntax:
.15>
.15>I guess I should have written the answer as following:
.15> Suppose you have a rule expression:
.15> "OCCURS( snmp * any event)" for the domain .foo
So may I correctly concur that
1) There is no "for domain" clause to be added to my existing rule to
correct the multiple-rule firing problem.
2) It only seemed that a "for domain" clause existed because of
MOLAR::PERRY's interpretation (and emphasis) of TRM::KWAK's
QAR159 text.
Thus, I have been of a frustrating exercise to implement a 'fix'
that doesn't even exist!! I have been busy trying to figure out
every nuance of of the "CREATE MCC 0 ALARMS" format syntax thinking that
I was going to somehow achieve a working rule that would implement the
mythical "for domain" clause.
Lee
|
| 4576.19 | Back to square one | CSOADM::ROTH | you just KEEP ME hangin' on... | Fri Apr 16 1993 16:22 | 7 |
|
Well, I implemented all of my SNMP 'link down' and 'link up' rules
using the "CREATE MCC 0 ALARMS" format with the "IN DOMAIN .<domain
name>" and I STILL HAVE THE SYMPTOMS THAT I DETAILED IN THE BASENOTE,
I.E. a trap from an unregistered box yields multiple rule firings!!
What do I try next?
|
| 4576.20 | | TOOK::R_SPENCE | Nets don't fail me now... | Wed Apr 21 1993 16:23 | 6 |
| A dumb question....
Are you having multiple rule firings or are you getting multiple
notifications of the same firing?
s/rob
|
| 4576.21 | 1 trap = 20 different rules fire, 1 firing per rule | CSOADM::ROTH | you just KEEP ME hangin' on... | Thu Apr 22 1993 17:44 | 22 |
| (Extract from .6 of this topic)
Alarm: critical SNMP LOCAL_NS:.test Rule vitalink_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule cin_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule sur_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule rif_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule col_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule soc_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule bmvx_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule tolct_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule tol_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule akr_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule noeca_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule rho_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule odb_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule obs_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule sys_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule str_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule nwoca_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule arl_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule mrd_SNMP_LINKDOWN has fired
Alarm: critical SNMP LOCAL_NS:.test Rule cle_SNMP_LINKDOWN has fired
|
| 4576.22 | let's try some more info... | CSOADM::ROTH | you just KEEP ME hangin' on... | Tue Apr 27 1993 15:35 | 175 |
|
Perhaps I can get this topic moving again. I suspect that maybe I
(1) Have a particular configuration that is provoking the problem or
(2) Have somthing set up terribly wrong (I have looked through the V1.3
manuals and saw nothing obviously bogus about my setup).
Below is a text representation of my database hierarchy on my MCC system with
the domains and the SNMP devices listed. For each domain, I specify the level
(in my case, State, City, Building and Room) and if the domain contains
(a) No rules at all or
(b) A "packet" of rules.
The "packet" of rules consists of these five (wildcarded) types:
Expression = (OCCURS (SNMP * linkDown)), -
Expression = (OCCURS (SNMP * linkUp)), -
Expression = (OCCURS (SNMP * IP reachability DOWN)), -
Expression = (OCCURS (SNMP * coldStart)), -
Expression = (OCCURS (SNMP * warmStart)), -
Each rule is created with the following CREATE syntax:
Create Domain LOCAL_NS:.<domain name> Rule <domain name>_<rule type>
Therefore, each rule is created in <domain name> and contains <domain name>
as part of the rule name (to keep my sanity).
!Note: Some domains have rules but no SNMP entities (yet)
!Note: Some domains have SNMP entities but no rules
in Domain ohio (STATE):
DOMAIN(akr) !CITY, "PACKET OF RULES" IN DOMAIN
SNMP(bbakr1)
DOMAIN(cin) !CITY, "PACKET OF RULES" IN DOMAIN
SNMP(oecn-swoca-ether)
SNMP(oecn-swoca-serial)
DOMAIN(cle) !CITY, "PACKET OF RULES" IN DOMAIN
SNMP(bbcle1)
SNMP(bbcle1-ether)
DOMAIN(col) !CITY, NO RULES IN DOMAIN
DOMAIN(arl) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(strs-arlngt)
DOMAIN(bar) !BUILDING, NO RULES IN DOMAIN
SNMP(ocb)
DOMAIN(eng) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(ode)
DOMAIN(mrd) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(mrdd-cdc)
DOMAIN(obs) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(bes-pilot)
DOMAIN(odb) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(bbodb1)
DOMAIN(odh) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(health)
DOMAIN(rho) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(bbrho1)
SNMP(bbrho2)
SNMP(bbrho1-bbsoc1-0)
SNMP(bbrho1-bbsoc1-1)
DOMAIN(rif) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(bbrif1)
SNMP(bbrif2)
DOMAIN(soc) !BUILDING, "PACKET OF RULES" IN DOMAIN
DOMAIN(sys) !ROOM, "PACKET OF RULES" IN DOMAIN
DOMAIN(digital) !ROOM, "PACKET OF RULES" IN DOMAIN
SNMP(bbsoc1)
SNMP(bbsoc2)
SNMP(bbsoc3)
DOMAIN(str) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(strs-broad)
DOMAIN(sur) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(odn-surface)
DOMAIN(bmvx) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(bmv)
DOMAIN(dysco) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(dys-co)
DOMAIN(vitalink) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(socc2)
SNMP(hwysoc)
DOMAIN(day) !CITY, "PACKET OF RULES" IN DOMAIN
DOMAIN(lim) !CITY, NO RULES IN DOMAIN
SNMP(oecn-noacsc)
DOMAIN(pik) !CITY, NO RULES IN DOMAIN
SNMP(oecn-scoca-ether)
SNMP(oecn-scoca-serial)
DOMAIN(tol) !CITY, "PACKET OF RULES" IN DOMAIN
SNMP(bbtol1)
DOMAIN(mavca) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(oecn-mavca)
DOMAIN(noeca) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(oecn-noeca)
SNMP(oecn-noeca-ether)
DOMAIN(nwoca) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(oecn-nwoca)
DOMAIN(tolct) !BUILDING, "PACKET OF RULES" IN DOMAIN
SNMP(oecn-tolct)
DOMAIN(wos) !CITY, NO RULES IN DOMAIN
SNMP(oecn-tccsa)
DOMAIN(yng) !CITY, "PACKET OF RULES" IN DOMAIN
SNMP(oenc-access)
Problem description:
If I put SNMP entity 'TEST' into domain SOC and have 'TEST' output a LINK
DOWN trap, call goes well- I see the rule fire in domain SOC.
If I delete (via double-slash in Toolbox) the SNMP entity 'TEST' and cause
another trap, I get a plethora of rule firings:
Rule vitalink_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.vitalink [68,51]
Rule yng_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.yng [68,52]
Rule akr_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.akr [68,53]
Rule sur_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.sur [68,54]
Rule rif_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.rif [68,55]
Rule cle_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.cle [68,56]
Rule tol_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.tol [68,57]
Rule odh_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.odh [68,58]
Rule arl_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.arl [68,59]
Rule obs_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.obs [68,60]
Rule cin_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.cin [68,61]
Rule sys_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.sys [68,62]
Rule day_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.day [68,63]
Rule nwoca_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.nwoca [68,64]
Rule soc_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.soc [68,65]
Rule odb_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.odb [68,66]
Rule bmvx_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.bmvx [68,67]
Rule rho_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.rho [68,68]
Rule rho_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.rho [70,69]
Rule tolct_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.tolct [68,70]
Rule str_SNMP_LINKDOWN has fired <time>Domain LOCAL_NS:.str [68,71]
Dunno if it is significant, but in my iconic map domains I have child entities
drawn in (i.e. SNMP BBSOC1 INTERFACE 3) for the SNMP entities.
Does any of this information help?
Thanks-
Lee Roth
|
| 4576.23 | ? | CSOADM::ROTH | you just KEEP ME hangin' on... | Fri Apr 30 1993 11:02 | 4 |
| Hello? Anyone have any ideas? The response in the QAR system (number 161)
is about the same as here- nothing.
Lee
|
| 4576.24 | you can receive traps from unregistered objects | MLNCSC::LASAGNA | NO!, le scarpe no !! | Wed May 05 1993 06:07 | 30 |
| Lee,
I don't have a real answer for what i consider a STRANGE behaviour,
anyway i noticed that if you still have in UCX the object you
deregistered, MCC continue to receive traps from that node.
infact if you issue the show snmp <node-unregistered-but-in-ucx>
the answer will be :
MCC> show snmp scossa
Using default ALL IDENTIFIERS
SNMP scossa
AT 5-MAY-1993 10:51:15 Identifiers
Examination of attributes shows:
Address = 16.192.16.50
Name = scossa
note that the Registered Name field is missed because its not
registered.
This can't give you an answer or you probably already knew this, but
have you tried to delete the object from UCX ?? you shouldn't receive
all those rule firings.
Hope this helps, Ciao
Andrea Lasagna.
|
| 4576.25 | | MOLAR::YAHEY::BOSE | | Fri May 28 1993 16:51 | 13 |
|
RE: .23
Lee,
We have managed to reproduce your problem on our system
and isolated it to be a bug in alarms. I believe William is taking
care of the problem.
I would like to apologise if we have given the impression
of being somewhat unresponsive.
Rahul.
|
| 4576.26 | good news! hurrah! | CSOADM::ROTH | you just KEEP ME hangin' on... | Tue Jun 01 1993 09:11 | 8 |
| THANK YOU!! THANK YOU!!
THANK YOU!! THANK YOU!!
THANK YOU!! THANK YOU!!
(I knew I wasn't CrAzY...)
Lee
|