Fault Management System Configuration
The Fault Management System (FMS) provides a framework for event detection, correlation, and alarm generation. Each event triggers an alarm based on correlation logic parameters specified by individual Protocol Modules. Events, as OPER_LOGs relayed from the VLOGd module, are processed according to the correlation rules in the configuration file alarm_def_config.yaml. The generated alarms persist to indicate faults and are maintained in a database accessible via show commands.
Note:
• FMS is disabled by default. Once enabled, it triggers alarms for all valid OPER_LOG events received by the FMS node.js process.
• The FMS event-alarm correlation configuration is stored in a YAML file (alarm_def_config.yaml), which cannot be modified via CMLSH commands. If changes are required, an operator with the appropriate privileges can edit the file in YAML syntax, but only before starting FMS. Once FMS is active, editing this file is prohibited, as changes take effect only after FMS is disabled, updated, and then re-enabled.
• The device’s logging level must be set to at least 4 (NOTIFY) to ensure that FMS receives notification events and can take appropriate action. Setting a lower logging level may prevent FMS from receiving clear events, resulting in unresolved active alarms. FMS does not manage the system logging level.
• FMS relies on the loopback interface (lo0) for communication with VLOGd, so the operational status of lo0 is essential for both FMS and VLOGd.
• If Localhost communication is blocked by the Access Control List (ACL), FMS must be disabled. Conversely, if FMS is enabled, the ACL must not block Localhost.
• If FMS reboots due to a device reboot, upgrade, downgrade, or manual restart, active alarms are closed. Use the
show alarm closed CLI command to view closed active alarms.
FMS applies correlation procedures based on the configurations specified
Table 1-1: FMS correlation procedures
Correlation type | Description |
|---|
Generalization | • Groups two or more events into a single alarm. • A generalized alarm will further use one of the correlation types (none, time-bound, counting and compression) for applying correlation logic to the new alarm. |
Time-bound | • Stipulates that when the event is received, a timer is started for that event. • While the timer is running, subsequent events of the same type are suppressed. • On the expiry of the timer, an alarm will be raised for that event stating the count for the number of times that event was received in this duration. |
Counting | Considers a specified number of similar events as one. In this correlation type, the respective alarm will be raised after the event has occurred for count times. |
Compression | Check multiple occurrences of the same event for duplicate/redundant event information, remove the redundancies, and report them as a single alarm. |
Severity | Correlates events based on the severity of the events. |
Implementation
FMS was developed with NodeJS with scripts written in JavaScript with a *.js extension and configuration files with a *.yaml extension. These files are in the below paths in OcNOS.
Table 1-2: FMS script and configuration files
/usr/local/bin/js | JavaScript files (*.js files) |
/usr/local/etc | Configuration files (*.yaml files) |
Enabling and Disabling the Fault Management System
Follow the below steps to enable or disable FMS:
Enabling FMS
OcNOS#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
(config)#
(config)#fault-management enable
(config)#
Disabling FMS
OcNOS#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
(config)#
(config)#fault-management disable
(config)#
Alarm Configuration File
The alarm configuration file contains the configurations/rules for the alarms that will be referred by FMS to generate alarms upon receiving events. This file is in *.yaml format (human readable) in /usr/local/etc.
This file can be edited before starting FMS to include correlation rules for specific events.
Alarm Configuration File Template
#-------Template-------
#- Event_Group:
# - ALARM_ID: # Integer number identifying alarm
# ALARM_TYPE_ID: # Alarm Type-id(AIS, EQPT, LOS, OTS, OPWR, UNKNOWN)
# EVENT: # Event name(oper_log)
# GENERALIZED_EVENT_NAME: # Event name for the Generalization Event Group
# ALARM_DESC: # Alarm string which will be generated
# CORRELATION_TYPE: # Correlation logic type(0:No-Correlation, 1:Generalization, 2:Timebound, 3:Counting, 4:Compression, 5:Drop-Event, 6:Severity)
# GENERALIZED_CORRELATION_TYPE # Correlation type, in which generalized event will be sent
# CORRELATION_COUNTER: # Counter value that will be considered during counting logic to raise alarm
# CORRELATION_TIMER_DURATION: # Timer duration to be considered for time bound logic
# CORRELATION_SEVERITY: # Alarm Severity(0:Critical, 1:Major, 2:Warning, 3:Minor, 4:Unknown)
# QUALIFIER_STRING_POSITION: # List of positions where qualifier values present
# QUALIFIER_POSITION_1_EVENT_1: # First position of the qualifier value in the first event
# RESOURCE_STRING_POSITION: # List of positions where resource values present
# RESOURCE_POSITION_1_EVENT_1: # First position of the resource value in the first event
# SNMP_TRAP: # SNMP TRAP (true(1) or false(0))
# SNMP_OID: # OID for SNMP TRAP
# NETCONF_NOTIFICATION: # Netconf Notification (true(1) or false(0))
# CLEAR_ALARM: # Clear Alarm (oper_log enum, Status for Alarm will be made In-active if this event is received)
# CLEAR_EVENT_PATTERN_VALUES: # Pattern values which will be searched in event's description to identify clear event and to clear active alarm (required if both active and clear event types are same)
# SNMP_TRAP_CLEAR: # true(1) or false(0, if CLEAR_ALARM is null then SNMP_TRAP_CLEAR will be null)
# SNMP_CLEAR_OID: # OID for SNMP TRAP CLEAR
# NETCONF_CLEAR_NOTIFICATION: # Clear Netconf Notification information
Auto Generating the Alarm Configuration File
The auto_yaml_generator.js file is a NodeJS script that generates the alarm configuration file (alarm_def_config.yaml) for the oper logs which are listed in the oper_logs_list.yaml file with the default values as shown below.
# Integer number identifying alarm
ALARM_ID: 1000
# Event name (oper_log)
EVENT: oper_log string
# Event name for the Generalization Event Group
GENERALIZED_EVENT_NAME: null
# Alarm string which will be generated
ALARM_DESC: oper_log string
# Correlation logic type (0: No-Correlation, 1: Generalization, 2: Time Bound, 3: Counting, 4: Compression, 5: Drop-Event)
CORRELATION_TYPE: 0
# Correlation type, in which generalized event will be sent
GENERALISED_CORRELATION_TYPE: null
# Counter value that will be considered during counting logic to raise alarm
CORRELATION_COUNTER: 3
# Timer duration to be considered for time bound logic
CORRELATION_TIMER_DURATION: 20000
# Alarm Severity(1:Emergency, 2:Alert, 3:Critical, 4:Error, 5:Warning, 6:Notification, 7:Informational, 8:Debugging, 9:Cli)
CORRELATION_SEVERITY: null
# QUALIFIER_STRING_POSITION
QUALIFIER_POSITION_1_EVENT_1: null
# RESOURCE_STRING_POSITION
RESOURCE_POSITION_1_EVENT_1: null
SNMP_TRAP: 0
# OID for SNMP TRAP
SNMP_OID: null
# Netconf Notification (true (1) or false (0))
NETCONF_NOTIFICATION: 1
# Clear Alarm (oper_log enum, Status for Alarm will be made In-active if this event is received)
CLEAR_ALARM: null
# Clear Event's pattern values which will be searched in event's description to identify clear event
CLEAR_EVENT_PATTERN_VALUES: null
# True (1) or False (0, if CLEAR_ALARM is null then SNMP_TRAP_CLEAR will be null)
SNMP_TRAP_CLEAR: 0
# OID for SNMP TRAP CLEAR
SNMP_CLEAR_OID: null
# Clear Netconf Notification information
NETCONF_CLEAR_NOTIFICATION: 0
Alarm Configuration File Generation Steps
1. List all the oper_log enums in the oper_logs_list.yaml file and keep the file in the same path with auto_yaml_generator.js.
2. Copy auto_yaml_generator.js and oper_logs_list.yaml files into /usr/local/bin/js.
3. Run the auto_yaml_generator.js script with the following command.
#node auto_yaml_generator.js
4. After executing the above commands, you will see the alarm-def-config.yaml file in the same directory.
Sample oper_logs_list.yaml File
EVENT_GROUP:
IFMGR_IF_DOWN,
IFMGR_IF_UP,
STP_SET_PORT_STATE,
STP_IPC_COMMUNICATION_FAIL,
STP_ROOTGUARD_PORT_BLOCK,
:
:
Alarm Descriptions
Table 1-3 describes the supported alarms.
Table 1-3: FMS alarms
Alarm | Description |
|---|
CMM_DDM_MONITOR_CURRENT | Transceiver Bias Current crossed the threshold limit |
CMM_DDM_MONITOR_FREQ | Transceiver Frequency crossed the threshold limit |
CMM_DDM_MONITOR_RxPOWER | Transceiver Rx Power crossed the threshold limit |
CMM_DDM_MONITOR_TEC | Transceiver Thermoelectric Cooler fault |
CMM_DDM_MONITOR_TEMP | Transceiver Temperature crossed the threshold limit |
CMM_DDM_MONITOR_TxPOWER | Transceiver Tx Power crossed the threshold limit |
CMM_DDM_MONITOR_VOLT | Transceiver Voltage crossed the threshold limit |
CMM_DDM_MONITOR_WAVE | Transceiver Wavelength crossed the threshold limit |
CMM_FAN_CTRL | Fan insertion, removal, speed, or fault condition alarm |
CMM_MONITOR_CPU | CPU load average crossed the threshold limit |
CMM_MONITOR_CPU_CORE | CPU core usage crossed the threshold limit |
CMM_MONITOR_CURRENT | Current crossed the threshold limit |
CMM_MONITOR_DISK_READ_ACTIVITY | Disk read activity crossed the threshold limit |
CMM_MONITOR_DISK_REMAIN_LIFE | Disk remaining life crossed the threshold limit |
CMM_MONITOR_DISK_WRITE_ACTIVITY | Disk write activity crossed the threshold limit |
CMM_MONITOR_FAN | FAN RPM crossed the threshold limit |
CMM_MONITOR_PSU_IIN | Power supply unit input current crossed the threshold limit |
CMM_MONITOR_PSU_IOUT | Power supply unit output current crossed the threshold limit |
CMM_MONITOR_PSU_PIN | Power supply unit input power crossed the threshold limit |
CMM_MONITOR_PSU_POUT | Power supply unit output power crossed the threshold limit |
CMM_MONITOR_PSU_POWER | Power supply unit insertion, removal, or fault condition |
CMM_MONITOR_PSU_PRESENCE | Power supply unit is present |
CMM_MONITOR_PSU_TEMP1 | Power supply unit temperature 1 crossed the threshold limit |
CMM_MONITOR_PSU_TEMP2 | Power supply unit temperature 2 crossed the threshold limit |
CMM_MONITOR_PSU_VIN | Power supply unit input voltage crossed the threshold limit |
CMM_MONITOR_PSU_VOUT | Power supply unit output voltage crossed the threshold limit |
CMM_MONITOR_RAM | RAM memory usage crossed the threshold limit |
CMM_MONITOR_SDCARD | Hard-disk usage crossed the threshold limit or fault condition |
CMM_MONITOR_TEMP | Temperature sensor crossed the threshold limit |
CMM_MONITOR_VOLTAGE | Voltage crossed the threshold limit |
CMM_TRANSCEIVER | Transceiver on fault condition |
HW_PROFILE_MONITOR | TCAM Utilization |
IFMGR_IF_DOWN | Interface state down |
IFMGR_IF_UP | Interface state up |
HW_PROFILE_MONITOR | TCAM group utilization |