How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?
We upgraded to OneFS 8.0.0.4 on June 25th. We started to get the Heartbeat alert emails daily for every node in the cluster. I tried to follow KB Article (000493496 - 8.0 Daily Hearbeat Alert and lowering frequency of it) but its not that accurate. I ended up changing the intervals to weekly but went into the GUI and just completely deleted the Heartbeat alert and Heartbeat channel out of the Alerts tab. But yesterday we received the emails again so something is still triggering it.
I ran the below commands and you can see the output below:
mycluster# isi event alerts view Heartbeat
Error in field(s): ID
Alert condition Heartbeat not found
mycluster# isi event settings view
Retention Days: 90
Storage Limit: 1
Maintenance Start: Never
Maintenance Duration: Never
Heartbeat Interval: weekly
Has anyone ran into this? Does anyone know how to stop the heartbeat completely?
In OneFS 8.0 and later versions, cluster would generate heartbeat event every day.
The Heartbeat event belongs to Software events category. When configure Alert that uses SMTP channel for Software Events, the SMTP recipients would receive daily email notification for heartbeat event.
Use below step to stop receiving daily heartbeat events:
1. From WebUI, modify Alerts that utilize SMTP channel, and uncheck "Software Events".
2. Add all software event IDs except for heartbeat manually in CLI
What the KB walks you through is removing the high-level categories of events and adding back all the individual events EXCEPT the heartbeat. You can't just remove an individual event from the default config, since that is defined with categories of events.
The categories you list (100000000, 200000000, etc.) each include many events.
The specific heartbeat event is 400050004, the one listed in the celog_send_events.py command-line.
In case it is useful for anyone, on 8.1 you need to specify the alert channel in order to add the eventgroups or you will get "Empty array field channel id".
In my case the name of the alert channel is "SMTP", and the alert name is "alert1":
isi event type list --category=400000000 --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400050004//' > /var/crash/list.txt
kbaryeh
5 Posts
5753
0
Posted July 10th, 2017 03:00
Have you tried running isi event settings modify --heartbeat-interval or you'd have to disable the alert channel it uses.
isi event alerts view Heartbeat works for me, our clusters are running 8.0.0.4
Name: Heartbeat
Eventgroup: 400050004
Category: -
Sev: *
Channel: Heartbeat Self-Test
Condition: ONGOING
Interval: Now