UNSOLVED

alanscopa2015

updated

9 years ago

A

alanscopa2015

1 Rookie

•

13 Posts

1

9014

July 7th, 2017 12:00

How do you stop/remove the Heartbeat alert event on OneFS 8.0.0.4?

We upgraded to OneFS 8.0.0.4 on June 25th.  We started to get the Heartbeat alert emails daily for every node in the cluster.  I tried to follow KB Article (000493496 - 8.0 Daily Hearbeat Alert and lowering frequency of it) but its not that accurate.  I ended up changing the intervals to weekly but went into the GUI and just completely deleted the Heartbeat alert and Heartbeat channel out of the Alerts tab.  But yesterday we received the emails again so something is still triggering it.

I ran the below commands and you can see the output below:

mycluster# isi event alerts view Heartbeat

Error in field(s): ID

Alert condition Heartbeat not found

mycluster# isi event settings view

     Retention Days: 90

        Storage Limit: 1

Maintenance Start: Never

Maintenance Duration: Never

Heartbeat Interval: weekly

Has anyone ran into this?  Does anyone know how to stop the heartbeat completely?

Thanks,

-Alan

  • kbaryeh

    5 Posts

    5753

    0

    Posted July 10th, 2017 03:00

    Have you tried running isi event settings modify --heartbeat-interval or you'd have to disable the alert channel it uses.

    isi event alerts view Heartbeat works for me, our clusters are running 8.0.0.4

    Name: Heartbeat

    Eventgroup: 400050004

    Category: -

    Sev: *

    Channel: Heartbeat Self-Test

    Condition: ONGOING

    Interval: Now

  • seoeook

    3 Posts

    5754

    2

    Posted July 10th, 2017 05:00

    In OneFS 8.0 and later versions,  cluster would generate heartbeat event every day.

    The Heartbeat event belongs to Software events category. When configure Alert that uses SMTP channel for Software Events, the SMTP recipients would receive daily email notification for heartbeat event.

    Use below step to stop receiving daily heartbeat events:

    1.  From WebUI, modify Alerts that utilize SMTP channel, and uncheck "Software Events".

    2.  Add all software event IDs except for heartbeat manually in CLI

    # isi event alerts modify --add-eventgroup= "100010043,400020001,400020002,400030001,400030002,400040002,400040005,400040008,400040009,400040010,400040011,400040012,400040013,400040014,400040015,400040016,400040017,400040018,400040019,400040020,400050001,400050002,400050003,400060001,400060002,400060003,400060004,400070004,400070005,400080001,400100001,400100002,400100003,400100004,400100005,400100011,400110001,400120001,400130001,400130002,400140001,400140002,400150001,400150002,400150003,400150004,400150005,400160001,400210001,400210002,499940001,499960001"

    3. Send out a test heartbeat event to verify no more email alerts are generated.

    # /usr/bin/isi_celog/celog_send_events.py 400050004

    KB 490436

    How to stop receiving email notification for HeartBeat event

  • alanscopa2015

    1 Rookie

    •

    13 Posts

    5754

    0

    Posted July 10th, 2017 06:00

    seoeook, when I run an isi event alerts view alert2 for a list of alerts IDs, this is what I see:

    mycluster# isi event alerts view alert2

          Name: alert2

    Eventgroup: -

      Category: 100000000, 200000000, 300000000, 600000000, 700000000, 800000000, 900000000

           Sev: *

       Channel: smtp2

    Condition: NEW EVENTS

    This is the only alert that uses SMTP that has Software Events checked off.  So from that list which one is the Heartbeat one?

  • seoeook

    3 Posts

    5755

    0

    Posted July 10th, 2017 17:00

    I do not understand the situation above.

    For me, proceed as documented in KB 490436

    Good.

    Thank you

  • gvaidman

    20 Posts

    5755

    0

    Posted July 11th, 2017 12:00

    What the KB walks you through is removing the high-level categories of events and adding back all the individual events EXCEPT the heartbeat.  You can't just remove an individual event from the default config, since that is defined with categories of events. 

    The categories you list (100000000, 200000000, etc.) each include many events. 


    The specific heartbeat event is 400050004, the one listed in the celog_send_events.py command-line.

  • _jvc73

    1 Rookie

    •

    1 Message

    5755

    2

    Posted May 28th, 2018 05:00

    In case it is useful for anyone, on 8.1 you need to specify the alert channel in order to add the eventgroups or you will get "Empty array field channel id".

    In my case the name of the alert channel is "SMTP", and the alert name is "alert1":

    isi event type list --category=400000000 --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400050004//' > /var/crash/list.txt

    isi event alerts modify alert1 --channel SMTP --add-eventgroup `cat /var/crash/list.txt`

    It can also be done in a single command line, without creating a text file:

    isi event alerts modify alert1 --channel SMTP --add-eventgroup `isi event type list --category=400000000 --format json | python -c 'import json;import sys;print ",".join([eg["id"] for eg in json.loads(sys.stdin.read())])' | sed 's/,400050004//'`

    Cheers

    Joseph

  • Murrieta

    1 Rookie

    •

    10 Posts

    3892

    0

    Posted December 25th, 2019 14:00

    Thanks Joseph. Running 8.2.1.0 and that worked beautifully.