[Bug]: New include-label parameter not matching in some cases

[gianmarco-mameli]

Please try to fill out as much of the information below as you can. Thank you!

• Yes, I've searched similar issues on GitHub and didn't find any.

Which version contains the bug?

0.6.0

Describe the bug

Hi,
I'm trying to use the newly added parameter include-label parameter (see #113 and #114), but I have some problems to make it work.
It seems working with a label like severity=warning, it matches correctly, but it's not working with labels like instance=hostname:8088 or job=haproxy-metrics

The commands looks like that taken from inspect in Icinga
'/usr/lib/nagios/plugins/check_prometheus' 'alert' '--hostname' 'prometheus.domain' '--port' '443' '--secure' '--timeout' '30' '--include-label' 'instance=hostname:8088'

'/usr/lib/nagios/plugins/check_prometheus' 'alert' '--hostname' 'prometheus.domain' '--port' '443' '--secure' '--timeout' '30' '--include-label' 'severity=warning'

How to recreate the bug?

Try a command like
'/usr/lib/nagios/plugins/check_prometheus' 'alert' '--hostname' 'prometheus.domain' '--port' '443' '--secure' '--timeout' '30' '--include-label' 'instance=hostname:8088
'/usr/lib/nagios/plugins/check_prometheus' 'alert' '--hostname' 'prometheus.domain' '--port' '443' '--secure' '--timeout' '30' '--include-label' 'job=haproxy-metrics'

[martialblog]

Hi, thanks for reporting it. I'll try to recreate this and then provide a fix.

[martialblog]

The issue here is that we're currently using the Rules API, not the Alert API, to determine the state of the alerts.

The reason behind this, that at the Rules API we can see all configured alerts and their state, which makes it easier to list/filter. The Alert API only returns active alerts.

However, only the alert instances have the instance labels (like job, or instance).

We would have to fetch Alert API when there are firing alerts to get the instance labels.

[martialblog]

@gianmarco-mameli Hi, I did some prototyping here https://github.com/NETWAYS/check_prometheus/tree/active-alert-labels

Feel free to test it.

[gianmarco-mameli]

Hi @martialblog , I've tested the new branch and I think that now it's working, example:
root@3a4aec3ebe69:/opt# ./check_prometheus 'alert' '--hostname' 'prometheus.domain' '--port' '443' '--secure' '--insecure' '--timeout' '30' '--include-label' 'job=dummy-metrics' [CRITICAL] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive \_ [CRITICAL] [TargetDown] - Job: [dummy-metrics] is firing - value: 50.00 - {"alertname":"TargetDown","job":"dummy-metrics","severity":"warning"} |total=1 firing=1 pending=0 inactive=0

The same command returns OK and no matching alerts with the 0.6.0 version

Thanks

[martialblog]

See #122

[martialblog]

@gianmarco-mameli Thanks again for the feedback. It's very appreciated and super valuable

[gianmarco-mameli]

@martialblog you're welcome, I'm happy to help when I can, have a good day.