[kwlug-disc] Prometheus for multiple server monitoring?

[Adam Glauser]

I have only limited experience with Prometheus, but I believe this just
means that the service is responding on its metrics endpoint, not saying
anything about what those metrics may be saying.

The way I understand it, Prometheus has a list of “targets” which it will
“scrape” in a mostly round-robin manner. The targets can be listed
individually in config, or be discovered by a service-discovery mechanism.

In your case, I think the Targets are the black box-exporter services. They
are responding, so are considered up, even though the SSH service they are
monitoring is down (probe_status=0)

Service discovery is pretty neat IMO. For example, Prometheus can access a
Docker daemon, then add a new Target for each container (potentially
filtering by container metadata like labels). So you can label all your
containers/Swarm Services that expose Prom. metrics, and they will
automagically show up as scrape targets and start reporting results.

On Sun, May 17, 2026 at 2:41 PM Khalid Baheyeldin <kb at 2bits.com> wrote:

> Thanks to everyone who chimed in about Prometheus.
>
> I was able to get it up and running on the same VPS as Nagios4.
>
> And was able to configure it to send emails when something is down and
> also when it is up again.
> This was mainly done with the prometheus-blackbox-exporter and
> prometheus-alertmanager packages, and checking SSH, HTTP and Drupal.
>
> For SSH, I use the 'ssh_banner' module, with:
> ({__name__="probe_success",job="SSH"} == 0)
>
> For HTTP, I use the 'http_2xx' module, with:
> ({__name__="probe_http_status_code",job="HTTPServers"} != 200)
>
> I think I covered everything that my Nagios4 instance currently does, and
> would be decommissioning it soon.
>
> But there is a snag, or misunderstanding on my part:
> When an alert fires, I do get an email about the failure.
> But, when I go to Status -> Targets, the hosts that failed still show as
> UP.
>
> Probably because there is still a response from them?
>
> Can one comment on this?
> _______________________________________________
> kwlug-disc mailing list
> To unsubscribe, send an email to kwlug-disc-leave at kwlug.org
> with the subject "unsubscribe", or email
> kwlug-disc-owner at kwlug.org to contact a human being.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.kwlug.org/pipermail/kwlug-disc_kwlug.org/attachments/20260522/36685ac1/attachment-0001.htm>