Pushgateway

https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway

https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/

https://github.com/prometheus/pushgateway

echo "some_metric 3.14" | curl --data-binary @- https://pushgateway.example/metrics/job/some_job

As node collector

https://github.com/prometheus/node_exporter/issues/279

curl -s http://localhost:9256/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- http://pushgateway.example.com:9091/metrics/job/node/instance/<INSTANCE_NAME>

Using Cron job to push every 30 seconds

sudo nano /usr/local/bin/prompush

#!/bin/bash
set -eu
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/c1e52efa-af1c-11ee-be01-e78fa8f1ca96-linuxnode/instance/lxd0

sudo chmod +x /usr/local/bin/prompush

/etc/cron.d/pushgateway

* * * * * nobody /usr/local/bin/prompush
* * * * * nobody sleep 30; /usr/local/bin/prompush

Ingress NGINX Annotations

htpasswd -c auth default
kubectl create secret generic pushgateway-ingress-basic-auth --from-file=auth

  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8,192.168.0.0/16"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: pushgateway-ingress-basic-auth
    # nginx.ingress.kubernetes.io/server-snippet: |
    #   satisfy  any;
    #   allow 66.220.144.0/20;
    #   deny   all;

And update script to use auth

#!/bin/bash
set -eu
basic="default:mysecret"
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/c1e52efa-af1c-11ee-be01-e78fa8f1ca96-linuxnode/instance/lxd0 -u ${basic}

trigger on no data

https://stackoverflow.com/questions/64729855/grafana-to-alert-if-prometheus-pushgateway-have-old-data

Ubuntu

Install exporter on Ubuntu

sudo apt update && sudo apt install -y prometheus-node-exporter

sudo vim /usr/local/bin/insights-push

#!/bin/bash
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/host/instance/$(hostname)

sudo chmod +x  /usr/local/bin/insights-push

sudo vim /etc/cron.d/insights-push

* * * * * nobody /usr/local/bin/insights-push

RHEL/Fedora

sudo dnf -y update

sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

sudo dnf -y install node_exporter
sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter