Latest revision as of 13:42, 21 June 2024

echo "some_metric 3.14" | curl --data-binary @- https://pushgateway.example/metrics/job/some_job

As node collector

https://github.com/prometheus/node_exporter/issues/279

curl -s http://localhost:9256/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- http://pushgateway.example.com:9091/metrics/job/node/instance/<INSTANCE_NAME>

Using Cron job to push every 30 seconds

sudo nano /usr/local/bin/prompush

#!/bin/bash
set -eu
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/c1e52efa-af1c-11ee-be01-e78fa8f1ca96-linuxnode/instance/lxd0

sudo chmod +x /usr/local/bin/prompush

/etc/cron.d/pushgateway

* * * * * nobody /usr/local/bin/prompush
* * * * * nobody sleep 30; /usr/local/bin/prompush

Ingress NGINX Annotations

htpasswd -c auth default
kubectl create secret generic pushgateway-ingress-basic-auth --from-file=auth

  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8,192.168.0.0/16"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: pushgateway-ingress-basic-auth
    # nginx.ingress.kubernetes.io/server-snippet: |
    #   satisfy  any;
    #   allow 66.220.144.0/20;
    #   deny   all;

And update script to use auth

#!/bin/bash
set -eu
basic="default:mysecret"
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/c1e52efa-af1c-11ee-be01-e78fa8f1ca96-linuxnode/instance/lxd0 -u ${basic}

trigger on no data

https://stackoverflow.com/questions/64729855/grafana-to-alert-if-prometheus-pushgateway-have-old-data

Ubuntu

Install exporter on Ubuntu

sudo apt update && sudo apt install -y prometheus-node-exporter

sudo vim /usr/local/bin/insights-push

#!/bin/bash
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/host/instance/$(hostname)

sudo chmod +x  /usr/local/bin/insights-push

sudo vim /etc/cron.d/insights-push

* * * * * nobody /usr/local/bin/insights-push
* * * * * nobody sleep 30; /usr/local/bin/insights-push

This is 30 seconds because you will want scraper to be 30 seconds

Prometheus Pushgateway Scraper

- job_name: 'pushgateway'
  honor_labels: true
  scrape_interval: 30s
  metrics_path: /metrics
  static_configs:
    - targets: ['prometheus-pushgateway.prometheus.svc:9091']

RHEL/Fedora

Using EPEL repo

sudo dnf -y update

sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

sudo dnf -y install node_exporter
sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter

BASH Script

#!/bin/bash
set -eu

version=1.8.1

# Update the system
# sudo dnf -y update
# sudo dnf install -y tar curl

# Download Node Exporter
curl -sLO https://github.com/prometheus/node_exporter/releases/download/v${version}/node_exporter-${version}.linux-amd64.tar.gz

# Extract the downloaded tar.gz file
tar xvfz node_exporter-${version}.linux-amd64.tar.gz

# Move the Node Exporter binary to the /usr/local/bin directory
sudo mv node_exporter-${version}.linux-amd64/node_exporter /usr/local/bin

sudo useradd -r -s /bin/false prometheus

sudo dnf install -y policycoreutils-python-utils
sudo semanage fcontext -a -t bin_t '/usr/local/bin/node_exporter'
sudo restorecon -v '/usr/local/bin/node_exporter'

# Create a new systemd service file for Node Exporter
echo "[Unit]
Description=Node Exporter
After=network.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target" | sudo tee /etc/systemd/system/node_exporter.service

# Reload the systemd daemon to read the new service
sudo systemctl daemon-reload

# Start the Node Exporter service
sudo systemctl start node_exporter

# Enable the Node Exporter service to start on boot
sudo systemctl enable node_exporter

# Verify that Node Exporter is running
sudo systemctl status node_exporter

Difference between revisions of "Pushgateway"