Difference between revisions of "Pushgateway"

From UVOO Tech Wiki
Jump to navigation Jump to search
 
(19 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway
 
https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway
 +
 +
https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/
  
 
https://github.com/prometheus/pushgateway
 
https://github.com/prometheus/pushgateway
 +
```
 +
echo "some_metric 3.14" | curl --data-binary @- https://pushgateway.example/metrics/job/some_job
 +
```
 +
 +
# As node collector
 +
 +
https://github.com/prometheus/node_exporter/issues/279
 +
```
 +
curl -s http://localhost:9256/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- http://pushgateway.example.com:9091/metrics/job/node/instance/<INSTANCE_NAME>
 +
```
 +
 +
## Using Cron job to push every 30 seconds
 +
 +
sudo nano /usr/local/bin/prompush
 +
```
 +
#!/bin/bash
 +
set -eu
 +
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/c1e52efa-af1c-11ee-be01-e78fa8f1ca96-linuxnode/instance/lxd0
 +
```
 +
 +
```
 +
sudo chmod +x /usr/local/bin/prompush
 +
```
 +
 +
/etc/cron.d/pushgateway
 +
```
 +
* * * * * nobody /usr/local/bin/prompush
 +
* * * * * nobody sleep 30; /usr/local/bin/prompush
 +
```
 +
 +
## Ingress NGINX Annotations
 +
 +
```
 +
htpasswd -c auth default
 +
kubectl create secret generic pushgateway-ingress-basic-auth --from-file=auth
 +
```
 +
 +
 +
```
 +
  annotations:
 +
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8,192.168.0.0/16"
 +
    nginx.ingress.kubernetes.io/auth-type: basic
 +
    nginx.ingress.kubernetes.io/auth-secret: pushgateway-ingress-basic-auth
 +
    # nginx.ingress.kubernetes.io/server-snippet: |
 +
    #  satisfy  any;
 +
    #  allow 66.220.144.0/20;
 +
    #  deny  all;
 +
```
 +
 +
And update script to use auth
 +
```
 +
#!/bin/bash
 +
set -eu
 +
basic="default:mysecret"
 +
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/c1e52efa-af1c-11ee-be01-e78fa8f1ca96-linuxnode/instance/lxd0 -u ${basic}
 +
```
 +
 +
# trigger on no data
 +
https://stackoverflow.com/questions/64729855/grafana-to-alert-if-prometheus-pushgateway-have-old-data
 +
 +
# Ubuntu
 +
 +
Install exporter on Ubuntu
 +
```
 +
sudo apt update && sudo apt install -y prometheus-node-exporter
 +
```
 +
 +
sudo vim /usr/local/bin/insights-push
 +
```
 +
#!/bin/bash
 +
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/host/instance/$(hostname)
 +
```
 +
 +
```
 +
sudo chmod +x  /usr/local/bin/insights-push
 +
```
 +
 +
sudo vim /etc/cron.d/insights-push
 +
```
 +
* * * * * nobody /usr/local/bin/insights-push
 +
* * * * * nobody sleep 30; /usr/local/bin/insights-push
 +
```
 +
This is 30 seconds because you will want scraper to be 30 seconds
 +
 +
### Prometheus Pushgateway Scraper
 +
```
 +
- job_name: 'pushgateway'
 +
  honor_labels: true
 +
  scrape_interval: 30s
 +
  metrics_path: /metrics
 +
  static_configs:
 +
    - targets: ['prometheus-pushgateway.prometheus.svc:9091']
 +
```
 +
 +
# RHEL/Fedora
 +
 +
## Using EPEL repo
 +
```
 +
sudo dnf -y update
 +
```
 +
 +
```
 +
sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
 +
```
 +
 +
```
 +
sudo dnf -y install node_exporter
 +
sudo systemctl enable --now node_exporter
 +
sudo systemctl status node_exporter
 +
```
 +
 +
## BASH Script
 +
```
 +
#!/bin/bash
 +
set -eu
 +
 +
version=1.8.1
 +
 +
# Update the system
 +
# sudo dnf -y update
 +
# sudo dnf install -y tar curl
 +
 +
# Download Node Exporter
 +
curl -sLO https://github.com/prometheus/node_exporter/releases/download/v${version}/node_exporter-${version}.linux-amd64.tar.gz
 +
 +
# Extract the downloaded tar.gz file
 +
tar xvfz node_exporter-${version}.linux-amd64.tar.gz
 +
 +
# Move the Node Exporter binary to the /usr/local/bin directory
 +
sudo mv node_exporter-${version}.linux-amd64/node_exporter /usr/local/bin
 +
 +
sudo useradd -r -s /bin/false prometheus
 +
 +
sudo dnf install -y policycoreutils-python-utils
 +
sudo semanage fcontext -a -t bin_t '/usr/local/bin/node_exporter'
 +
sudo restorecon -v '/usr/local/bin/node_exporter'
 +
 +
# Create a new systemd service file for Node Exporter
 +
echo "[Unit]
 +
Description=Node Exporter
 +
After=network.target
 +
 +
[Service]
 +
User=prometheus
 +
ExecStart=/usr/local/bin/node_exporter
 +
 +
[Install]
 +
WantedBy=default.target" | sudo tee /etc/systemd/system/node_exporter.service
 +
 +
# Reload the systemd daemon to read the new service
 +
sudo systemctl daemon-reload
 +
 +
# Start the Node Exporter service
 +
sudo systemctl start node_exporter
 +
 +
# Enable the Node Exporter service to start on boot
 +
sudo systemctl enable node_exporter
  
−
echo "some_metric 3.14" | curl --data-binary @- https://pushgateway.example/metrics/job/so
+
# Verify that Node Exporter is running
−
me_job
+
sudo systemctl status node_exporter
 +
```

Latest revision as of 13:42, 21 June 2024

https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-pushgateway

https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/

https://github.com/prometheus/pushgateway 

echo "some_metric 3.14" | curl --data-binary @- https://pushgateway.example/metrics/job/some_job

As node collector

https://github.com/prometheus/node_exporter/issues/279 

curl -s http://localhost:9256/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- http://pushgateway.example.com:9091/metrics/job/node/instance/<INSTANCE_NAME>

Using Cron job to push every 30 seconds

sudo nano /usr/local/bin/prompush 

#!/bin/bash
set -eu
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/c1e52efa-af1c-11ee-be01-e78fa8f1ca96-linuxnode/instance/lxd0

sudo chmod +x /usr/local/bin/prompush

/etc/cron.d/pushgateway 

* * * * * nobody /usr/local/bin/prompush
* * * * * nobody sleep 30; /usr/local/bin/prompush

Ingress NGINX Annotations

htpasswd -c auth default
kubectl create secret generic pushgateway-ingress-basic-auth --from-file=auth

  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8,192.168.0.0/16"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: pushgateway-ingress-basic-auth
    # nginx.ingress.kubernetes.io/server-snippet: |
    #   satisfy  any;
    #   allow 66.220.144.0/20;
    #   deny   all;

And update script to use auth 

#!/bin/bash
set -eu
basic="default:mysecret"
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/c1e52efa-af1c-11ee-be01-e78fa8f1ca96-linuxnode/instance/lxd0 -u ${basic}

trigger on no data

https://stackoverflow.com/questions/64729855/grafana-to-alert-if-prometheus-pushgateway-have-old-data

Ubuntu

Install exporter on Ubuntu 

sudo apt update && sudo apt install -y prometheus-node-exporter

sudo vim /usr/local/bin/insights-push 

#!/bin/bash
curl -s http://localhost:9100/metrics | grep -v "\(\(^\| \)go_\|http_request\|http_requests\|http_response\|process_\)" | curl --data-binary @- https://pushgateway.example.com/metrics/job/host/instance/$(hostname)

sudo chmod +x  /usr/local/bin/insights-push

sudo vim /etc/cron.d/insights-push 

* * * * * nobody /usr/local/bin/insights-push
* * * * * nobody sleep 30; /usr/local/bin/insights-push

This is 30 seconds because you will want scraper to be 30 seconds

Prometheus Pushgateway Scraper

- job_name: 'pushgateway'
  honor_labels: true
  scrape_interval: 30s
  metrics_path: /metrics
  static_configs:
    - targets: ['prometheus-pushgateway.prometheus.svc:9091']

RHEL/Fedora

Using EPEL repo

sudo dnf -y update

sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

sudo dnf -y install node_exporter
sudo systemctl enable --now node_exporter
sudo systemctl status node_exporter

BASH Script

#!/bin/bash
set -eu

version=1.8.1

# Update the system
# sudo dnf -y update
# sudo dnf install -y tar curl

# Download Node Exporter
curl -sLO https://github.com/prometheus/node_exporter/releases/download/v${version}/node_exporter-${version}.linux-amd64.tar.gz

# Extract the downloaded tar.gz file
tar xvfz node_exporter-${version}.linux-amd64.tar.gz

# Move the Node Exporter binary to the /usr/local/bin directory
sudo mv node_exporter-${version}.linux-amd64/node_exporter /usr/local/bin

sudo useradd -r -s /bin/false prometheus

sudo dnf install -y policycoreutils-python-utils
sudo semanage fcontext -a -t bin_t '/usr/local/bin/node_exporter'
sudo restorecon -v '/usr/local/bin/node_exporter'

# Create a new systemd service file for Node Exporter
echo "[Unit]
Description=Node Exporter
After=network.target

[Service]
User=prometheus
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=default.target" | sudo tee /etc/systemd/system/node_exporter.service

# Reload the systemd daemon to read the new service
sudo systemctl daemon-reload

# Start the Node Exporter service
sudo systemctl start node_exporter

# Enable the Node Exporter service to start on boot
sudo systemctl enable node_exporter

# Verify that Node Exporter is running
sudo systemctl status node_exporter
Retrieved from "https://tech.uvoo.io/index.php?title=Pushgateway&oldid=5307"