Prometheus delete data

Simple Deletion by instance name

Enable - web.enable-admin-api in values

  extraFlags:
    - web.enable-lifecycle
    - web.enable-admin-api

Update helm chart

helm upgrade

Get ip address of pod and delete data

curl -X POST -g 'http://10.244.1.172:9090/api/v1/admin/tsdb/delete_series?match[]={__name__="metric_name_to_delete"}'
curl -X POST -g 'http://10.244.1.172:9090/api/v1/admin/tsdb/delete_series?match[]={instance="myhost.example.com:9100"}'

Get ip address

ip=$(kubectl get pod -l app.kubernetes.io/component=server -o jsonpath='{.items[*].status.podIP}')

curl -X POST -g 'http://$ip:9090/api/v1/admin/tsdb/delete_series?match[]={instance="https://www.uvoo.io"}'
or
wget --post-data 'match[]={instance="https://www.uvoo.io"}'  'http://$ip:9090/api/v1/admin/tsdb/delete_series'

From local ip on pod did not some to work

More notes

<br />https://faun.pub/how-to-drop-and-delete-metrics-in-prometheus-7f5e6911fb33

https://stackoverflow.com/questions/54704117/how-can-i-delete-old-jobs-from-prometheus

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="blackbox-exporter"}' Now the blackbox-exporter series have been deleted, but if you want to delete them entirely even from the disk, you will have to use this endpoint

curl -X POST 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones' you might have to wait for a while, it might be up to 2 hours or even more in order to see that the job label has been deleted, since you deleted all its series, after that period you can run again :

curl http://localhost:9090/api/v1/label/job/values {"status":"success","data":["blackbox exporter", "cadvisor","node-exporter","postgres-exporter","prometheus"]} no trace for blackbox-exporter anymore

notice that if you want to delete series of a certain job that contain a space on its name, you will have to replace the space with %20

let's say I want to delete the series of blackbox exporter job, here is what you will have to do:

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="blackbox%20exporter"}'

<br /><br />

prometheus:

   image: prom/prometheus:latest
   container_name: prometheus
   restart: unless-stopped
   volumes:
     - ./prometheus.yml:/etc/prometheus/prometheus.yml
     - prometheus_data:/prometheus
   command:
     - '--config.file=/etc/prometheus/prometheus.yml'
     - '--storage.tsdb.path=/prometheus'
     - '--web.console.libraries=/etc/prometheus/console_libraries'
     - '--web.console.templates=/etc/prometheus/consoles'
     - '--web.enable-lifecycle'
     - '--web.enable-admin-api'                               # << Like this
   ports:
     - 9090:9090
   expose:
     - 9090
   networks:
     - monitoring

```