Prometheus delete data

https://faun.pub/how-to-drop-and-delete-metrics-in-prometheus-7f5e6911fb33

https://stackoverflow.com/questions/54704117/how-can-i-delete-old-jobs-from-prometheus

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="blackbox-exporter"}'
Now the blackbox-exporter series have been deleted, but if you want to delete them entirely even from the disk, you will have to use this endpoint

curl -X POST 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'
you might have to wait for a while, it might be up to 2 hours or even more in order to see that the job label has been deleted, since you deleted all its series, after that period you can run again :

curl http://localhost:9090/api/v1/label/job/values
{"status":"success","data":["blackbox exporter", "cadvisor","node-exporter","postgres-exporter","prometheus"]}
no trace for blackbox-exporter anymore

notice that if you want to delete series of a certain job that contain a space on its name, you will have to replace the space with %20

let's say I want to delete the series of blackbox exporter job, here is what you will have to do:

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="blackbox%20exporter"}'

prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
      - '--web.enable-admin-api'                               # << Like this
    ports:
      - 9090:9090
    expose:
      - 9090
    networks:
      - monitoring