Sensu Go Upgrade

Upgrade Notes

• https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/upgrade/
• https://docs.sensu.io/sensu-go/latest/operations/deploy-sensu/install-sensu/
• https://docs.sensu.io/sensu-go/latest/operations/deploy-sensu/cluster-sensu/

WARNING

Aug 20 01:13:49 azv-sensu00 sensu-backend[23588]: {"component":"etcd","level":"error","msg":"request cluster ID mismatch (got 6757f62c6737d318 want 3b0efc7b379f89be)","pkg":"rafthttp","time":"2021-08-20T01:13:49Z"}

- memberid: 1117328545714761148
  name: ""
  err: 'etcdclient: no available endpoints'
  healthy: false


sensuctl cluster member-add mynode http://10.x.x.x:2380
then start cluster
sudo systemctl start sensu-backend && journalctl -xe --no-pager
Blow out etcd too via rm -rf /var/lib/sensu/sensu-backend/etcd/* if needed

NOTE: For systems that use systemd, run sudo systemctl daemon-reload before restarting the services.

To restart the Sensu agent, run:

sudo service sensu-agent restart
Copy
To restart the Sensu backend, run:

sudo service sensu-backend restart

Upgrade Steps for Cluster only to 5.14.2-7022

1. Let's make sure all packages are put on hold so apt upgrade doesn't upgrade sensu stuff

List Possible Versions

apt-cache madison sensu-go-backend

List Current Version

dpkg --list sensu-go-backend

1. Pin package version

vim /etc/apt/preferences.d/sensu

Package: sensu-go-backend
Pin: version 5.14.0-6578
Pin-Priority: 999

1. Upgrade Operating System to next LTS version and Apps via do-release-upgrade

sudo apt update && sudo apt upgrade # && apt autoremove
sudo reboot
sudo do-release-upgrade -c
sudo do-release-upgrade -f DistUpgradeViewNonInteractive

1. Check service status and listeners

systemctl status sensu-backend
ss -lntup
journalctl -f

1. Check cluster status and make sure this node member is online in cluster and healthy

sensuctl cluster member-list
sensuctl cluster health  # ???

Optional or other checks to look at

1. Double check sources and make sure it is now bionic and not xenial

view /etc/apt/sources.list.d/sensu_stable.list

# this file was generated by packagecloud.io for
# the repository at https://packagecloud.io/sensu/stable

deb https://packagecloud.io/sensu/stable/ubuntu/ bionic main
deb-src https://packagecloud.io/sensu/stable/ubuntu/ bionic main

1. Update to latest 5.14.2 version. We can remove pin and use hold if we want to

apt-cache madison sensu-go-backend
for pkg in $pkgs; do
  sudo apt-mark unhold $pkg
  sudo apt upgrade $pkg=5.14.2-7022
done

1. Set Sensu packages to hold on update again

for pkg in $pkgs; do
  sudo apt-mark hold $pkg
  sudo apt-mark showhold $pkg
done

Other Handy commands

List and remove one of the cluster members by id, disable services, or probably just upgrade. In order to do operating system upgrade we will have to uninstall sensu-go-backend

sensuctl cluster member-list
sensuctl cluster member-remove <id>

0. Optional. You can clone one of the hosts to just work through upgrade process on a different day to get familiar of process before the real cluster upgrade. Just an option. I don't think we need this but it is still is a good idea to do a test install before upgrade to make sure one is familiar with the parts of the upgrade.. https://docs.sensu.io/sensu-go/latest/operations/deploy-sensu/install-sensu/ There is focal/20.04 support for sensu

1. Snapshot each member of the current sensu cluster so we can revert quickly/easily if everything goes sideways when we move to using upgraded cluster.

2. Bring one of the hosts out of the cluster so we can upgrade OS and sensu related applications. Ubuntu attended do-release will prompt you on certain packages if relevant.

sudo apt update && sudo apt upgrade
sudo reboot
sudo do-release-upgrade
3. Now that OS is upgraded lets update our sensu repo and upgrade their packages.
Update /etc/apt/sources.list.d/sensu_stable.list to use focal instead of xenial

deb https://packagecloud.io/sensu/stable/ubuntu/ xenial main
deb-src https://packagecloud.io/sensu/stable/ubuntu/ xenial main
4. Make sure services like consul, sensu-backend ... are all working properly as much as possible. Check all logs for errors or warnings. Do local connection checks to services (like rabbit, redis) using creds.

5. Upgrade host #2 And repeat 2,3,4. If that looks good you should be able to move host1, host2 online and shutdown host3 and actually do the tests. Usually 30 mins to 60 mins or less if sufficient to test.

Here is current bounded application listeners on one of the host to give you an example of what is running. Where it is current upgrading should be pretty simple. I recommend going to 20.04. It's not that much harder to do that. We can go to 20.04 with minimal effort and then we won't be in a pinch again in two years if we support services that long.

Other docs to look at though we are only upgrading sensu go backend with OS
https://docs.sensu.io/sensu-go/latest/operations/deploy-sensu/install-sensu/ Ubuntu 20.04/focal is supported
https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/migrate/

sudo systemctl start sensu-backend && journalctl -xe --no-pager