Sensu Go Upgrade

Upgrade Notes

https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/upgrade/

NOTE: For systems that use systemd, run sudo systemctl daemon-reload before restarting the services.

To restart the Sensu agent, run:

sudo service sensu-agent restart
Copy
To restart the Sensu backend, run:

sudo service sensu-backend restart

Upgrade Steps for Cluster only to 5.14.2-7022

1. Let's make sure all packages are put on hold so apt upgrade doesn't upgrade sensu stuff

pkgs="sensu-go-backend sensu-go-cli sensu-go-agent"
for pkg in $pkgs; do
  sudo apt-mark hold $pkg
  sudo apt-mark showhold $pkg
done

1. List and remove one of the cluster members by id, disable services, or probably just upgrade. I don't think it really matters

sensuctl cluster member-list
sensuctl cluster member-remove <id>

3.

sudo apt update && sudo apt upgrade
sudo reboot
sudo do-release-upgrade

1. Upgrade Sensu Repo to use bionic instead of xenial /etc/apt/sources.list.d/sensu_stable.list

# this file was generated by packagecloud.io for
# the repository at https://packagecloud.io/sensu/stable

deb https://packagecloud.io/sensu/stable/ubuntu/ bionic main
deb-src https://packagecloud.io/sensu/stable/ubuntu/ bionic main

1. Update to 5.14.2 version

apt-cache madison sensu-go-backend
for pkg in $pkgs; do
  sudo apt-mark unhold $pkg
  sudo apt upgrade $pkg=5.14.2-7022
done

1. Set Sensu packages to hold on update again

for pkg in $pkgs; do
  sudo apt-mark hold $pkg
  sudo apt-mark showhold $pkg
done

0. Optional. You can clone one of the hosts to just work through upgrade process on a different day to get familiar of process before the real cluster upgrade. Just an option. I don't think we need this but it is still is a good idea to do a test install before upgrade to make sure one is familiar with the parts of the upgrade.. https://docs.sensu.io/sensu-go/latest/operations/deploy-sensu/install-sensu/ There is focal/20.04 support for sensu

1. Snapshot each member of the current sensu cluster so we can revert quickly/easily if everything goes sideways when we move to using upgraded cluster.

2. Bring one of the hosts out of the cluster so we can upgrade OS and sensu related applications. Ubuntu attended do-release will prompt you on certain packages if relevant.

sudo apt update && sudo apt upgrade
sudo reboot
sudo do-release-upgrade
3. Now that OS is upgraded lets update our sensu repo and upgrade their packages.
Update /etc/apt/sources.list.d/sensu_stable.list to use focal instead of xenial

deb https://packagecloud.io/sensu/stable/ubuntu/ xenial main
deb-src https://packagecloud.io/sensu/stable/ubuntu/ xenial main
4. Make sure services like consul, sensu-backend ... are all working properly as much as possible. Check all logs for errors or warnings. Do local connection checks to services (like rabbit, redis) using creds.

5. Upgrade host #2 And repeat 2,3,4. If that looks good you should be able to move host1, host2 online and shutdown host3 and actually do the tests. Usually 30 mins to 60 mins or less if sufficient to test.

Here is current bounded application listeners on one of the host to give you an example of what is running. Where it is current upgrading should be pretty simple. I recommend going to 20.04. It's not that much harder to do that. We can go to 20.04 with minimal effort and then we won't be in a pinch again in two years if we support services that long.

Other docs to look at though we are only upgrading sensu go backend with OS
https://docs.sensu.io/sensu-go/latest/operations/deploy-sensu/install-sensu/ Ubuntu 20.04/focal is supported
https://docs.sensu.io/sensu-go/latest/operations/maintain-sensu/migrate/