Cluster Upgrade

There are 2 main scenarios that affect the way a cluster can be upgraded. Where the engineer has access to the root account and where they do not have access to the root account. Cloudian Certified Engineers must be familiar with both methods

Prerequisites and Requirements

Warning

All upgrades will have prerequisites and requirements. These will be listed in the release notes and associated documentation and must be read and understood thoroughly prior to undertaking an upgrade. Cloudian® release notes contain details of important key additions, enhancements, and fixes that are included in the particular release.

Info

Let us now consider some of the information from HS-8.1-releasenotes. For berevity only some of the available information is posted below. For a live environment you must read the full release notes for the version of firmware you are deploying.

We will now confirm that we understand any and all requirements we must meet before we continue with the upgrade.

Read the Release Notes - General Notes

The labs are already running Rocky Linux with our system set to UEFI
Should your system need upgrading from CentOS to Rocky you will need to utilise the latest version of the Software Upgrade Tool.
As we are already on version 8.0.1 we should be aware that the common.csv has been retired and we are now using the HS configuration database (managed through hsctl commands)
We are not using HyperSearch within this the training labs so the next note can be safely ignored
We are not employing SSL-offloading within the labs so again this next statement can be safely ignored after we have read and understood it all

Read the Release Notes - Upgrade Notes

As this is not a new install we will focus on the upgrade notes

We are running a supported version of HyperStore to continue with this upgrades
In the labs we understand that we are working with a limited system, in a live system the hardware requirements muct be achieved prior to considering upgrading HyperStore
We are not upgrading from 7.5.x, however we should still read through the notes to fully understand the potential issues. In this case we do not need to turn off concurrent repair nor restart the Hyperstore Service, and can continue to the next step
Best Practice suggests that any customizations within a cluster should always be fully documented. We should therefore already be aware of these non-default settings as we start the upgrade. We should still confirm any conflicts (differences) which are identified and apply them again as part of the upgrade process

Prepare System for Upgrade

Info

Experience has shown that there are a number of checks that help prepare the system for an upgrade. These should be completed prior to extracting the binaries

Instructions

SSH onto node 1 (Config Master) as sa_admin and start a shell session
First we need to check that all nodes are included in the upgrade process. To do this we will start the Cloudian installer
```
hspkg install
```

Instructions

Once we have started the installer script, select Advanced Configuration options
Follow this with option r ) Exclude node(s) from configuration updates
Type none at the prompt

Instructions

Having confirmed no nodes are excluded, we must exit the script
Next we run hsctl config preview to identify any outstanding changes yet to be applied to the cluster
```
hsctl config preview
```
Follow this with hsctl config apply ALL to ensure all changes are committed. We should run the apply regardless of whether we identify any changes outstanding or not
```
hsctl config apply All
```

Root Account Enabled Upgrade

Download Upgrade Contents

Info

In a real world scenario you need to use your existing HyperStore license when extracting the binaries, this will be available for download from the customer portal. Care must be taken to ensure you select the correct license from those available to you.

Instructions

To simplify matters we will download all files required for the upgrade
Login using SSH to your config master using root credentials
Change to the CloudianPackages directory

cd /root/CloudianPackages/

1. Create an upgrade directory and change to this new directory

mkdir binaries8.1
cd binaries8.1

Download the following files:
Cloudian HyperStore binary
Cloudian lab mem80 script

wget --user=admin --password=cloudian123 http://10.150.1.252:8080/images/CloudianHyperStore-8.1.bin  
wget --user=admin --password=cloudian123 http://10.150.1.252:8080/infra/mem80.sh

Note

As from Version 8.0, the license is no longer needed when unpacking the binaries for an upgrade. The extraction process will use the active license from the cluster.

Extract the binaries

Instructions

Use the chmod command to make the Cloudian HyperStore binary executable
For the lab only, we will also make the mem80 script executable

chmod +x *.bin *.sh

1. Ensure the file is executable

ll

Extract binary

./CloudianHyperStore-8.1.bin

Rerun Memory Tunables (LAB ONLY!)

Warning

As mentioned in previous labs, it is critical that mem80.sh is run immediately after unpacking and before running the installer. Failure to run this script may make your lab environment unresponsive and likely require the nodes to be terminated and recreated requiring you to setup all your networking again

Instructions

Run the shell mem80.sh script and check the tuneables have been set by viewing the changes through hsctl

./mem80.sh
hsctl config get all |grep heap

Hyperstore upgrade

Instructions

Once extracted, run the cloudianInstall.sh from the extracted contents from the /opt/cloudian-staging/8.1/ directory
```
/opt/cloudian-staging/8.1/cloudianInstall.sh
```
You will be presented with a menu, and an option to upgrade from 8.1
Select this option (3)
When asked to run basic tests, type no
When asked if you want to proceed with upgrade, type yes
Monitor the upgrade

Root Account Disabled Upgrade

Info

Within the labs we have mounted some external media which houses the required files. You will need to mount your own drive or copy over the files using SCP or similar.

Additionally in a real world scenario you need to use your existing HyperStore license when extracting the binaries, this will be available for download from the customer portal. Care must be taken to ensure you select the correct license from those available to you.

Configure HSH if not already completed

Instructions

SSH into your Config / Installer node (Node 1) as training
Check / Ensure HSH is currently disabled by running
```
sudo hsctl config get hsh.enabled
```
Enable HSH by running
```
sudo hsctl config set hsh.enabled=true
```
Apply the new config to all nodes in the cluster
```
sudo hsctl config apply hsh
```
Exit the training ssh session.
```
exit
```

Download Upgrade Contents

Info

Within the labs we have mounted some external media which houses the required files to save time. In your own environments you will need to mount your own drive or copy over the files using SCP or similar.

Instructions

SSH into your Config Controller Node (node1) as sa_admin
Create an upgrade directory
```
mkdir CloudianPackages8.1
```
Change to that directory
```
cd CloudianPackages8.1
```

Copy the upgrade binary to your upgrade directory

cp /studentfiles/CloudianHyperStore-8.1.bin .

Copy the upgrade signature file to your upgrade directory
```
cp /studentfiles/CloudianHyperStore-8.1.bin.sig .
```
Ensure files are as expected
```
ls -l
```

Software Upgrade

Instructions

Extract binary by executing

hsrun --root CloudianHyperStore-8.1.bin

Warning

Ensure that the version you are unpacking is the version you are upgrading to. Unpacking an older binary version (such as your current deployment version) will cause problems with your cluster.

Instructions

Once extracted, you will get an output such as the one in the screenshot above.
Run the command hspkg install to continue with the upgrade
```
hspkg install
```

Instructions

You will be presented with a menu, and an option to upgrade from 8.0.1 to 8.1
Type '3' to select the option to upgrade then press Enter
When asked if you would like to run basic tests after the upgrade, answer No

The upgrade will take about 50 minutes to complete.

Instructions

Once the upgrade finishes, you will return to the menu of the cloudianInstall.sh script
Select 'x' to exit the script

Warning

It is important to exit out of the install script when you are done even if you have further commands you wish to run as the binaries have been updated. It is also best practice to always sign out of any script you have invoked as operations done through the CMC will also utilize the same script. If there is a process using the script (such as your terminal), these operations will fail.