Firmware upgrade

Upgrade procedure

This procedure should only be used when converting a cluster TO HyperCloud FROM HyperDrive or when performing a HyperCloud upgrade FROM a HyperCloud version prior to 2.1.

Overview

Updating the firmware involves flashing the new binary onto the target machine using the flash_all.sh script. This will update all of the firmware for the node (BMC, UEFI, etc). Once the script has finished flashing, the system will boot.

If this procedure is performed on a live Ceph cluster, it is recommended that Ceph is paused on each target machine, as each node needs to be shutdown first. See Pausing Ceph for how to do this. Once this has been done, shutting down the SOC as a first step for firmware updates is required.

Prerequisites

The BMC admin account needs to be enabled on the target machines.
The BMC IP address of the target machines must be known and accessible.
Network IPMI needs to be enabled on the BMC for all target machines.
The new firmware tarball must be unpacked and executed on a host machine that is connected to the BMC network of all the target machines.

Detailed instructions

Package(s)

HyperCloud Firmware v2.2.1: https://git.softiron.com/jenkins/hc-firmware-updater/v2.2.1/hypercloud-2.2.1-firmware.tar.gz - This is a SoftIron internal link.

Enable BMC admin account

If the target machine has not had the BMC admin account activated, this will be the first step required to log into the machine as admin and perform the upgrades. The application bmc_password, found in both SI Linux and HyperCloud, is run from the SOC side to set the admin password. Entering this command will prompt the user for an admin password; this process is required for each individual upgrade target.

$ bmc_password
Enter password. Limited to 16 characters
Enter password again.
BMC Password Updated

Confirm the BMC IP address

Execute ipmitool from the SOC to confirm the BMC IP address.

$ ipmitool lan get 1
IP Address  : X.X.X.X

Enable network IPMI access

For the target node, verify that network IPMI has been enabled and that you know the IPMI username and password. This can be done in two different ways:

Try to run ipmitool -H <BMC_IP> -U <ipmiusr> -P <ipmipasswd> mc info

Enable network IPMI and set the IPMI username and password for IPMI user number 2 on each node:

ipmitool user enable 2
ipmitool user set name 2 ipmiusr
ipmitool user set password 2 <password>

Perform the firmware update

After enabling and setting the BMC password(s) and IPMI network access, shutdown the OS on the target machine:

$ shutdown -h now || halt
Connection to hypercloud-node closed by remote host.
Connection to hypercloud-node closed.

Download and install BMC firmware

On the host machine, download the firmware tarball

On the host machine, extract the firmware tarball:

tar -xf hypercloud-2.2.1-firmware.tar.gz
cd hypercloud-2.2.1-firmware

Verify that the node you are about to flash has already been shutdown or halted.
On the host machine, execute the upgrade of the firmware on this one target node by running the flash_all script and specifying the BMC admin password, IPMI username, and IPMI password:
```
./flash_all.sh -U <ipmiusr> -P <ipmipasswd> -S <adminpasswd> -F rootfs.ubifs -H <BMC_IP>
```
This process of applying the firmware upgrade to a node will take about 5 minutes.

While the firmware updates apply, the serial terminal (either USB console or IPMI serial over LAN) may be disconnected on the node being updated.

Once the firmware upgrades are completed, the node will automatically boot.
Wait for the OS to boot