Skip to content

HyperCloud Releases

IMPORTANT NOTE

Firmware guidelines from previous releases are still applicable. If upgrading from a release prior to 2.1.x, it is recommended to manually update firmware prior to upgrading.

HyperCloud 2.3.5

Released June 10, 2024

Common

New Features:

  • Added support for HTTP to HTTPS redirection by default.

Bug Remediation:
(Including internal SoftIron CVEs)

  • Resolved misspelling of certain backend nodes after the si-* node name transition in a prior release.
  • Resolved an issue where cluster control facts (such as license or authorized SSH keys) may be overwritten when the dashboard is rebooted.
  • Resolved issue where changing dashboard network details without NTP configured may result in losing connectivity to the dashboard.
  • Resolved issue where the dashboard backend database may not be able to handle more connection requests on a very large, busy cluster.
  • Resolved Manifold API CLI client timeout issues.
  • Resolved an issue where unsupported reboot scripts were bundled in the distribution. Users may find this inclusion confusing while undocumented.

Enhancements:

  • Updated full disk encryption service to support longer key lengths. RSA keys of 1024, 2048, 3072, and 4096 may be used.
  • Updated compression library for xz, improving performance of some daemons such as the dashboard startup.
  • Updated cluster reboot process to check for and resolve stray storage devices, in the event stray devices were added to the cluster by a previously resolved bug.
  • Updated back-end hypervisor orchestration to more quickly boot virtual machines. Previously, the deploy and boot process may take up to two minutes. This has been reduced to 30 seconds or less.

HyperCloud (stable)

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v6.10
  • X86 UEFI (>= Hx7xxx models): v1.30
  • BMC: v8.30

HyperCloud 2.3.4

Released May 17, 2024

Common

Bug Remediation:
(Including internal SoftIron CVEs)

  • Resolved an issue where the next-generation UI, Glasshouse, would report your session expired when an incorrect password is entered.
  • Resolved an issue where some erroneous text may be printed to the console during a cluster-wide reboot.
  • Resolved an issue where downloading a marketplace appliance may erroneously report the incorrect product name.
  • Resolved an issue in the next-generation UI, Glasshouse, where some functions would be non-functional due to expiring back-end authentication tokens without any user feedback.

HyperCloud

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v6.10
  • X86 UEFI (>= Hx7xxx models): v1.30
  • BMC: v8.30

HyperCloud 2.3.3

Released May 2, 2024

Common

External library and security remediations:
(Fixed or verified inapplicability)

  • CVE-2024-2201 (InSpectre Gadget)

Bug Remediation:
(Including internal SoftIron CVEs)

  • Resolved a race condition where an image upload's temporary files may be cleaned up before the image is actually imported to the datastore via the Glasshouse GUI.
  • Resolved an issue where the built-in account passwords could erroneously be changed via the Glasshouse GUI.
  • Resolved an issue in the Glasshouse GUI where some graphs were being improperly displayed.
  • Resolved an issue where confusing data may be displayed in the Glasshouse GUI for some attributes of marketplaces and datastores.
  • Resolved issue where creation of a blank image may fail in the Glasshouse UI.

HyperCloud

Bug remediation:
(including internal SoftIron CVEs)

  • Resolved issue where FILE datastore uploads may fail via the Legacy Web GUI.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v6.10
  • X86 UEFI (>= Hx7xxx models): v1.30
  • BMC: v8.30

HyperCloud 2.3.2

Released April 29, 2024

Common

External library and security remediations:
(Fixed or verified inapplicability)

  • CVE-2024-2511

Bug Remediation: (Including internal SoftIron CVEs)

  • Resolved an issue with uploading large files in the next-gen Glasshouse GUI.
  • Resolved an issue where the reboot command may fail if no virtual machines are running on a node.
  • Resolved issue where NTP may fail to properly configure on the Dashboard.
  • Resolved multiple issues that may result in automatic reboot of VMs not occurring in the event of a host failure in the cluster.
  • Resolved issue where the assigned disk BUS field may be populated with incorrect data when choosing a device prefix (such as vd for VirtIO, sd for SCSI, or hd for IDE).
  • Resolved issue where some Dashboard commands may cut off part of the compute node hostnames.

HyperCloud

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v6.10
  • X86 UEFI (>= Hx7xxx models): v1.30
  • BMC: v8.30

HyperCloud 2.3.1

Released April 14, 2024

Common

External library and security remediations:
(Fixed or verified inapplicability)

  • CVE-2023-30590

Bug Remediation:
(Including internal SoftIron CVEs)

  • Resolved data corruption issue with creating template and image from existing VM via "save as" function.
  • Resolved version reporting issue with some internal tools.
  • Resolved image upload issues with next-generation UI, "Glasshouse". This includes both "image" and "file" uploads.
  • Resolved issue where cluster is temporarily in an unsupported storage state when transitioning from one to three nodes for a fully redundant storage cluster. Now, a second cluster "monitor" daemon does not come online until there are at least three hosts available to avoid potential cluster split-brain, preventing the cluster from coming back up in the event of a failure during this migration.
  • Resolved hang on boot, when time synchronization fails due to network unavailability.
  • Resolved issue where auto-respawn of workloads, if a compute node fails, would not properly trigger.
  • Resolved issue where the syslog data may contain binary data.
  • Resolved issue where the storage SMART daemon may log excessive and erroneous temperature data.
  • Resolved issue where arguments to hypervisor kernel module would not properly be passed.

Enhancements:

  • Removed ability to rename nodes from the next-gen UI. This is not a supported action.
  • Removed "automatic VLAN ID" support from next-gen UI. This is not a supported action.
  • Updated "halt" and "reboot" commands to print impact of performing these actions on nodes running workloads.
  • Updated kernel to no longer panic on first OOPS to aid troubleshooting. Instead, we don't panic until 10,000 OOPS events.

HyperCloud

Bug remediation:
(including internal SoftIron CVEs)

  • Resolved issue with NVIDIA GRID support on SoftIron accelerated compute nodes equipped with two NVIDIA GPUs.
  • Resolved firmware version reporting issue.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v6.10
  • X86 UEFI (>= Hx7xxx models): v1.30
  • BMC: v8.30

HyperCloud 2.3.0

Released April 1, 2024

Common

New Features:

  • Unified multiple SoftIron products to share a common base of libraries, including:
    • HyperCloud
    • VM Squared
    • Cloud Appliance Images
      • Load Balancer
      • Gateway
      • iSCSI
      • MinIO
  • Added next-generation UI from SoftIron, Glasshouse. This is the default UI in VM Squared, but is also available for use in HyperCloud.

Enhancements:

  • Enabled per-process accounting to allow for more granular resource utilization tracking.
  • Updated user-interactive processes to have higher preemption priority over background tasks to improve user interface responsiveness.
  • Improved handling of memory pressure on all node types.

Bug remediation:
(including internal SoftIron CVEs)

  • Resolved issue where backend storage placement groups may get stuck in a peering state forever.
  • Resolved issue with hypercloud and vmsquared CLI wrappers when arguments have spaces.

External library and security remediations: (Fixed or verified inapplicability)

  • No change from previous release.

HyperCloud

Bug remediation:
(including internal SoftIron CVEs)

  • Resolved issue with power usage reporting on nodes with GPUs combined with more resource-intensive CPUs.
  • Resolved issue with single-replica storage when deleting exclusion groups and/or vdisks.

Enhancements:

  • Updated cluster reboot scripts to support a cleanup flag.
  • Updated cluster decom scripts to support static nodes.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v6.10
  • X86 UEFI (>= Hx7xxx models): v1.30
  • BMC: v8.30

HyperCloud 2.2.4

Released March 25, 2024

External library and security remediations:
(Fixed or verified inapplicability)

  • CVE-2023-4692
  • CVE-2023-4693
  • CVE-2023-52434
  • CVE-2023-7192
  • CVE-2024-0646
  • CVE-2024-26585
  • CVE-2023-23005
  • CVE-2023-33951
  • CVE-2023-33952
  • CVE-2023-34256
  • CVE-2023-52452
  • CVE-2024-0565
  • CVE-2024-26587
  • CVE-2024-26590
  • CVE-2023-7104
  • CVE-2023-46838
  • CVE-2023-50431
  • CVE-2023-52429
  • CVE-2023-52434
  • CVE-2023-52435
  • CVE-2023-52443
  • CVE-2023-52444
  • CVE-2023-52445
  • CVE-2023-52447
  • CVE-2023-52448
  • CVE-2023-52449
  • CVE-2023-52451
  • CVE-2023-52454
  • CVE-2023-52456
  • CVE-2023-52457
  • CVE-2023-52458
  • CVE-2023-52462
  • CVE-2023-52463
  • CVE-2023-52464
  • CVE-2024-0340
  • CVE-2024-1085
  • CVE-2024-1086
  • CVE-2024-1151
  • CVE-2024-23849
  • CVE-2024-23850
  • CVE-2024-23851
  • CVE-2024-24860
  • CVE-2024-26581
  • CVE-2024-26582
  • CVE-2024-26583
  • CVE-2024-26586
  • CVE-2024-26588
  • CVE-2024-26589
  • CVE-2024-26591
  • CVE-2024-26592
  • CVE-2024-26593
  • CVE-2024-26594
  • CVE-2024-26597
  • CVE-2024-26598
  • CVE-2024-26599
  • CVE-2024-26600
  • CVE-2024-26601
  • CVE-2024-26602
  • CVE-2024-26603

Bug remediation:
(including internal SoftIron CVEs)

  • Resolved issue where hypercloud-reboot-cluster scripts may try to "reboot" public clouds when cloud bursting is configured.

Enhancements:

  • Updated snapshot daemon to support disabling calculation of usage which may cause excessive resource consumption when backing up multi-hundred-TiB or PiB-sized images.
  • Updated SSH on all nodes to use pre-computed moduli, dramatically reducing CPU usage on boot.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v5.11
  • X86 UEFI (>= Hx7xxx models): v1.20
  • BMC: v8.21

HyperCloud 2.2.3

Released February 29, 2024

External library and security remediations:
(Fixed or verified inapplicability)

  • CVE-2024-0853
  • CVE-2024-0985
  • CVE-2023-52425
  • CVE-2023-52426
  • CVE-2023-46045

Bug remediation:
(including internal SoftIron CVEs)

  • Resolved issue where configured backups may be missed if there was a scheduling delay.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v5.11
  • X86 UEFI (>= Hx7xxx models): v1.20
  • BMC: v8.21

HyperCloud 2.2.2

Released February 7, 2024

Enhancements:

  • Improved the snapshot/backup service to perform remote backup full transfers only at onset (subsequent transfers will always be differentials).
  • Improved the snapshot/backup service to restore using clones rather than full copies of the original data for remote backups.
  • Improved cluster floating services to allow booting of cluster, once bare minimum services are available, as opposed to waiting until all services are up.

Security and Bug Remediation:

External library security remediations:
(fixed or verified inapplicability)

  • glibc qsort corruption bug (No Assigned CVE)
  • CVE-2023-22796
  • CVE-2023-6246
  • CVE-2023-6779
  • CVE-2023-6780
  • CVE-2024-0232
  • CVE-2024-0553
  • CVE-2024-0567
  • CVE-2024-0727
  • CVE-2023-6129
  • CVE-2023-6237
  • CVE-2023-45853
  • CVE-2023-1032
  • CVE-2023-1206
  • CVE-2023-3212
  • CVE-2023-3390
  • CVE-2023-35827
  • CVE-2023-3609
  • CVE-2023-3611
  • CVE-2023-3776
  • CVE-2023-39189
  • CVE-2023-39192
  • CVE-2023-39193
  • CVE-2023-39194
  • CVE-2023-40283
  • CVE-2023-4206
  • CVE-2023-4207
  • CVE-2023-4208
  • CVE-2023-4244
  • CVE-2023-42753
  • CVE-2023-42754
  • CVE-2023-45862
  • CVE-2023-45863
  • CVE-2023-4623
  • CVE-2023-46862
  • CVE-2023-4921
  • CVE-2023-5178
  • CVE-2023-5197
  • CVE-2023-5633
  • CVE-2023-5717
  • CVE-2023-6546
  • CVE-2023-6560
  • CVE-2023-42755
  • CVE-2023-44466
  • CVE-2023-51780
  • CVE-2023-51781
  • CVE-2023-51782
  • CVE-2024-0340
  • Resolved issue where the S3 object storage service may sometimes become unavailable.
  • Resolved issue that causes hypercloud-decom-node to fail when running from the dashboard.
  • Resolved issue where HYPERCLOUD_LOCAL_SNAPUSAGE may not report all volumes.
  • Resolved issue where snapshot daemon's local CLI commands may return an error.
  • Resolved issue where snapshot daemon may not properly name persistent replicated images.
  • Resolved issue where the software defined storage may erroneously wipe and ingest cache devices if the node was previously in an unsupported topology.
  • Resolved issue where zgrep could not properly locate the grep command.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v5.11
  • X86 UEFI (>= Hx7xxx models): v1.20
  • BMC: v8.21

HyperCloud 2.2.1

Released January 1, 2024

Security and Bug Remediation:

  • Remediated CVE-2023-6931
  • Remediated CVE-2023-6932
  • Remediated CVE-2023-4132
  • Remediated CVE-2023-45871
  • Remediated CVE-2023-40791
  • Remediated CVE-2023-1194
  • Remediated CVE-2023-6622
  • Remediated CVE-2023-6817
  • Remediated CVE-2020-35512
  • Remediated CVE-2022-42010
  • Remediated CVE-2022-42011
  • Remediated CVE-2022-42012
  • Remediated CVE-2023-34969
  • Resolved issue where link to test next generation GUI from legacy GUI is missing.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v5.11
  • X86 UEFI (>= Hx7xxx models): v1.20
  • BMC: v8.21

HyperCloud 2.2.0

Released December 30, 2023

IMPORTANT NOTE

Upgrade to BMC Release >= v8.20 to take advantage of automatic firmware management features moving forward. This will represent the LAST discrete firmware upgrade required in HyperCloud clusters.

IMPORTANT NOTE

Upgrade to BMC release >= v6.12 PRIOR to upgrading to HyperCloud >= 2.0.2. This is especially important on HD21XXX-model nodes. If the cluster is upgraded to HyperCloud >= 2.0.2 without upgrading the BMC, HyperCloud will no longer be able to detect the caddies and cannot bring the node storage back online.

IMPORTANT NOTE

When upgrading from HyperCloud 2.0.0 or 2.0.1, double check the output of ceph fs status from the Dashboard CLI prior to beginning. If the number of clients reported exceeds 3 times the number of nodes in the cluster, this must be addressed first via support by wiping all MDS sessions and restarting all ceph-mds daemons. Failure to address this before the upgrade may lead to a storage outage during the upgrade process.

New Features:

  • Added link to next generation (beta) GUI from the default GUI login screen. Users are encouraged to test the new features in this release.
  • Added support for the decommissioning of dynamic storage nodes and compute nodes via the hypercloud-decom-node command. Static storage node support will come in a subsequent release.
  • Added support to the backup service to retry backups if the backup window was missed.

Enhancements:

  • Updated backend software-defined storage platform to more aggressively perform background data integrity checks.
  • Updated storage nodes to always suspend bringing storage online if minimum firmware or hardware layout requirements are not met.
  • Updated system utilities to reliably name NICs and disks the same and with helpful names across reboots.
  • Updated system reboot and shutdown routines to print more informational messages.
  • Updated network configuration to handle case where the switches come up much later than the nodes.
  • Updated next generation (beta) GUI to talk to the HyperCloud API over a relative path to support instances where a user port forward access over SSH.
  • Updated logging configuration to log more information from the software-defined storage.
  • Updated HyperCloud decommission commands to print more helpful feedback.

Security and Bug Remediation:

  • Remediated CVE-2023-48795 (Terrapin)
  • Remediated CVE-2023-40238 (logoFAIL)
  • Remediated CVE-2023-5868
  • Remediated CVE-2023-5869
  • Remediated CVE-2023-5870
  • Remediated CVE-2023-46136
  • Remediated CVE-2023-41913
  • Remediated CVE-2022-3775
  • Remediated CVE-2023-46218
  • Resolved issue where firmware updates would sometimes fail to apply.
  • Resolved issue where remote backups would fail if the destination pool does not match the source pool name.
  • Resolved issue where remote backups would fail if the remote image is in an unexpected state.
  • Resolved issue where superseded hotpatches may not get removed after an upgrade completes.
  • Resolved issue where large clusters may have substantial clock drift during upgrades from releases before 2.1.
  • Resolved issue where large image imports may cause the dashboard to run out of memory.
  • Resolved issue where the S3-compatible object storage service was not being started in FIPS mode on FIPS-enabled clusters.
  • Resolved issue where node affinity was not properly being enforced within the HyperCloud Orchestrator.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v5.11
  • X86 UEFI (>= Hx7xxx models): v1.20
  • BMC: v8.21

HyperCloud 2.1.2

Released November 13, 2023

Security and Bug Remediation:

  • Remediated CVE-2023-5678
  • Remediated CVE-2023-36632
  • Resolved an issue where networking may not come up bonded with the correct protocol. Addresses a regression introduced in HyperCloud 2.1.0.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v5.10
  • X86 UEFI (>= Hx7xxx models): v1.10
  • BMC: v8.20

HyperCloud 2.1.1

Released November 10, 2023

New Features:

  • Added support for disabling encryption for remote backups, potentially greatly improving performance when the network is fast enough that SSH encryption is the bottleneck. To enable, create the file /var/run/cluster-control/facts/ssh-allow-cipher-none and reboot the dashboard.
  • Added support for communicating with the HyperCloud API over the default TLS port, 443.

Enhancements:

  • Updated the HyperCloud snapshot daemon to replicate multiple disks per VM in parallel. Previously, multiple VMs would replicate in parallel, but one disk at a time in serial.
  • Updated the next-generation GUI to communicate with the HyperCloud API over the default TLS port, 443. Useful for more strict network environments where it may be challenging to allow another port or add another certificate exception in the event the cluster is still using a self-signed certificate.
  • Updated the Ceph monitor daemon to add sanity checking when evaluating CRUSH rules in the event set_choose_tries is abnormally high when an end user manually creates a CRUSH rule. If set very high and OSDs are down, the cluster may lose quorum and can take an hour or even days to recover.
  • Updated the system to more gracefully suspend storage services on cluster node shutdown and reboot events.

Bug Remediation:

  • Fixed an issue where hypercloud-reboot- scripts may hang during an upgrade due to slight time skew between nodes.
  • Fixed an issue where +dirty may be added to the version of HyperCloud in the GUI, CLI, and API.
  • Fixed an issue where Ceph may enter an erroneous HEALTH_WARN state claiming that there are "laggy" OSDs.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v5.10
  • X86 UEFI (>= Hx7xxx models): v1.10
  • BMC: v8.20

HyperCloud 2.1.0

Released November 4, 2023

New Features:

  • Added support for automatic management of SoftIron firmware which presently includes BMC, UEFI, and u-Boot firmware images in the distribution upgrade bundle. BMC release >= v8.20 is required to enable this functionality.
  • Added support for bursting to and managing Microsoft Azure Virtual Machines.
  • Added support for monitoring individual node serial number and BMC IP addresses via cluster control.
  • Added support for sane scheduler defaults when using mixed CPU architecture clusters. The default instance type is a KVM VM on x86_64, otherwise it can be overridden per deployment.
  • Added support for logging to external syslog server over UDP. Added support for automatically repairing data checksum errors as they are found without requiring operator intervention.
  • Added support for viewing storage nodes in the next generation UI.
  • Added support for monitoring hardware/firmware features for storage and compute nodes (e.g., fan speed, chassis and disk temperatures, firmware versions, etc.) in the next generation UI.
  • Added support for tuning whether or not to allow overlapping backup schedules in the event one runs long.
  • Added version 2 of the HyperCloud RESTful API. Public documentation for the API is available at: HyperCloud API: https://softiron.github.io/manifold-doc/

Enhancements:

  • Disabled the ability to delete the "serveradmin" user. Deleting this user breaks some GUI functionality and thus it should never be deleted.
  • Improved time synchronization between cluster nodes, resolving an issue where the cluster may experience issues if nodes are brought online that are widely out of sync with one-another.
  • Added more sanity checking to virtual network creation. It is no longer possible to create virtual networks on HyperCloud internal networks. Additionally, if a virtual network reuses a VLAN ID, its bridge name must also match. Reusing the same bridge across multiple VLAN IDs is also not supported, as this configuration may cause a network loop.
  • Added network performance testing tools to all node types.
  • Disabled write caching on journal SSDs. This actually improves performance and reliability.
  • Improved various part of the boot process to speed up node boot time.

Firmware Bundled:

  • ARM UEFI: v1.40
  • X86 UEFI (<= Hx6xxx models): v5.10
  • X86 UEFI (>= Hx7xxx models): v1.10

  • BMC: v8.20

    • This release also adds a local IPMI client interface for running ipmitool directly on the BMC. This is a work-around to fix remote IPMI security weaknesses by disabling IPMI ports at the BMC firewall and implementing a local IPMI client directly on the BMC. This has the benefit of removing the need for IPMI users and their associated management. By logging directly into the BMC and using IPMI in the shell customers will only require BMC admin credentials.

    Note

    The existing IPMI mechanisms will continue to be supported.

Security and Bug Remediation:

  • Remediated CVE-2023-5363
  • Remediated CVE-2023-38545
  • Remediated CVE-2023-38546
  • Remediated CVE-2023-5156
  • Resolved an issue in the GUI where creating a template from a running instance may fail.
  • Resolved an issue where the HyperCloud orchestrator reported the metadata pool's usage information rather than the data pool when using a separate data pool for EC datastores.
  • Resolved an issue where the device health metrics pool was not properly being managed by HyperCloud.
  • Resolved an issue where a disk may get ingested and added to the CRUSH map with an improper weight.
  • Resolved an issue where poweroff, reboot, and terminate may not actually perform the action on the instance.
  • Resolved an issue where temporary files on nodes may cause the rootfs to fill up.
  • Resolved an issue where the kernel build time was reported as the epoch time.
  • Resolved an issue where 10Gb/s NICs may not properly show up on SoftIron x86 nodes.
  • Resolved an issue where the HyperCloud Orchestrator temporary staging files may cause the dashboard rootfs to fill up.
  • Resolved an issue where SSH options were not properly used by the backup service.
  • Resolved an issue where the dashboard may take a long time to boot up due to a race condition between compute nodes.

HyperCloud 2.0.4

Released September 27, 2023

Enhancements:

  • Added support for specifying a separate data pool for remotely replicated backup images. This is useful if there is a desire to place the image metadata on a triple replicated pool, but the image backing data is on an EC-protected pool.
  • Added support for clean up of superseded customer-specific hot fixes with no user interaction required.

Security and Bug Remediation:

  • Remediated CVE-2023-4527
  • Remediated CVE-2023-38039
  • Remediated CVE-2023-4807
  • Remediated CVE-2021-3695
  • Remediated CVE-2021-3696
  • Remediated CVE-2021-3697
  • Resolved issue where restoration of backups in an EC-protected data pool may attempt to pull data from the incorrect pool.
  • Resolved issue where during a reboot of a note, OSDs may never come online on clusters with full disk encryption enabled.
  • Resolved issue where replication of an EC-protected image to a remote cluster may attempt to pull data from the incorrect pool.
  • Resolved issue where in certain error conditions, the kernel will no longer be able to communicate with the BMC.

HyperCloud 2.0.3

Released August 31, 2023

New Features:

  • Added EXPERIMENTAL support for v2 of the HyperCloud API

Enhancements:

  • Updated /etc/os-release to include version rather than reference to another file
  • Updated maximum disks per instance from 20 to 255
  • Updated IOMMU configuration for nodes with 100G NICs, potentially significantly improving performance
  • Updated hypervisor to require less "boilerplate" information for ARM VMs

Security and Bug Remediation:

  • Remediated CVE-2022-48560
  • Remediated CVE-2022-48564
  • Remediated CVE-2023-40217
  • Remediated CVE-2021-32292
  • Remediated CVE-2022-48554
  • Remediated CVE-2023-4016
  • Remediated CVE-2023-39417
  • Remediated CVE-2023-39418
  • Remediated CVE-2023-28736
  • Remediated CVE-2023-28938
  • Remediated CVE-2023-33953
  • Remediated CVE-2023-20569
  • Resolved issue with non-admin users authenticating against the HyperCloud API
  • Resolved issue where single replica ceph (EXPERIMENTAL FEATURE) heavily weights data placement into PG 0
  • Resolved API timeouts on the legacy web frontend
  • Resolved issue on the console where sending SIGINT (Ctrl+C) may result in the shell exiting
  • Resolved issue with S3 user syncing after enabling bucket-level syncing
  • Resolved issues with enabling LDAP authentication on frontend
  • Resolved issue where hypercloud-decom-node should not have previously been allowed on static nodes
  • Resolved issues with LLDP on nodes with 100G NICs
  • Resolved issue with ARM compute nodes unexpectedly rebooting when static nodes are X86
  • Resolved issue with slow DB queries when polling for large amounts of accounting data
  • Resolved issue with EFI VMs where the EFI vars file may not properly get cleaned up
  • Resolved issue with EFI VMs where the EFI vars file permissions may be too restrictive, causing the VM to not boot
  • Resolved issue where the node hosting the dashboard may not get rebooted during a cluster rolling reboot
  • Resolved issue where during a rolling reboot nodes may transition to "ON" too quickly, even if they are not ready to host workloads
  • Resolved issue where on "Density" nodes (those that utilize bcache) removing and re-adding a caddy may result in data loss

HyperCloud 2.0.2

Released August 1, 2023

IMPORTANT NOTE

Upgrade to BMC firmware release ≥ v6.12 PRIOR to upgrading to HyperCloud 2.0.2. This is especially important on HD21XXX-model nodes. If the cluster is upgraded to HyperCloud 2.0.2 without upgrading the BMC, HyperCloud will no longer be able to detect the caddies and cannot bring the node storage back online.

IMPORTANT NOTE

When upgrading from HyperCloud 2.0.0 or 2.0.1, double check the output of ceph fs status from the Dashboard prior to beginning. If the number of clients reported exceeds 3 times the number of nodes in the cluster, this must be addressed first via support by wiping all MDS sessions and restarting all ceph-mds daemons. Failure to address this before the upgrade may lead to a storage outage during the upgrade process.

New Features:

  • Added support for setting BLOCKIOSIZE_LOGICAL and BLOCKIOSIZE_PHYSICAL on a per-disk basis, if the default of 512 is not desired for the workload

Enhancements:

  • Updated SSD cache tuneables for Density (HDD) nodes to allow for significant performance improvements on some workloads
  • Updated backup services to support better multi-threading
  • Updated cluster name on CLI to be more readable

Security and Bug Remediation:

  • Remediated CVE-2022-2873
  • Remediated CVE-2022-28733
  • Remediated CVE-2022-28734
  • Remediated CVE-2022-28736
  • Remediated CVE-2022-40617
  • Remediated CVE-2023-32732
  • Remediated CVE-2023-20593
  • Remediated CVE-2023-38408
  • Remediated CVE-2023-3446
  • Remediated CVE-2023-0330
  • Remediated CVE-2023-2454
  • Remediated CVE-2023-2455
  • Remediated CVE-2023-1999
  • Remediated CVE-2023-2975
  • Resolved issue with booting VMs on ARM compute nodes
  • Resolved issue that may cause the internal shared cluster filesystem to eventually become unavailable due to infinite loop condition
  • Resolved issue where orchestrator CLI may not properly parse user inputs if provided during template instantiation
  • Resolved issue with backend storage object storage device memory management
  • Resolved issue where a shell may spin and consume 100% CPU if the remote session unexpectedly disconnects
  • Resolved issue where backup daemon CLI only displays workloads owned by the admin account

HyperCloud 2.0.1

Released July 3, 2023

Bug Remediation:

  • Resolved an issue with PCI pass-through of GPU caddies
  • Resolved an issue where OneGate authentication may fail
  • Updated snapshot daemon (snapper) to latest release, resolving some minor issues with archive backups
  • Removed unused and deprecated Marketplaces

HyperCloud 2.0.0

Released June 29, 2023

New Features:

  • Introduced HyperCloud Native RESTful API
  • Migrated to new HyperCloud Orchestrator-based orchestration framework
  • Introduced new, threaded, high performance backup daemon for local, remote, and archive backups handled natively within HyperCloud
  • Migrated to new, modern cryptographic framework while maintaining FIPS 140-2 compliance. This software is FIPS 140-3 in process.
  • Updated S3 backend to run on all nodes in the cluster, dramatically improving S3 performance.
  • Updated to run additional S3 daemon on the dashboard itself, to be used for manual configuration of S3 to S3 relationships.
  • Added support for GPU and HyperCast nodes via PCI pass-through to workloads.
  • Added support for LDAP (and AD) integration for API and Web GUI.

New Beta Features:

  • Introduced single replica Ceph support for nested SDS solutions
  • Introduced new Node.js-based Web UI as an alternative to current Web UI
  • Introduced Firecracker hypervisor for micro VMs.

Improvements:

  • Enabled NIC offloading by default. For some workloads this may result in 600% performance improvement.
  • Upgraded to latest LTS Linux kernel, 6.1. Most notably, this can result in dramatic performance improvements for some workloads, especially nested virtualization.
  • Updated hypercloud-generate-support-bundle to gather firmware information as part of the support tarball.

Security and Bug Remediation:

  • Implemented 100% CVE Remediation in the software development lifecycle for HyperCloud. As of shipping, HyperCloud 2.0 has no known CVEs.
  • Resolved issue where in some rare cases a second simultaneous dashboard may boot
  • Resolved issue where forwarded SSH agents may break hypercloud-reboot-* commands
  • Resolved issue with generating network topology diagrams if interfaces have special characters

SoftIron Security Remediation:

HyperCloud 1.4.3

Released June 5, 2023

Security and Bug Remediation:

  • Upgraded Linux kernel to remediate CVE-2023-32233.
  • Updated automatic filesystem detection algorithm for disks to be more robust, resolving an issue with ambivalent detection inside Ceph BlueStore devices inside bcache block devices.
  • Resolved an issue with importing marketplace appliance images from an S3-based marketplace.
  • Added bcache CLI tools, simplifying disk management and replacement.
  • Resolved issue where VM logs may not show up in the Web GUI.

HyperCloud 1.4.2

Released May 4, 2023

Security and Bug Remediation:

  • Remediated CVE-2023-0465 and CVE-2023-0466 in the HyperCloud integrated cryptographic libraries.
  • Updated bcache tools for managing SSD+HDD node disks to support a much more robust CLI, simplifying drive management to include replacements.
  • Updated IPMI utilities to support connecting to SoftIron node BMCs for Serial over LAN access from the Dashboard, if there is a Layer 3 route from the Dashboard to the BMC network.
  • Resolved an issue where bcache devices may not come back online after a reboot due to ambivalent detection of the underlying filesystem type.
  • Resolved an issue where the HyperCloud Remote Backups Marketplace cannot find replicated images if they are pushed to the non-default RBD image pool.

HyperCloud 1.4.1

Released April 3, 2023

Bug Remediation:

  • Resolved an issue introduced with scheduler changes in HyperCloud 1.4.0 that may cause a cluster deadlock on very busy clusters.

HyperCloud 1.4.0

Released March 30, 2023

New Features:

  • Added tools to assist with the replacement of drives on HD11XXX and HD21XXX storage nodes
  • Updated GUI to more align with SoftIron brand language. Updated verbiage in numerous locations to reflect HyperCloud branding
  • Added support for LLDP and resolved issue with LLDP and CDP PDUs being dropped on some NIC models
  • Updated to Ceph Pacific 16.2.11
  • Updated to set the default I/O scheduler to "none" on all OSD block devices
  • Added version information to build artefacts
  • Generate SHA256SUMS file for build artefacts
  • Added firmware support for SoftIron 100G NICs
  • Added support for job control on the console. SIGINT, SIGTSTP, and SIGABRT should now function as expected.

Manufacturing Improvements:

  • HyperCloud now builds manufacturing images as part of the CI pipeline. SoftIron Manufacturing may pull these files from the artefact directory for the release. Both ARM and x86 M.2 images are available. Check for hypercloud-<VERSION>-<ARCHITECTURE>-diskimage.tar.gz

Open Source Disclosure:

  • Source disclosure tarballs are now generated as part of the CI process. They may be pulled from the artefact directory for the release.

Security and Bug Remediation:

  • Remediated CVE-2022-4304, CVE-2023-0215, CVE-2023-0286, and CVE-2023-0464 in the HyperCloud cryptographic libraries
  • Resolved issue with enumerating replicated persistent disks in the "HyperCloud Remote Backups" Marketplace
  • Resolved issue with adding SoftIron marketplaces via the GUI
  • Improved log rotation to no longer fill root disk on very busy HyperCloud clusters
  • Resolved issue that may cause GUI log rotation to fail
  • Removed support for volatile disks from the GUI. This was an unsupported configuration for workload attached disks.
  • Resolved issue where the cluster may erroneously generate a new UUID on installation if the first static node is ever reinstalled after the cluster is built
  • Resolved race condition that may cause the Dashboard to fail to start, especially on clusters with large numbers of compute nodes
  • Removed support for multipathing from HyperCloud, as no SoftIron hardware requires it
  • Resolved issue with "hypercloud-reboot" that may cause the command to hang forever on some clusters based on method used to install the cluster
  • Disable Transparent Huge Pages on storage nodes to resolve memory consumption bug
  • Resolved bonding bug on SoftIron 10GBASE-KR NICs
  • Disabled erroneous, unsupported features on the Ceph dashboard
  • Disable Ceph OSD mirroring by default
  • Resolved issue with bursting workloads to AWS that may cause the deployment to timeout

Documentation:

SoftIron internal documentation is available at "https://docs.softiron.cloud/".

Marketplaces:

HyperCloud marketplaces are now available at "https://marketplace.softiron.cloud/". Three marketplaces exist: Official, Community, and Partner. These marketplaces are intended to be added to a HyperCloud instance with internet access. Details to add the marketplaces to HyperCloud can be found at: SoftIron Marketplaces.

HyperCloud 1.3.1

Released March 8, 2023

Hardware-enablement release to add support for the HD21216.

  • Added support for installation on HD21XXX Density Nodes
  • Resolved minor installation issues on HD11XXX Density Nodes

HyperCloud 1.3

Released December 5, 2022

Bug Remediation:

  • Updated ulimits for various Ceph daemons to remediate a bug with ceph-mgr crashing on very large clusters (>500 OSDs)

HyperCloud 1.2

Released December 5, 2022

Bug Remediation:

  • Corrected a syntax error in the multipath configuration file
  • Resolved an issue causing dashboard to not start on multiarch clusters after a clean install
  • Resolved an issue where GRUB config files may be generated incorrectly on install
  • Resolved an issue that may cause the installation to fail if the static nodes are ARM CPU architecture
  • Resolved an issue with the hotpatch filesystem not starting on ARM CPU architecture static nodes

HyperCloud 1.1

Released November 27, 2022

New Features:

  • Full support for bcache, adding support for our hybrid ARM SSD+HDD nodes (such as the HD11120)
  • Installation method from manufacturing and in the field (see repository README)

Bug Remediation:

  • Resolved issue with logrotate not working properly due to matching on wildcards as well as base.conf having erroneous permissions
  • Resolved potential dashboard boot race condition

HyperCloud 1.0

Released November 4, 2022

Initial Release of HyperCloud

SoftIron Security Remediation:

Release Cadence

HyperCloud updates are released in general every quarter, typically around the end of March, June, September, and December. Major, API impacting releases are generally released as the June release. The prior major non-API-impacting release is supported for three months after the last major release is announced. Only the latest release of a major release branch is supported.

Release notes are generated for each release. Typically these release notes are emailed directly to a contact defined by the customer. The release notes will include any configuration guidance to address vulnerabilities.

SoftIron has the ability to e-mail impacted customers of new releases of software if a critical vulnerability is found. In addition, SoftIron has applied to be listed as a vendor to self-report to the NVD. This has been pending action from NIST or MITRE since 2022 Q3.

HyperCloud has very few tuneables as far as security, namely whether to use full-disk encryption (which requires a physical HSM) or enabling FIPS mode. Using various tuneable settings, customers have used tools such as OpenVAS, Nessus, and BURP Suite to perform security vulnerability scanning of the HyperCloud released software with great success.

Additionally, SoftIron reserves the right to amend the aforementioned policy and release schedule as it deems fit.