Hardware Management, Automation, Provisioning & Monitoring for Neoclouds and Cloud Providers

Executive Summary

The data center hardware management landscape is undergoing its most significant architectural shift in two decades. The replacement of the aging Intelligent Platform Management Interface (IPMI) with DMTF Redfish, the rise of the Open Compute Project's (OCP) Hardware Management Project, and the explosion of GPU-dense neocloud infrastructure have collided to expose deep interoperability gaps, scaling ceilings, and operational blind spots that the industry is now racing to address. This report maps the full stack — from the physical BMC/IPMI layer through Redfish standards, OpenBMC, OCP contributions, bare metal provisioning frameworks, GPU-specific management, and the critical unresolved pain points facing operators of large-scale GPU clusters.

1. Foundation: BMC, IPMI, and the Out-of-Band Management Layer

1.1 What Is a BMC?

The Baseboard Management Controller (BMC) is a dedicated embedded processor — historically ARM- or SuperH-based — that provides out-of-band (OOB) server management independent of the host OS state. The BMC monitors and manages temperature, fan speeds, voltages, power supply, remote power control, virtual KVM, and system event logs. It communicates to the host via the LPC/eSPI bus and exposes network interfaces (dedicated or shared) for remote management. Critically, the BMC keeps running even when the host is powered off or unresponsive, making it the foundational layer for all infrastructure automation.[^1][^2]

1.2 IPMI: The Legacy Workhorse

IPMI (Intelligent Platform Management Interface) was introduced in 1998 and remains embedded in virtually all servers deployed today. IPMI 2.0 added encryption and LAN channel support. Key capabilities include power control (on/off/reset), sensor data reading (temperature, voltage, fan speeds), System Event Log (SEL) access, and Serial-over-LAN (SOL) for console access. Despite its ubiquity, IPMI carries structural limitations that make it unfit for modern hyperscale environments:[^3]

• Binary protocol: Not human-readable, not web-standard, and not easily parseable by modern tooling[^3]
• No extensibility: Fixed command set; adding new device types (GPUs, CXL, retimers) is not natively supported
• Security vulnerabilities: Multiple critical CVEs continue to emerge; in January 2025, NVIDIA's Offensive Security Research Team discovered stack overflow vulnerabilities in Supermicro BMC firmware authentication affecting X11/X12/X13/H12/H13/B12/B13 and newer platforms. Prior disclosures in April 2024 involved command injection attacks in the SMTP and SNMP configuration paths[^4][^5]
• No native event subscription model: SNMP traps were the only async notification mechanism
• Lack of ASLR: NVIDIA researchers found that at least one BMC server process loads at a consistent base address, lacking basic memory safety[^6]

Despite these limitations, IPMI is not disappearing from installed-base fleets anytime soon. The challenge for neoclouds and cloud providers is bridging IPMI-managed legacy nodes with Redfish-native modern infrastructure.

2. DMTF Redfish: The Modern Management Standard

2.1 Overview and Architecture

DMTF's Redfish, first introduced in August 2015, is a RESTful API standard using HTTPS and JSON (OData-compatible) designed to replace IPMI. It is developed by more than 60 member organizations including AMD, Alibaba, Cisco, Dell, Google, HPE, Huawei, IBM, Intel, Lenovo, and NVIDIA. Redfish 2025.1 was the latest release as of April 2026, adding significant improvements to scalability and telemetry.[^7][^8][^9]

Key Redfish architectural properties:

• Separation of protocol and data model: Schema is versioned independently, enabling evolution without breaking implementations
• OEM extensions: Vendors can inject proprietary namespaces (Oem property block) into any resource, enabling differentiation while maintaining base compliance
• Event Service: Subscription-based async event delivery (replacing SNMP traps)
• UpdateService: Standardized firmware update orchestration using ApplyTime and MaintenanceWindow semantics
• TelemetryService: Metric collection and reporting (currently being overhauled for streaming)

2.2 DMTF PMCI: The Inside-the-Box Layer

While Redfish handles north-bound management APIs, the DMTF Platform Management Communications Infrastructure (PMCI) Working Group defines the intra-platform communication protocols:[^10]

• MCTP (Management Component Transport Protocol): Transport-layer protocol for component communication over I2C, PCIe, USB, and other physical media. MCTP 2.0 (DSP0256) added major improvements to discoverability of MCTP communication from host software[^7]
• PLDM (Platform Level Data Model): Application layer on top of MCTP for firmware updates, FRU data, BIOS configuration, sensor monitoring, and — critically now — AI accelerator management[^11][^12]
• NC-SI (Network Controller Sideband Interface): BMC-to-NIC sideband communication for shared LOM scenarios

These protocols are increasingly important as GPU management (OAMs, PCIe retimers, NVLink switches) requires standardized in-platform communication between BMC and accelerator management controllers.

2.3 SPDM: Hardware Security and Attestation

The Security Protocol and Data Model (SPDM), standardized by the DMTF PMCI WG, enables authenticated, measured communication between the BMC and endpoint devices over MCTP. SPDM supports:[^13]

• Certificate-chain-based device authentication (slots 0–7)
• Firmware measurement and verification
• Establishment of encrypted communication sessions between management controller and managed components

HPE iLO 6 adopted SPDM for component authentication in its latest generation. Dell iDRAC10 and Cisco UCS M6+ also implement SPDM. This is increasingly critical for supply chain security, confidential compute attestation, and zero-trust infrastructure.[^14]

2.4 Redfish Interoperability Gaps

A fundamental but underappreciated problem: Redfish compliance does not guarantee interoperability. The Redfish specification marks only @odata.id, @odata.type, Id, and Name as "required" properties — all other properties, including FirmwareVersion, are optional. This means:[^15]

• Implementations that omit critical operational properties are still considered compliant
• OEM extension injection creates vendor-specific divergence even across nominally Redfish-compliant devices
• RackN, a bare metal automation vendor, documented in early 2026 that iDRAC10 introduced an undocumented behavior change in its Redfish API that broke production automation workflows[^16]
• Error responses differ across vendors in both code and payload structure[^16]

The DMTF introduced Redfish Interoperability Profiles (DSP0274) to address this: a profile defines the minimum required properties for a specific deployment context, allowing operators to validate compliance against a known baseline rather than the permissive base spec.

3. OpenBMC: The Open-Source BMC Platform

3.1 Architecture and Design

OpenBMC is a Linux Foundation collaborative project providing a fully open-source BMC firmware stack. It is built on the Yocto Project and delivers a complete Linux distribution specifically for BMC hardware. Core architectural components include:[^17][^1]

D-Bus serves as the central IPC mechanism, with the Object Mapper (xyz.openbmc_project.ObjectMapper) providing runtime service discovery instead of static configuration. This loose coupling allows modular services to be added or replaced independently.[^18]

OpenBMC's key feature list includes: hardware monitoring (temperature, fans, voltage), power sequencing, fan control, sensor telemetry, event logging, remote access (KVM, SOL), firmware/BIOS configuration, and out-of-band Redfish API.[^19]

3.2 Hyperscaler Adoption

OpenBMC is now the dominant platform management firmware for hyperscale deployments:

• Meta: Transitioned from their own proprietary "OpenBMC" codebase to the Linux Foundation version starting with the "Bletchley" chassis controller project in 2022[^20]
• Google: Active contributor to GPU management workstreams; presented GPU management requirements at OCP Hardware Management calls[^21]
• Dell: iDRAC10 is explicitly described as "built on OpenBMC principles" with DC-SCM hardware compliance[^22]
• Astera Labs / ASPEED / Insyde: Demonstrated COSMOS SDK integration into OpenBMC at OCP 2025, extending management to PCIe retimers and scale-up switches as first-class OpenBMC citizens[^23][^24]

4. OCP Standards: Hardware Management Ecosystem

4.1 OCP Hardware Management Project

The OCP Hardware Management Project is the most active industry body coordinating hardware management standards across GPU vendors, hyperscalers, and OEMs. Key active workstreams as of 2025–2026:

GPU Management Interfaces (GMI) Workstream

• Collaboration between Microsoft, AMD, NVIDIA, Google, and Meta
• Published OCP GPU UBB Redfish Interoperability Profile (RIP) v1.0 — standardizes the Redfish requirements between hyperscaler BMC and UBB (Universal Baseboard) accelerator management controllers[^25][^26]
• Published DMTF Message Registry for GPU-specific events enabling standardized alert semantics across vendors[^27]
• Active development of GPU streaming telemetry interface targeting transport-agnostic (Redfish, gRPC, MQTT) standardized data packets from accelerator management controllers to hyperscaler management controllers[^27]
• OCP GPU RAS 1.0 specification: Addresses GPU error containment, handling of synchronous distributed training interruptions at supercomputer scale, and aligns with the broader RAS API workstream[^25]

Hardware Fault Management (HFM) Sub-Project

• Goal: standardize hardware fault monitoring, reporting, and analysis in a platform/vendor-agnostic way[^28]
• Covers both out-of-band (BMC-mediated) and in-band (kernel/driver-mediated) fault management paths[^29]
• Integration with GPU RAS spec; contributors include AMD, ARM, Google, Intel, Meta, and Microsoft[^30]
• RAS API workstream developing standard APIs for error category reporting and recovery coordination between BMC, firmware, OS, and hyperscaler infrastructure[^31]

Impactless Firmware Update Workstream

• Developed a PLDM-based "Copy / Arm / Activate" framework for GPU firmware updates that minimizes disruption: the Copy and Arm phases do not interrupt GPU operation; only the Activate phase (a GPU reset) causes downtime[^32]
• 22 compliance tests defined for GPU firmware update validation[^32]
• Addresses firmware signing requirements, update time SLAs, and disruption definitions[^32]

Consolidated Boot and Management Interface (CBMI) Workstream (proposed Q1 2025)

• Addresses multi-interface, multi-protocol fragmentation in DCSCM deployments where a single module must support multiple CPU/accelerator vendors each requiring different management protocols[^33]
• Proposed by Intel and Microsoft; targeting vendor-neutral consolidated boot and management interfaces for DC-SCM

4.2 DC-MHS: Data Center Modular Hardware System

OCP DC-MHS is a modular server hardware specification developed by AMD, Ampere, Dell, Google, HPE, Intel, Meta, Microsoft, and NVIDIA that disaggregates traditional monolithic server motherboards:[^34][^35]

The key insight: by separating the BMC (via DC-SCM) from the CPU/memory substrate (HPM), organizations can independently evolve management firmware across generations without replacing the entire server board. v2.1 of the DC-SCM specification now includes 19 CLA member companies. Major ODMs including ASUS, Supermicro, QCT, and Dell have adopted DC-MHS architecture.[^36][^37][^38][^33]

Management implications: DC-MHS enables true multi-vendor composability — the same DC-SCM module can be used across different HPMs from different vendors, reducing management plane fragmentation.[^35]

4.3 OCP Global Summit Hardware Management Track (2024–2025)

The 2024 OCP Global Summit (San Jose, October 15–17) featured a dedicated half-day hardware management track covering:[^39][^40]

• GPU management workstream progress and UBB RIP v0.91 → v1.0 publication
• Hardware Fault Management Sub-Project status: out-of-band requirements complete; in-band methods in progress
• DCSCM CLA workstream proposal for consolidated boot/management interfaces
• DMTF MCTP/PLDM enhancements for accelerator management
• Standards-based GPU firmware update (Google, NVIDIA, Microsoft)
• RAS API design for standardized error category reporting

The 2025 OCP Global Summit (San Jose, October 13–16) DMTF sessions included:[^41]

• Redfish Message Registry for standardized event semantics and hyperscale filtering
• GPU streaming telemetry standardization panel (AMD, Google, Meta, Microsoft, NVIDIA)
• SPDM update and attestation for confidential compute
• PLDM/MCTP enhancements for advanced OCP use cases
• Rack management/monitoring based on OpenRMC-DM
• Ultra Ethernet management using OCP standards
• Turbocharging firmware deployment using Agentic AI

5. Vendor BMC/Management Platforms: Pros & Cons Analysis

5.1 Dell iDRAC (iDRAC10)

Dell's iDRAC10 is the latest generation integrated remote access controller, built on OpenBMC principles and designed for the DC-SCM architecture within OCP DC-MHS-compliant PowerEdge servers.[^22]

Key capabilities:

• Agent-free local and remote server management; full Redfish API with telemetry streaming
• Lifecycle Controller: integrated BIOS/firmware/OS deployment from within the BMC
• GPU management: inventory, per-GPU power, thermal, health, utilization — without OS agents[^42]
• Integration with Dell OpenManage Enterprise (OME) for centralized management of up to 25,000 devices[^43]
• Dell AI Factory integration: SmartFabric Manager automated blueprints, rack-scale integration with OME, Integrated Rack Controller (IRC) for leak detection and response[^43]
• Telemetry streaming to Splunk, Grafana, and third-party AIOps via Rsyslog and SNMP[^42]
• DC-SCM hardware compliance; multi-generational support commitment[^22]

5.2 HPE iLO 6

HPE's iLO 6 is the sixth generation of their Integrated Lights-Out management controller, shipping with HPE ProLiant Gen11 servers.[^14]

Key capabilities:

• HPE Silicon Root of Trust: Hardware-anchored firmware integrity securing millions of lines of firmware code across 4+ million HPE servers[^14]
• SPDM component authentication: Extended to partner ecosystem via HPE iLO 6[^14]
• PLDM firmware updates: Less-disruptive firmware update mechanism[^14]
• iLO Federation: Group-level power control, virtual media, firmware updates (requires iLO Advanced license)[^44]
• Group firmware update: Critical for fleet-scale security patching; only available with iLO Advanced[^44]
• Redfish-compliant REST API with certificate management for trusted boot[^14]

5.3 Supermicro IPMI / Redfish

Supermicro servers are widely deployed in neoclouds (including as H100/H200 OEM platforms) and use a combination of IPMI and Redfish starting from X10 (Intel) and H11 (AMD) platforms.[^45]

Key capabilities:

• Full IPMI 2.0 + Redfish on current-generation platforms
• IPMI-based remote KVM, power control, sensor monitoring
• Redfish support added incrementally; not uniformly feature-complete across all SKUs
• Lower ASP vs. Dell/HPE — cost-effective for commodity GPU deployments

5.4 Lenovo XClarity Controller (XCC)

The Lenovo XClarity Controller replaces the IMM2 (Integrated Management Module II) as the BMC for ThinkSystem servers. It is backed by the Pilot4 XE401 chip with dual-core ARM Cortex-A9.[^46]

Key capabilities:

• Redfish 1.15 support with REST API[^46]
• Three tiers: Standard, Advanced, Enterprise — all providing OOB remote access
• XClarity Administrator: multi-server discovery, inventory, firmware compliance, OS deployment, event management — agentless (no CPU/memory overhead on managed hosts)[^47]
• XClarity Integrator: connects to VMware vCenter, Microsoft Admin Center, Microsoft System Center[^48]
• XClarity Energy Manager: power and temperature management[^48]
• Warranty status monitoring; Call Home / Service Data Upload[^47]

5.5 Comparative Summary

6. Bare Metal Provisioning: Tools and Architecture

6.1 Use Cases in Neocloud and Cloud Provider Contexts

Bare metal provisioning for neoclouds differs fundamentally from traditional enterprise deployment:

• Time to first GPU: Operators need to go from physical delivery to production-capable GPU node in hours, not days
• Fleet homogeneity at scale: Thousands of identical nodes require zero-touch provisioning pipelines
• Heterogeneous hardware evolution: H100 → H200 → B200 transitions require flexible provisioning logic
• Continuous reprovisioning: Nodes are frequently re-imaged between tenants or workloads
• Firmware compliance gating: Nodes must pass firmware validation before joining the production pool

6.2 Tool Landscape

OpenStack Ironic Ironic is the de facto open-source standard for bare metal as a service. It is API-first, vendor-agnostic, and supports IPMI and Redfish as native driver backends. Key strengths:[^50]

• Standardized lifecycle from enrollment through retirement
• Multiple re-provisioning without manual intervention
• Officially supported by Dell, HPE, Supermicro with vendor-specific interface extensions maintained upstream[^50]
• Integrates with Metal3.io for Kubernetes-native deployments[^51]

Canonical MAAS MAAS (Metal as a Service) provides cloud-like bare metal management via a REST API, CLI, and GUI. It handles auto-discovery via PXE, IPMI-based OOB power management, OS deployment (Ubuntu, RHEL, CentOS, ESXi, Windows), and IPAM/DHCP/DNS. MAAS has added NVIDIA BlueField-3 DPU provisioning support, making it a natural fit for AI infrastructure deployments. Supports x86, ARM64 (Ampere Altra), POWER, and IBM Z architectures.[^52][^53][^54]

Tinkerbell (CNCF Sandbox) Tinkerbell takes a workflow-based approach: operators compose explicit, version-controlled workflows (inventory → wipe → write image → OS handoff). Kubernetes-native (kubectl-managed), with BMC power/boot management via Rufio (Redfish/IPMI). Originated at Equinix Metal and donated to CNCF. Well-suited to heterogeneous GPU fleets where provisioning logic must be explicit and auditable.[^55][^56]

Metal3.io (CNCF Incubating) Metal3.io became a CNCF incubating project in August 2025. It provides Kubernetes-native bare metal management via a Custom Resource Definition (CRD) model using BareMetalHost objects. Capabilities include:[^57][^51]

• Pre-installation BIOS/firmware configuration and RAID setup
• Firmware upgrades for provisioned hosts
• Cluster API (CAPM3) integration for Kubernetes-on-baremetal lifecycle management
• Backed by OpenStack Ironic for actual provisioning operations

RackN Digital Rebar A commercial platform providing Day 0, 1, and 2 automation with reusable Infrastructure-as-Code workflows. Notable for its zero-touch hardware onboarding from first-boot, multi-OEM support (IPMI, Redfish, vendor APIs), and drift detection. Positions itself as filling the gap between provisioning tools (which only handle Day 0) and configuration management tools (which assume an already-running OS).[^58][^59]

6.3 Provisioning Stack in Practice

The most capable neocloud and hyperscale operators typically compose multiple layers:[^55]

Orchestration Layer:    Kubernetes + Cluster API (CAPM3/CAPI)
Provisioning Layer:     MAAS | Ironic | Tinkerbell
Configuration Layer:    Ansible | Salt | Chef
IaC Layer:              Terraform | OpenTofu
OOB Control Layer:      Redfish / IPMI via BMC

7. Monitoring, Observability, and Orchestration

7.1 GPU-Specific Monitoring

Standard CPU monitoring tools miss approximately 85% of GPU-specific failure modes. A 1,000-GPU cluster produces roughly 500 GB of metrics daily, requiring specialized time-series infrastructure.[^60]

NVIDIA DCGM (Data Center GPU Manager) DCGM is the foundational GPU telemetry layer for production clusters. Key capabilities:[^61][^62]

• 100+ GPU metrics including utilization (SM, Tensor Cores, FP64), memory usage, ECC errors, PCIe bandwidth, NVLink traffic, clock throttling reasons, remapped rows[^60]
• 1-second granularity metric collection
• Active health monitoring and diagnostics (including hardware-level fault injection tests)
• Policy management: GPU compute mode, ECC settings, persistence mode
• Integration with Prometheus via dcgm-exporter for Kubernetes clusters[^61]
• DCGM 3.3+ adds NVIDIA Blackwell GPU support and enhanced MIG monitoring[^60]

NVIDIA Mission Control Mission Control 2.3 (latest) is NVIDIA's full-stack AI factory management platform targeting enterprise and neocloud operators. Capabilities include:[^63]

• Workload scheduling and orchestration (DGX SuperPOD scale)
• Autonomous hardware recovery: self-healing BMC, HGX, and Mellanox firmware validation with automated remediation[^64]
• Periodic health checks: BMC IPMI version, NVIDIA module loaded, OS version, GPU topology, GPU telemetry, power limits, InfoROM version, remapped row events[^65][^64]
• DGX Blackwell (GB200 NVL72, GB300 NVL72) full support in v2.3[^63]
• Air-gapped deployment option; virtualized control plane
• Leak detection validation checks

CoreWeave's Operational Approach CoreWeave treats the BMC as a primary reliability tool: every node is aggressively screened via BMC before customer delivery, and a custom node lifecycle management pipeline automates detection and remediation of hardware issues. This reflects a broader neocloud pattern: the operators who operationalize BMC automation most aggressively achieve the highest cluster reliability.[^66]

7.2 Predictive Maintenance and AIOps

ML-based predictive maintenance is now achieving 94–96% accuracy with 72-hour advance warning for hardware failures in GPU clusters. Ensemble models significantly outperform individual models (94% vs. 76%). AIOps platforms (Datadog, Dynatrace, New Relic) are integrating native GPU metrics alongside traditional infrastructure observability.[^60]

7.3 Observability Stack Architecture

8. Critical Gaps and Pain Points: Large GPU Clusters and Hyperscale Operations

8.1 IPMI Legacy Debt and Protocol Fragmentation

Installed base servers using IPMI cannot benefit from Redfish automation without either firmware updates (if the platform supports it) or protocol translation bridges. Many neocloud deployments — particularly those using older Supermicro platforms — face a fragmented management plane where some nodes speak Redfish and others speak only IPMI. This prevents unified automation and forces operators to maintain parallel toolchains.[^67]

8.2 Redfish Scale Ceiling: Event Storms and Polling Overhead

The DMTF's own presentation at OCP Global Summit 2025 explicitly identified the operational scale ceiling of current Redfish implementations:[^67]

• Event storms: During GPU firmware pushes across a cluster, all nodes simultaneously generate events — subscription management systems are overwhelmed
• Polling degradation: RESTful polling does not scale linearly; at thousands of nodes, latency and throughput bottlenecks emerge
• Managing 1,000s of subscriptions: Each node's Event Service subscription must be managed; lifecycle (creation, cleanup, re-subscription after BMC reset) adds significant operational overhead
• Legacy IPMI bridging: Many hyperscale environments still rely on IPMI-based tools; Redfish coexistence requires complex adapter layers

The solution path — federated Redfish service architecture, optimized Event Service with prefix-based subscription models, custom schema overlays for GPU hardware profiles — is defined but not yet broadly implemented.[^67]

8.3 Vendor OEM Extension Proliferation

Even across Redfish-compliant BMCs, vendor OEM extensions create a long-tail of integration work. Every ODM and CSP deals with a multitude of vendor-specific APIs when trying to unify device-level software features into their BMC management stack. This is the central challenge for multi-vendor neocloud deployments: a fleet combining Dell, Supermicro, and Gigabyte GPU servers requires three separate Redfish behavior profiles, different error code interpretations, and different OEM extension namespaces for GPU-specific operations.[^68]

8.4 GPU Firmware Update Complexity

GPU firmware updates at scale remain one of the most operationally painful procedures in neocloud management:[^69][^70]

• Updates require a specific ordering: BMC firmware first, then compute tray (HGX) firmware second, each followed by AC power cycles
• Failure at any step leaves the node in an inconsistent state, often requiring manual intervention
• Traditional GPU firmware updates require taking the GPU offline entirely; the PLDM-based Copy/Arm/Activate framework (developed jointly by Google, NVIDIA, and Microsoft) enables updating while the GPU is in use — with disruption limited to the reset phase only[^32]
• Coordinating firmware version compatibility across BMC, HGX baseboard, GPU device firmware, VBIOS, InfoROM, and InfiniBand HCA firmware creates a combinatorial validation matrix
• At hyperscale, even a 1% error rate in firmware updates means hundreds of nodes requiring manual remediation

8.5 GPU RAS: Error Containment and Distributed Training Fragility

For synchronous distributed training workloads (the dominant pattern for LLM pretraining), a single uncontained GPU error can interrupt an entire training job spanning thousands of GPUs. This is fundamentally different from traditional multi-tenant VM workloads where errors are isolated per-VM. The OCP GPU RAS 1.0 specification specifically addresses:[^25]

• Error containment boundaries for GPU errors in distributed training contexts
• Handling "error storms" where cascading errors spread across interconnected nodes
• Distinguishing correctable vs. uncorrectable errors with different recovery paths
• Coordination between BMC (out-of-band), OS RAS drivers (in-band), and hyperscaler management infrastructure

The industry has not yet standardized the precise boundary between BMC-handled recovery and software-stack-handled recovery, creating ambiguity in how different CSPs implement autonomous healing.[^29]

8.6 GPU Telemetry Gaps: Connectivity and Interconnect Visibility

Current monitoring stacks have deep visibility into GPU compute health (via DCGM) and server infrastructure (via BMC/Redfish), but a significant blind spot exists for the interconnect fabric between GPUs:[^68]

• PCIe retimers, NVLink switches, UALink switches, and CXL fabric devices are not first-class management objects in any current standard
• Link-level health (lane-level signal integrity, bit error rates, equalization settings) is not exposed via Redfish or DCGM
• Astera Labs' COSMOS SDK + OpenBMC integration (demonstrated at OCP 2025) is an early attempt to fill this gap for PCIe retimers and scale-up switches, but is not yet standardized[^24]

8.7 Lack of Streaming Telemetry Standard

The existing Redfish Telemetry Service (TelemetryService, MetricReportDefinitions) does not produce interoperable reports — each vendor's output requires custom parsing. The DMTF is developing a new streaming telemetry proposal featuring:[^71]

• TLV (Type-Length-Value) per-metric records bundled for delivery
• TelemetryFeed definitions for configuring stream contents without a priori knowledge of the resource tree ("blind deploy" across a fleet)[^72]
• Metric name/label format standardization for integration with time-series databases (Prometheus, InfluxDB, OpenTSDB)

Until this standard is finalized and implemented by vendors, operators must maintain custom metric parsers per vendor.

8.8 Multi-Vendor Rack-Scale Management Fragmentation

As AI infrastructure shifts from server-centric to rack-scale architectures (NVL72, GB200 "Mega-racks"), the management domain expands beyond individual servers to include rack-level power distribution, liquid cooling loops, NVLink domain management, and cross-server NVLink switching fabrics. No single management standard currently covers all of these domains coherently:[^68]

• Server BMC: Handles individual node management (Redfish/OpenBMC)
• Rack-level PDU: Typically proprietary APIs or basic SNMP
• Cooling system: Vendor-proprietary management (Schneider, Vertiv, Rittal)
• NVLink/InfiniBand fabric: NVIDIA UFM (Unified Fabric Manager) — proprietary
• CXL fabric: Emerging management standards, not yet mature

The OCP DC-MHS + DC-SCM + CBMI workstreams are attempting to address some of this, but the path to a unified rack-scale management API remains years away from standardization.

8.9 GPU Cluster Utilization: The Hidden Operational Cost

Average GPU cluster utilization across neoclouds hovers around 40–50%. NVIDIA's internal experience operating large-scale GPU clusters identified three key operational challenges: researcher productivity, resource utilization, and operational efficiency. A multi-level fair-sharing approach at NVIDIA achieves ~95% occupancy by operating clusters as large multi-tenant shared resource pools — but this requires sophisticated scheduling and fault isolation that most neocloud operators have not yet built.[^73][^74]

8.10 Security: BMC as Attack Surface

The BMC's privileged position — persistent access to all server hardware, independent of host OS state — makes it a high-value attack target. Key concerns for neocloud operators:

• Supermicro CVEs disclosed in 2024–2025 affect authentication design (not just implementation), meaning the root cause is architectural[^4]
• BMC firmware authentication bypass enables persistent malware that survives OS reinstall and even BMC software updates in some scenarios[^4]
• Lack of ASLR in at least some BMC server processes enables predictable exploit primitives[^6]
• SPDM, Silicon Root of Trust (HPE), and Caliptra (OCP's open-source RoT project) are the emerging mitigations, but adoption is uneven across vendors

8.11 Neocloud-Specific Operational Realities

Neocloud operators face unique management challenges not encountered at the same scale in traditional enterprise IT:[^75]

• Multiple disruptive interruptions per day at large GPU cluster scale — 99.999% availability (five-nines) is not achievable with current hardware reliability; four-nines is the practical ceiling[^75]
• Deep diagnostics deficit: When jobs slow or fail, isolating the cause requires actionable telemetry at GPU, NIC, switch, and BMC levels simultaneously — most operators lack this correlation layer[^75]
• Bare-metal delivery velocity: GPU rack deliver-to-production timelines directly impact ROI; operators who automate provisioning pipelines most aggressively win on unit economics[^66]
• Firmware compliance gating: Neoclouds running production SLAs must validate all firmware versions before a node joins the fleet — manual validation processes don't scale to thousands of nodes

9. Emerging Solutions and Industry Trajectory

9.1 Convergence on OpenBMC + Redfish + DC-MHS

The industry is converging on a three-layer open foundation: OpenBMC as the BMC OS, Redfish as the northbound API, and DC-MHS/DC-SCM as the hardware modularity layer. This combination enables true multi-vendor interoperability in principle — though vendor OEM extensions continue to fragment practical deployments.

9.2 GPU-Specific Redfish Profiles

The OCP GPU Management Interfaces Working Group's Redfish Interoperability Profile (RIP) for GPU UBBs, published in 2024, represents the first industry-wide attempt to define minimum required Redfish capabilities for GPU management contexts. As this profile gains compliance testing infrastructure, it creates a new baseline above which GPU platform vendors must implement — reducing the long-tail integration work for CSPs.[^26][^25]

9.3 Agentic AI for Firmware Management

The 2025 OCP Global Summit featured a presentation on "Turbocharging Firmware Development and Deployment using Agentic AI," signaling that AI-driven automation of BMC/BIOS firmware lifecycle management is moving from concept to implementation. This aligns with the broader industry trend toward autonomous hardware recovery (NVIDIA Mission Control) and predictive maintenance.[^41]

9.4 Rack-Scale Management as a First-Class Domain

The transition from server-centric to rack-scale AI architectures (NVL72, GB200 NVLink domains) is forcing a rethink of management scope. Astera Labs, ASPEED, and Insyde Software demonstrating COSMOS integration into OpenBMC for PCIe retimers and scale-up switches at OCP 2025 illustrates how the management domain is expanding. The emerging model treats the entire rack as a single managed entity — "Rack as a Computer."[^23][^24]

9.5 Confidential Compute Management Complexity

As confidential computing (Intel TDX, NVIDIA confidential modes) matures, hardware attestation workflows (via SPDM) become part of the management plane. The 2025 OCP Global Summit specifically covered "Orchestrating Confidential Compute using OCP Secure Boot, Attestation and CXL IDE, TSP, DMTF SPDM Specs", reflecting the convergence of security attestation and hardware management.[^41]

10. Conclusion: Prioritized Gaps for Neocloud Operators

Based on the complete assessment, the following gaps represent the highest-priority unsolved problems for operators building and scaling GPU-dense neocloud infrastructure:

References

1. openbmc/docs | DeepWiki - This document provides a high-level introduction to OpenBMC, covering its fundamental purpose, core ...

2. The Bmc Becomes Open Source

3. BMC and IPMI Management Interface Limitations - LinkedIn - How do humans actually talk to BMC? Last post continuation. We cannot talk about BMC without talking...

4. Vulnerability in Supermicro BMC IPMI Firmware, January 2025 - There is a vulnerability in the BMC firmware image authentication design. An attacker can modify the...

5. Vulnerabilities in Supermicro BMC Firmware, April 2024 - Three security issues have been discovered in select Supermicro motherboards. These issues affect th...

6. Analyzing Baseboard Management Controllers to Secure Data ... - Modern data centers depend on Baseboard Management Controllers (BMCs) for remote management. These e...

7. Home

8. Why is Redfish different from other REST APIs - Part 1 - Redfish is a standard RESTful API designed to deliver simple and secure management for converged, hy...

9. Formal Analysis of SPDM: Security Protocol and Data Model version ... - ... servers and storage. It is currently standardizing a security protocol called SPDM, which aims t...

10. PMCI - Platform Management Communications Infrastructure - DMTF - The Platform Management Communications Infrastructure (PMCI) Working Group defines standards to addr...

11. Platform Management Communication Infrastructure Enhancements ... - ... OCP has been leveraging DMTF MCTP, PLDM, and NC-SI specifications for platform management commun...

12. [PDF] MCTP and PLDM Enhancements for Advanced OCP Use Cases - • Creates specifications for MCTP, PLDM, and NC-SI. • Applicability to OCP: platform components comm...

13. Cisco UCS Manager System Monitoring Guide Using the CLI ... - The SPDM enables access to low-level security capabilities and operations by specifying a managed le...

14. What's new with HPE iLO 6 | Chalk Talk - YouTube - HPE Integrated Lights Out (iLO) 6 enables customers to securely configure, monitor, and update your ...

15. Introduction to Redfish interoperability profiles | HPE Developer Portal - The HPE Developer portal

16. Redfish API Woes and the Stress of Bare Metal Management - RackN - Did you run into this unofficial Redfish spec change in iDRAC 10? Find out what's changed and how ou...

17. OpenBMC

18. Architecture Overview | openbmc/docs | DeepWiki - This document provides a high-level overview of the OpenBMC architecture, focusing on core system co...

19. What are OpenBMC and UEFI? - Arm Learning Pathslearn.arm.com › learning-paths › openbmc-rdv3 › 1_introduction_openbmc - This advanced topic is for firmware developers, platform software engineers, and system integrators ...

20. Meta's OpenBMC process: a case study from the Bletchley system - OSFC - Change the way of firmware development, collaborate with others and share knowledge.

21. OCP Hardware Management Project call (Feb 20, 2024) - YouTube - Share your videos with friends, family, and the world.

22. Support for Integrated Dell Remote Access Controller 10 (iDRAC10) - Enhanced layout and intuitive workflows allow users to find and manage system settings more quickly,...

23. Insyde® Software Collaboration with Astera Labs' Open Rack ... - We have integrated key features from our COSMOS SDK into OpenBMC, so hyperscalers can adopt standard...

24. COSMOS and OpenBMC Demo at OCP 2025 - ASTERA LABS, INC. - Through collaboration with ASPEED Technology and Insyde Software, Astera Labs has extended COSMOS fe...

25. Standardizing Hyperscaler Requirements for Accelerators - YouTube - ... OCP work has improved GPU management over the last 12 months ... requirements- make the implemen...

26. GPU Profiles for Hyperscale Use Cases - YouTube - Hari Ramachandran,Siva Sathappan,Linda Wu - Microsoft,AMD,NVIDIA - The OCP GPU Management Interfaces...

27. Panel Standardizing GPU Management Redfish, Telemetry, and ... - ... GPUs to power AI and high-performance computing workloads reinforces the need for standardized, ...

28. OCP Hardware Management Project call (Aug 19, 2025)

29. Hardware Fault Management Sub-Project call (Jul 12, 2024) - ... t available. OCP HM - Hardware Fault Management Sub-Project call (Jul 12, 2024). 41 views · 1 ye...

30. OCP Hardware Management Project call (Sep 16, 2025)

31. Hardware Fault Management Sub-Project call (May 17, 2024) - ... t available. OCP HM - Hardware Fault Management Sub-Project call (May 17, 2024). 15 views · 1 ye...

32. A Standards-based approach to Firmware Update of GPUs at ... - Presented by Sujoy Sen (Google) | Vishal Jain (Nvidia) | Bhushan Mehendale (Microsoft) With the adve...

33. OCP Hardware Management Project call (Dec 17, 2024) - YouTube - This content isn't available. OCP Hardware Management Project call (Dec 17, 2024). 36 views · 10 mon...

34. Data Center Modular Hardware System Specification (DC-MHS) the ... - ... Data Center Modular Hardware System specifications have been a smash in the OCP community with g...

35. [PDF] An-Evaluation-of-the-Open-Compute-Modular-Hardware ... - This project focuses on delivering a modular server hardware specification that enables hardware ven...

36. What is DC-MHS? - ASUS Servers - The Data Center Modular Hardware System, otherwise known as DC-MHS or sometimes Server/MHS, is a sta...

37. What Is a Data Center Modular Hardware System (DC-MHS)? - Data center modular hardware system (DC-MHS) refers to a highly flexible and scalable approach to bu...

38. QCT Embraces the Future with the Adoption of DC-MHS - By using standardized hardware modules, data centers can easily scale their infrastructure, reduce c...

39. 2024 OCP Global Summit - Hardware Management - YouTube - An overview of workstreams of hardware management project. Open Compute Project. 21:56 Applying Mode...

40. DMTF Presentations from the OCP Global Summit Now Available - DMTF recently participated in the Open Compute Project (OCP) Global Summit, October 15-17, which hos...

41. DMTF Represented at the 2025 OCP Global Summit - The Open Compute Project (OCP) Global Summit is right around the corner, October 13-16, 2025, at the...

42. iDRAC10 Powering Smarter GPU Management in PowerEdge AI ... - Dell's Integrated Remote Access Controller (iDRAC) automates server lifecycle management and enables...

43. Dell Technologies Accelerates Enterprise AI with Powerful ... - SmartFabric Manager now includes Dell AI factory integration, simplifying AI infrastructure deployme...

44. HPE iLO 6 Essentials to Advanced Upgrade What Changes in the UI - We upgraded a server from HPE iLO 6 Essentials to Advanced to see what newly enabled features appear...

45. Redfish® API | Supermicro Server Management Utilities - Redfish is an open industry standard specification and schema developed by DMTF (SPMF) group that sp...

46. Introduction | Lenovo XClarity Controller | Lenovo Docs - The Lenovo XClarity Controller (XCC) is the next generation management controller that replaces the ...

47. Lenovo XClarity Administrator Product Guide - There are various management tasks for each supported endpoint, including viewing status and propert...

48. Management offerings | ThinkSystem SR650 | Lenovo Docs - Lenovo XClarity Controller. Baseboard management controller (BMC). Consolidates the service processo...

49. Management options | ThinkSystem SR665 | Lenovo Docs - Overview

50. Ironic Bare Metal: Home - Ironic is an open source project that fully manages bare metal infrastructure. It discovers bare-met...

51. Metal3.io becomes a CNCF incubating project - Cluster API provides infrastructure-agnostic Kubernetes lifecycle management, and Metal3.io brings t...

52. What MAAS can do - Docs - Canonical MAAS | Discourse - In essence, MAAS lets you treat a data center of bare-metal machines as if it were a cloud, enabling...

53. Data Centre AI evolution: combining MAAS and NVIDIA smart NICs - Among them, Canonical's metal-as-a-service (MAAS) enables the management and control of smart NICs o...

54. MAAS | Bare metal automation for your data center - Canonical - Key features · Automated hardware lifecycle management · Infrastructure-as-code automation · Integra...

55. The best tools for bare metal automation that people actually use - Provisioning is handled by MAAS or Ironic or Tinkerbell. Kubernetes-native environments add the Bare...

56. Tinkerbell | CNCF - Bare metal provisioning engine, supporting network and ISO booting, BMC interactions, metadata servi...

57. Metal Kubed: Metal³ - Metal3.io aims to build on baremetal host provisioning technologies to provide a Kubernetes native A...

58. RackN Digital Rebar - Digital Rebar scales IaC through reusable provisioning, configuring, and orchestration automation ac...

59. RackN: Home - The RackN Digital Rebar platform scales IaC automation with reusable workflows that can be deployed ...

60. GPU Cluster Monitoring: Real-Time Performance Analytics and ... - NVIDIA DCGM 3.3+ adding Blackwell GPU support and enhanced MIG monitoring. AIOps platforms (Datadog,...

61. Monitoring GPUs in Kubernetes with DCGM | NVIDIA Technical Blog - Monitoring GPUs is critical for infrastructure or site reliability engineering (SRE) teams who manag...

62. NVIDIA DCGM - Manage and Monitor GPUs in Cluster Environments

63. Run Models for AI Factories |Mission Control - NVIDIA - Mission Control powers every aspect of AI factory operations, from developer workloads to infrastruc...

64. NVIDIA Mission Control autonomous hardware recovery's ... - Updates BMC firmware to the required version. Updates HGX firmware to the required version. Updates ...

65. NVIDIA Mission Control autonomous hardware recovery's ... - This comprehensive testing framework can be flexibly executed across various scales of infrastructur...

66. The Future of AI Clusters for Enterprise in 2025 - CoreWeave - Inside CoreWeave's approach to building next-gen AI infrastructure using NVIDIA H100, Blackwell, DPU...

67. [PDF] Enabling Data Center Management with DMTF Redfish - Friction in deploying and operating. Redfish at scale. Adopting Redfish at hyperscale is more than j...

68. Building the Software Stack for AI Infrastructure 2.0 - Building the Software Stack for AI Infrastructure 2.0: Why Standards-Based Connectivity Management M...

69. Deployment Summary Validation Checklist - NVIDIA Documentation - Failure here typically means the BMC update was not successful or the compute tray firmware bundle d...

70. [PDF] Research on Server Performance Stability Assurance Mechanisms ... - Data sources include: Firmware change order records from 3 US data centers in 2024,. BMC/BIOS upgrad...

71. Streaming Telemetry Proposal Update - YouTube - Jeff Autor Principal Software Architect - Vertiv This session will provide an update on the streamin...

72. [PDF] Redfish Telemetry Streaming and Reporting - DMTF - Are there existing formats to which we can align? • Do we need to support multiple formats / styles?...

73. Experience Operating Large GPU Clusters at Organizational Scale - Speakers: Vikas Mehta, Bugra Gedik, Mohamed Fawzy, & Vipin Sirohi from NVIDIA We outline Nvidia's ex...

74. CoreWeave vs Lambda GPU Cloud: 2025 Comparison Guide - Deep dive into CoreWeave vs Lambda GPU Cloud. Compare InfiniBand networking, Kubernetes orchestratio...

75. Weighing up the enterprise risks of neocloud providers - One of the most notable cloud technology trends in 2025 was the (seemingly) overnight emergence of t...