Oracle Exadata Hardware Components — What Every DBA Must Know
In Article 1 we established what Exadata is and why it exists. Before you can understand how Exadata performs the way it does, you need a clear picture of the physical components inside the rack. Every feature — Smart Scan, RDMA, XRMEM, HCC — is delivered by specific hardware components working together. Without knowing what the hardware does, the software features are just marketing terms.
This article covers every physical component in an Exadata X10M rack — what it is, what it does, and what the current generation specs look like. By the end you will be able to look at an Exadata rack, name every component, and explain what each one does in a technical conversation.
All specifications in this article refer to the Exadata X10M generation unless otherwise stated. X10M is the most widely deployed current-generation system. Where X11M specs differ significantly they are noted. Specs for specific rack configurations — full, half, quarter, eighth — are covered in the configuration section.
What Is Inside an Exadata Rack
An Exadata rack is a standard 42U data center rack that Oracle ships pre-cabled and pre-configured. Everything inside it is an Oracle-tested and Oracle-certified component. You do not build an Exadata rack from parts — it arrives as a complete engineered system ready for network connection and software installation.
A full Exadata X10M rack contains six types of physical components:
Component 1 — Database Servers (Database Nodes)
The database servers are where Oracle Database and Oracle Grid Infrastructure run. From a DBA perspective, these are the servers you interact with most — you log in here to start and stop databases, run SQL, check alert logs, and perform ADOP patching. They are functionally equivalent to a standard Oracle database server, but with Exadata-specific capabilities layered on top.
What the database server does
- Runs Oracle Database — parses SQL, generates execution plans, manages transactions
- Runs Oracle Grid Infrastructure — manages RAC clustering, ASM, and the Oracle Clusterware stack
- Communicates with storage cells over the RoCE network using the iDB (Intelligent Database) protocol to request data and issue Smart Scan operations
- Hosts the Oracle database buffer cache — frequently accessed blocks are cached here to avoid repeated storage reads
- Runs Oracle Linux — the operating system on all Exadata database servers
Exadata X10M Database Server Specifications
| Component | X10M Specification | Previous Gen (X9M) for Reference |
|---|---|---|
| CPU | 2 x AMD EPYC 4th Gen — 96 cores each (192 cores total per server) | 2 x Intel Xeon — 32 cores each (64 cores total) |
| Base memory | 512 GB DDR5-4800 MT/s | 384 GB DDR4-3200 MT/s |
| Max memory | 3 TB DDR5 (expandable in factory — 512 GB, 1.5 TB, 2.25 TB, 3 TB options) | 2 TB DDR4 |
| Local storage | 2 x 3.84 TB NVMe SSD (expandable to 4 x 3.84 TB) | 2 x 3.2 TB NVMe SSD |
| Form factor | 2U — moved from 1U in X9M to accommodate larger AMD CPUs and more memory | 1U |
| RoCE network | Dual-port PCIe Gen 5 — 2 x 100 Gb/sec active-active = 200 Gb/sec total | 2 x 100 Gb/sec RoCE |
| Operating system | Oracle Linux with KVM virtualization support | Oracle Linux with KVM |
| Remote management | Oracle ILOM (Integrated Lights Out Manager) — out-of-band management | Oracle ILOM |
The jump from 64 cores (X9M) to 192 cores (X10M) per database server is not a typo. It is a 3x increase in compute capacity — driven by the move to AMD EPYC 4th generation processors. This dramatically increases how many parallel query processes and concurrent users a single database server can handle.
Full rack database server count
A full Exadata X10M rack ships with 2 database servers as standard. It can be configured with up to 8 database servers within a single rack. The number of database servers you need depends on your OLTP and parallel query workload — storage cells are always more numerous than database servers in a well-configured system.
Capacity on Demand (CoD)
Exadata database servers support Capacity on Demand — a feature that allows customers to purchase a server with a fixed number of active CPU cores initially and enable additional cores as the business grows. Each X10M database server must have a minimum of 14 cores enabled at installation. This allows organisations to right-size their Oracle Database licence costs from day one and grow incrementally.
Component 2 — Storage Cells (Cell Servers)
Storage cells are the component that makes Exadata unique. Unlike a standard SAN or NAS, an Exadata storage cell is a real server — it has CPUs, memory, an operating system (Oracle Linux), and runs Oracle's Exadata System Software. This software intelligence is what enables Smart Scan, Storage Indexes, and I/O offloading.
Every storage cell runs a process called cellsrv — the Cell Server daemon. This is the core software that receives I/O requests from database servers, decides whether to perform a Smart Scan or a conventional I/O, filters data based on SQL predicates, and returns the result. The DBA interacts with storage cells through a command-line interface called cellcli, which is covered in full in Article 7.
Three types of storage cells in X10M
| Cell Type | Abbreviation | Storage Media | Best For |
|---|---|---|---|
| High Capacity | HC | 12 x 22 TB NVMe-connected hard disks + 4 x 6.8 TB NVMe Flash for cache | Large data volumes where cost per TB matters — data warehouses, archive |
| Extreme Flash | EF | 4 x 6.8 TB performance flash + 4 x 30.72 TB capacity flash = 122.88 TB raw flash | Highest performance workloads — OLTP, financial trading, IOT |
| Extended | XT | 12 x 22 TB hard disks — no high-performance flash cache | Online archive, dev/test, less frequently accessed data |
Exadata X10M Storage Cell Specifications (HC and EF)
| Component | HC Storage Cell | EF Storage Cell |
|---|---|---|
| CPU | AMD EPYC 4th Gen — 64 cores (up from 32 in X9M) | AMD EPYC 4th Gen — 64 cores |
| DRAM (cell memory) | 256 GB DDR5-4800 MT/s | 256 GB DDR5-4800 MT/s |
| XRMEM | 1.25 TB DDR5 — Exadata RDMA Memory for ultra-low latency access | 1.25 TB DDR5 XRMEM |
| Flash (performance) | 4 x 6.8 TB F680 NVMe PCIe Gen4 — used for Smart Flash Cache and Smart Flash Log | 4 x 6.8 TB F680 NVMe PCIe Gen4 |
| Flash (capacity) | None | 4 x 30.72 TB NVMe PCIe Gen4 — data storage on all-flash |
| Hard disks | 12 x 22 TB — 264 TB raw per cell (22% increase over X9M's 18 TB disks) | None — all-flash configuration |
| Read IOPS per cell | 2.8 million 8K read IOPS (21% increase over X9M) | Higher than HC — optimised for flash |
| XRMEM read latency | 17 microseconds (down from 19 µs in X9M) | 17 microseconds |
| Operating system | Oracle Linux | Oracle Linux |
| Management interface | cellcli — local command line on each cell | cellcli |
Smart Flash Cache and Smart Flash Log
The performance flash drives on HC and EF storage cells serve two purposes beyond data storage:
- Smart Flash Cache — Exadata uses the performance flash as a large, intelligent read cache. Recently and frequently accessed data blocks are cached in flash, providing sub-millisecond access without going to spinning disk. The cache is database-aware — it caches Oracle database blocks, not raw I/O blocks.
- Smart Flash Log — Redo log writes are mirrored to flash simultaneously with disk writes. Redo log I/O is on the critical path of every database transaction. By writing redo to flash first, Exadata reduces redo write latency dramatically — improving OLTP transaction commit speed.
Component 3 — RoCE Network Fabric
The RoCE (RDMA over Converged Ethernet) network fabric is the high-speed internal network that connects every database server to every storage cell inside the Exadata rack. It is one of the most important components in the system — without a fast, low-latency internal network, Smart Scan and RDMA cannot deliver their performance benefits.
What RDMA means in practice
RDMA stands for Remote Direct Memory Access. It allows one server to directly read or write the memory of another server without involving the CPU or operating system of either server. The network card handles the transfer directly — there is no extra copying, no kernel context switches, and no OS overhead.
For Exadata this means a database server can read data directly from a storage cell's XRMEM (storage-side DRAM) at 17 microsecond latency — bypassing the entire OS I/O stack on both ends. This is what makes XRMEM practical as an acceleration tier.
X10M RoCE network specifications
| Specification | Value |
|---|---|
| Protocol | RoCE — RDMA over Converged Ethernet |
| Network card | Dual-port PCIe Gen 5 NIC on every DB node and storage cell |
| Speed per port | 100 Gb/sec per port |
| Total throughput per server | 200 Gb/sec (2 ports active-active) |
| Switches | 2 x RoCE switches per rack — fully redundant, no single point of failure |
| Topology | All DB nodes and storage cells connected to both switches — full mesh at the switch level |
| RDMA latency (XRMEM) | 17 microseconds end-to-end including storage software |
| RAC cluster interconnect | RoCE fabric doubles as the RAC private interconnect — no separate interconnect hardware needed |
On older Exadata generations (X7 and earlier), the internal network was InfiniBand — a separate high-speed network technology. Starting with X8M, Oracle switched from InfiniBand to RoCE. RoCE uses standard Ethernet hardware and switches while still supporting RDMA — making it cheaper to operate and easier to extend across racks. If you see references to InfiniBand in older Exadata documentation, that refers to pre-X8M systems. X10M and X11M use RoCE exclusively.
How the network carries multiple traffic types
The RoCE fabric inside Exadata carries three distinct types of network traffic simultaneously:
- Storage I/O — data reads and writes between DB nodes and storage cells, including Smart Scan results
- RAC private interconnect — Oracle RAC cache fusion traffic between database nodes for block transfers and lock management
- RDMA access to XRMEM — direct memory reads from storage cell DRAM via RDMA, bypassing cellsrv entirely for certain access patterns
All three traffic types share the same physical network infrastructure but are managed separately through network quality of service policies that Oracle pre-configures in the Exadata System Software.
Component 4 — Management Network Switch
Every Exadata rack includes a dedicated 1 GbE management switch — separate from the high-speed RoCE fabric. This switch connects to the ILOM (Integrated Lights Out Manager) port on every server in the rack, providing out-of-band management access.
Out-of-band means you can access a server's console, power it on or off, and check hardware health even when the operating system is down or the server is unresponsive. This is essential for Exadata administration — if a database node fails, you still need to manage it remotely without physical access to the data center.
# SSH to the ILOM interface of a DB node (out-of-band) ssh root@dbnode01-ilom # Check server power state show /System power_state # View server fault indicators show /System component_state # Check storage cell ILOM (same approach) ssh root@cell01-ilom
Component 5 — Client Access Network Switch
The client access network switch provides the connection between the Exadata database servers and your application tier — the servers running Oracle EBS, custom applications, or any other client that connects to the Oracle database.
Exadata X10M includes a 10 GbE client access switch inside the rack. For most enterprise environments this is the network that your application servers, middle-tier servers, and end-user connections use to reach the database. The client network is completely separate from the internal RoCE storage fabric — client traffic never touches the high-speed internal network.
For high-bandwidth environments, the client network switch in the rack can be supplemented with external 25 GbE or higher switches connected to the database servers' additional network ports. The database servers in X10M support up to 5 network interface cards — providing flexibility for high-throughput client connectivity alongside the RoCE fabric.
Component 6 — Power Distribution Units (PDUs)
Every Exadata rack includes two redundant Power Distribution Units — one on each side of the rack. The PDUs distribute power from the facility's electrical supply to every component in the rack.
The dual-PDU design provides full power redundancy. Each PDU connects to a separate facility power circuit. If one circuit or PDU fails, all components continue running from the other. Every server in the rack has dual power supplies — one connected to each PDU — so there is no single point of failure in the power path from facility breaker to server.
PDU specifications for X10M
- 2 x redundant PDUs — one A-side, one B-side
- Available in single-phase or three-phase configurations depending on facility power
- High voltage and low voltage options available — confirm with your data center team before ordering
- PDUs include power monitoring — Oracle Enterprise Manager and ILOM can report per-outlet power consumption
- Total rack power consumption varies by configuration — a fully populated X10M full rack requires planning for approximately 15 to 25 kVA depending on component count
Rack Configurations — Full, Half, Quarter, Eighth
Exadata is available in four rack size configurations. The same full rack hardware is used for all — smaller configurations simply have fewer servers installed, with the option to add more as requirements grow.
| Configuration | Database Servers | Storage Cells (HC) | Typical Use |
|---|---|---|---|
| Full Rack | 8 DB nodes | 14 to 19 cells | Large production environments, database consolidation |
| Half Rack | 4 DB nodes | 7 to 9 cells | Medium production, UAT environments at scale |
| Quarter Rack | 2 DB nodes | 3 cells | Smaller production, development, proof of concept |
| Eighth Rack | 2 DB nodes | 3 cells | Entry-level — smallest supported configuration |
X10M also offers an Elastic Configuration option — a custom number of DB nodes and storage cells anywhere between Eighth Rack and Full Rack. This allows right-sizing to your specific workload without being locked into the standard quarter/half/full rack increments. Oracle Exadata Configuration Assistant (OECA) is used to specify and validate elastic configurations.
Scaling Beyond One Rack
When a single rack is not enough, multiple Exadata racks connect to each other using the integrated RoCE network fabric via short-range RoCE cables and additional switches. The result is a single logical Exadata system that is simply larger.
A system composed of four full racks of Exadata X10M is four times as powerful as one rack — it provides four times the I/O throughput, four times the storage capacity, and four times the processing power. RAC automatically uses all database servers across all racks. ASM automatically uses all storage cells across all racks as a single storage pool.
The maximum supported configuration is substantial: a single rack of Exadata Database Machine X10M with nine database servers and nine High Capacity storage servers, along with 13 fully populated Exadata Storage Expansion X10M racks, delivers a raw disk capacity of 64 Petabytes and 15,552 CPU cores dedicated to SQL processing.
Key Software Running on the Hardware
Understanding the hardware is only half the picture. Each component runs specific Oracle software that ties the system together.
| Hardware Component | Key Software Running |
|---|---|
| Database servers | Oracle Linux, Oracle Grid Infrastructure, Oracle Database, Oracle RAC |
| Storage cells | Oracle Linux, Exadata System Software (cellsrv, MS, RS daemons) |
| All servers | Oracle ILOM — remote management firmware on every server |
| Management switch | Standard switch firmware — no Oracle-specific software |
| RoCE switches | RoCE switch firmware — pre-configured by Oracle during manufacturing |
The three daemons that run on every storage cell are worth knowing by name. cellsrv handles all storage I/O requests including Smart Scan. MS (Management Server) handles cellcli commands and monitoring. RS (Restart Server) monitors cellsrv and MS and automatically restarts them if they fail. We cover these in detail in Article 7 on Exadata administration.
How to Identify Your Components
When you first connect to an Exadata system, these commands help you understand what hardware you are working with.
# Check Exadata model and generation imageinfo | grep -i "machine\|version\|kernel" # List all cluster nodes (DB nodes in the RAC cluster) olsnodes -n # Check CPU and memory on this DB node lscpu | grep -E "Socket|Core|Thread" grep MemTotal /proc/meminfo # Check Exadata System Software version imageinfo -ver
# dcli runs a command on all storage cells simultaneously # cell_group file lists all cell hostnames — one per line dcli -g /opt/oracle.SupportTools/onecommand/cell_group \ "cellcli -e LIST CELL ATTRIBUTES name,cellVersion" # Check cellsrv status on all cells dcli -g /opt/oracle.SupportTools/onecommand/cell_group \ "service celld status"
Summary — Hardware Components at a Glance
| Component | What It Does | Key X10M Fact |
|---|---|---|
| Database Servers | Run Oracle Database and Grid Infrastructure | 192 cores, 512 GB to 3 TB DDR5, 200 Gb/sec RoCE |
| HC Storage Cells | Intelligent disk-and-flash storage with Smart Scan | 64 cores, 1.25 TB XRMEM, 264 TB raw disk per cell |
| EF Storage Cells | All-flash intelligent storage — highest IOPS | 64 cores, 1.25 TB XRMEM, 122.88 TB raw flash per cell |
| RoCE Switches | High-speed RDMA network — storage + RAC interconnect | 2 redundant switches, 200 Gb/sec per server, 17µs latency |
| Management Switch | Out-of-band management via ILOM | 1 GbE — always available even if OS is down |
| Client Network Switch | Application server connections to the database | 10 GbE — separate from internal RoCE fabric |
| PDUs | Redundant power distribution | 2 x dual-feed PDUs — no single point of power failure |
No comments:
Post a Comment