Exadata Patching — How to Patch Database Nodes, Storage Cells, and Switches
Patching an Exadata system is significantly more complex than patching a standard Oracle database server. An Exadata environment has multiple distinct components — database servers, storage cells, and network switches — each running their own software stack, each patched with different tools, and each requiring a specific sequence relative to the others. Patching one component out of order can leave the system in an inconsistent state.
This article covers the complete Exadata patching process from end to end — what the bundle patches contain, the correct patching order, how rolling and non-rolling modes work, the patchmgr commands for each component, how OPatch fits into the process for DB software, and how to verify patch levels across all components when you are done.
Always read the MOS patch readme before starting. Exadata bundle patches frequently include specific pre-patch steps, known issues, and order-of-operations requirements that are specific to that release. The MOS readme supersedes general guidance including the guidance in this article. Reference MOS Doc ID 888828.1 — Exadata Database Machine and Exadata Storage Server Supported Versions — for the current patch list.
What Gets Patched in an Exadata System
An Exadata system has four distinct software layers, each patched separately with different tools. Understanding this is the foundation of everything else in this article.
| Component | Software Patched | Tool Used | Downtime Impact |
|---|---|---|---|
| Database Servers — Infrastructure | Oracle Linux OS, firmware, ILOM, drivers, Exadata System Software on DB nodes | patchmgr (DBBP) | Rolling — one node at a time, RAC stays up |
| Database Servers — DB Software | Oracle Grid Infrastructure, Oracle Database (CPU, PSU, RUs) | OPatch + datapatch | Rolling — one node at a time via GI rolling patches |
| Storage Cells | Oracle Linux OS, Exadata System Software, firmware, cellsrv, flash drivers | patchmgr (EXABP) | Rolling — one cell at a time, ASM handles I/O |
| RoCE / Network Switches | Switch firmware and software | patchmgr | Rolling — one switch at a time |
The Two Exadata Bundle Patches — DBBP and EXABP
Oracle delivers Exadata infrastructure patches as two bundle patches. Understanding the difference between them is essential before you download anything.
DBBP — Database Server Bundle Patch
The Database Server Bundle Patch (DBBP) is the infrastructure patch for the database server nodes. It updates everything on the DB nodes except the Oracle Database and Grid Infrastructure software itself.
DBBP includes:
- Oracle Linux OS updates — kernel, system libraries, security patches
- System firmware — BIOS, BMC/ILOM, HBA, NIC firmware
- Storage and network driver updates
- Exadata-specific utilities — imageinfo, patchmgr itself, support tools
- Hardware compatibility updates for new component generations
DBBP does not include Oracle Grid Infrastructure or Oracle Database software patches. Those come from the standard Oracle Database CPU/RU patches applied via OPatch — the same process used on any Oracle database server.
EXABP — Exadata Storage Server Bundle Patch
The Exadata Storage Server Bundle Patch (EXABP) is the patch for storage cells. It updates everything that runs on the storage servers.
EXABP includes:
- Oracle Linux OS updates on the storage cell
- Exadata System Software — cellsrv, MS, RS daemon updates
- Storage cell firmware — disk controller, flash device firmware, ILOM
- Flash driver and NVMe driver updates
- Cell offload server (CELLOFLSRVn) updates for newer Oracle DB versions
- Security patches for the cell OS and software stack
EXABP and DBBP are released on the same quarterly schedule as Oracle Database CPUs — January, April, July, and October. They are aligned with the quarterly CPU cycle so that all Exadata components can be brought to a consistent patch level in a single quarterly patching activity.
The Correct Patching Order
This is the most critical section of this article. Applying Exadata patches in the wrong order can break component compatibility, cause cellsrv to fail, or leave the system partially updated. Oracle mandates a specific sequence. Never deviate from it without explicit Oracle Support instruction.
| Step | Component | Tool | Why This Order |
|---|---|---|---|
| 1 | RoCE / Network Switches | patchmgr | Network infrastructure must be updated first to support newer protocols used by DB nodes and cells |
| 2 | Storage Cells (EXABP) | patchmgr -cells | Storage software must be updated before DB node infrastructure. DB node connects to cells — newer cell software is backward compatible with older DB software but not always forward compatible |
| 3 | Database Server Infrastructure (DBBP) | patchmgr -dbnodes | DB node OS and firmware updated after cells are at new version. Node reboots happen here. |
| 4 | Oracle Grid Infrastructure (GI) | OPatch (GI rolling) | GI patch applied after OS is at new level. Rolling — one node at a time via GI rolling patch mechanism |
| 5 | Oracle Database CPU/RU | OPatch + datapatch | DB patch applied last. datapatch applies SQL changes after all nodes are patched. |
Always apply the network switch patch first, then cells, then DB nodes, then GI, then DB software. If the MOS readme for your specific patch bundle states a different order, follow the readme. Some patches modify this sequence for specific dependency reasons.
Rolling vs Non-Rolling Patching Mode
patchmgr supports two patching modes — rolling and non-rolling. The choice between them determines whether databases stay online during patching and how long the total patching window takes.
Rolling mode — zero downtime for RAC databases
In rolling mode, patchmgr patches one component at a time while the others continue serving workloads. For storage cells, it patches one cell at a time — the other cells continue serving I/O to all databases. ASM handles the temporary unavailability of the cell being patched through its normal redundancy mechanisms without data movement. For database nodes, rolling means patching one RAC node at a time while the other nodes continue running the database.
- Databases remain online throughout patching
- Takes longer overall — each cell or node is patched sequentially
- Requires ASM redundancy — data must be mirrored so I/O can continue while one cell is down
- Recommended for production environments where availability is critical
Non-rolling mode — faster but requires downtime
In non-rolling mode, patchmgr patches all components in parallel simultaneously. For storage cells, all cells are patched at the same time — databases cannot access any storage during this period. For database nodes, all nodes are patched and rebooted simultaneously.
- Databases must be shut down before starting
- Total patching time is significantly shorter — all cells patch in parallel
- Appropriate for development, test environments, or planned maintenance windows
- Non-rolling is the default mode for storage cells when no -rolling flag is specified
| Factor | Rolling Mode | Non-Rolling Mode |
|---|---|---|
| Database availability | Online throughout — no outage | Must be shut down — full outage |
| Total patch time | Longer — sequential component by component | Shorter — all cells patch simultaneously |
| ASM requirement | Requires normal or high redundancy disk groups | No redundancy requirement during patching |
| Default for cells | No — specify -rolling flag explicitly | Yes — default when no flag specified |
| Recommended for | Production — maximum availability | Development, test, short maintenance windows |
| Per-cell patch time | Approximately 1 to 2 hours per cell | Same per cell but all cells run in parallel |
Manual rolling update — avoid when possible. There is a third option called manual rolling update where each cell is taken completely offline (griddisks dropped from ASM), patched, then added back — forcing two full ASM rebalances per cell. This method requires enough free space in the disk group to hold all data from the most full cell. It is extremely time-consuming and should be used only when patchmgr's built-in rolling mode is not available for a specific patch.
Pre-Patch Checks — Exadata Specific
Exadata has its own set of pre-patch checks that go beyond the standard Oracle database pre-patch checklist. Run all of these before starting any patchmgr operation.
1. Run ExaCheck (formerly exachk)
ExaCheck is Oracle's health check tool specifically for Exadata. It validates hundreds of configuration items across DB nodes, storage cells, and network infrastructure. Run it and resolve all critical and warning findings before patching.
# ExaCheck is usually located here on the DB node /opt/oracle.SupportTools/exachk/exachk # Or via the support tools bundle /opt/oracle.SupportTools/onecommand/exachk # Run against all components ./exachk -a # Review the HTML report generated in the output directory # Fix all FAIL and WARNING items before starting patching
2. Verify current version on all components
3. Verify ASM disk group health and redundancy
4. Check for active database sessions and pending operations
5. Verify SSH key-based authentication from DB node to all cells
patchmgr requires password-less SSH access from the primary database node to all storage cells. If SSH authentication fails for any cell, patchmgr cannot patch that cell.
Step 1 — Patching Network Switches
Network switch patching is the first step. On X10M and X11M systems using RoCE switches, patchmgr handles the switch patching. The process patches one switch at a time — during the patching of one switch, all traffic routes through the other, so database I/O continues without interruption.
Step 2 — Patching Storage Cells with patchmgr (EXABP)
Storage cell patching uses patchmgr with the -cells flag. patchmgr is run from the primary database node — it connects to all cells via SSH and orchestrates the entire update process including pre-checks, OS updates, firmware updates, cellsrv updates, and post-patch verification.
Prepare patchmgr for cell patching
Pre-check — validate cells are ready for patching
Apply the cell patch — rolling mode
What patchmgr does to each cell during patching
- Copies the patch bundle to the cell via SSH/SCP
- Runs the cell pre-check to confirm readiness
- For rolling mode: waits until ASM confirms redundancy is maintained
- Stops cellsrv on the cell being patched
- Applies Oracle Linux OS updates, firmware updates, and Exadata System Software
- Reboots the storage cell — typically 15 to 30 minutes per cell
- Validates the cell post-reboot and confirms cellsrv is running at the new version
- For rolling mode: confirms ASM I/O is restored before moving to the next cell
Step 3 — Patching Database Server Infrastructure (DBBP)
After storage cells are patched, the database server infrastructure is updated using patchmgr with the -dbnodes flag. This updates Oracle Linux, firmware, and Exadata tools on the DB nodes. It requires a node reboot — RAC ensures the database stays available during rolling node reboots.
Always run patchmgr with nohup and redirect output to a log file. If your SSH session disconnects during node patching — which commonly happens when patchmgr reboots the node you are connected to — the patchmgr process continues running in the background. You can reconnect to another node and check the log file to confirm progress.
Step 4 — Patching Oracle Grid Infrastructure and Database (OPatch)
After the infrastructure is at the new level, Oracle Grid Infrastructure and Oracle Database software is patched using the standard OPatch process — the same process used on any Oracle database platform. The key difference on Exadata is that GI rolling patches are applied one node at a time using the GI rolling patch mechanism.
Step 5 — Verifying Patch Levels Across All Components
After patching is complete, verify every component is at the expected version. Do not close the change ticket until all components are confirmed.
Rolling Back a Cell Patch
If a storage cell patch causes issues, patchmgr supports rollback to the previous image. Rollback is only available when the cell was patched successfully — a cell that failed mid-patch cannot be rolled back and requires Oracle Support assistance.
Firmware updates applied as part of the cell patch are not rolled back when rolling back to a previous software image. After rollback, if you need to revert firmware as well, Oracle Support will guide the firmware downgrade procedure. This is another reason to always test patches in a non-production environment before applying to production.
Exadata Patching Checklist
| # | Task | Tool | Done |
|---|---|---|---|
| 1 | Read the MOS readme for DBBP and EXABP — note any special steps | MOS | [ ] |
| 2 | Download DBBP, EXABP, GI patch, DB CPU from MOS | MOS | [ ] |
| 3 | Run ExaCheck and resolve all FAIL and WARNING findings | exachk | [ ] |
| 4 | Verify current version on all DB nodes — imageinfo | imageinfo | [ ] |
| 5 | Verify current version on all cells — imageinfo -ver via dcli | dcli | [ ] |
| 6 | Verify ASM disk groups healthy — no errors, no rebalance in progress | SQL | [ ] |
| 7 | Verify SSH key-based auth from primary DB node to all cells | dcli | [ ] |
| 8 | Take RMAN full backup of all databases | RMAN | [ ] |
| 9 | Patch network switches — patchmgr -switches | patchmgr | [ ] |
| 10 | Run patchmgr pre-check for cells — resolve any FAIL findings | patchmgr -precheck | [ ] |
| 11 | Patch storage cells — patchmgr -cells -rolling (or non-rolling) | patchmgr | [ ] |
| 12 | Verify all cells at new version and all griddisks active | dcli / cellcli | [ ] |
| 13 | Patch DB node infrastructure — patchmgr -dbnodes -rolling | patchmgr | [ ] |
| 14 | Verify all DB nodes rejoined cluster and databases running | crsctl | [ ] |
| 15 | Apply GI patch — opatchauto | OPatch | [ ] |
| 16 | Apply DB CPU patch — opatch apply on each DB home | OPatch | [ ] |
| 17 | Run datapatch -verbose and verify completion | datapatch | [ ] |
| 18 | Verify all component versions — imageinfo, opatch lsinventory, cellcli | Multiple | [ ] |
| 19 | Run Smart Scan health check — confirm offload statistics healthy | V$SYSSTAT | [ ] |
| 20 | Run ExaCheck again post-patch and confirm no new failures | exachk | [ ] |
Summary
- DBBP patches the database server infrastructure — OS, firmware, Exadata tools. Applied with patchmgr -dbnodes.
- EXABP patches storage cells — Oracle Linux, Exadata System Software, cellsrv, firmware. Applied with patchmgr -cells.
- Correct patching order is always: switches → storage cells → DB node infrastructure → Grid Infrastructure → Oracle Database.
- Rolling mode patches one component at a time while RAC databases stay online. Non-rolling patches all in parallel but requires database downtime. Non-rolling is the default for cells — specify -rolling explicitly for production.
- patchmgr is run from the primary database node — it orchestrates SSH connections to all cells and handles the entire update process including reboot and post-patch verification.
- OPatch and datapatch handle GI and Oracle Database software — the same process as any Oracle database, applied after the infrastructure layer is at the new version.
- Always run ExaCheck before and after patching. Always take a full RMAN backup before starting. Always read the MOS readme for the specific patch bundle — it supersedes general guidance.
No comments:
Post a Comment