Exadata Administration Essentials — cellcli, dcli, and Managing Storage Cells Day to Day
Managing an Exadata system as a DBA involves two distinct workspaces — the database tier you already know, and the storage cell tier that is unique to Exadata. The storage cell tier has its own command-line interface, its own objects, its own alert system, and its own administration tasks. A DBA who only manages the database layer is managing half of Exadata.
This article covers everything you need for day-to-day storage cell administration. It explains the storage object model — the relationship between physical disks, celldisks, and griddisks — walks through the essential cellcli and dcli commands, shows you how to work with cell alerts and predictive failure warnings, and provides a complete routine health check reference you can run every day.
Two OS users for cellcli: celladmin can run all cellcli commands including those that modify configuration. cellmonitor can run read-only LIST commands only. For routine monitoring, always use cellmonitor. Reserve celladmin for configuration changes. Commands in this article that modify the cell are clearly marked.
The Exadata Storage Object Model
Before running any cellcli commands, you need to understand the three-layer object hierarchy that Exadata uses to represent storage. Every disk in an Exadata cell exists at three levels simultaneously, and each level has a different name, a different purpose, and different administration commands.
| Object | What It Represents | Who Manages It | Typical Naming |
|---|---|---|---|
| PHYSICALDISK | The actual physical hard disk or flash drive inside the storage cell. Identified by its slot position (e.g. 35:3) or flash device name (e.g. FLASH_4_0). | Exadata System Software — auto-detected | 35:3 (slot:disk) or FLASH_4_0 |
| CELLDISK | The logical representation of a physical disk within cellsrv. One celldisk per physical disk. This is the layer where Exadata System Software manages the disk. | Exadata System Software — created automatically | CD_00_cell01 |
| GRIDDISK | Logical partitions carved from a celldisk and presented to Oracle ASM. One celldisk can have multiple griddisks (e.g. one for DATA, one for RECO). ASM sees griddisks as disk devices. | DBA — can be created, dropped, resized | DATA_CD_00_cell01, RECO_CD_00_cell01 |
The relationship is: one PHYSICALDISK → one CELLDISK → one or more GRIDDISKs. When a physical disk fails, the celldisk becomes unavailable, which takes its griddisks offline, which causes ASM to drop those disks from the disk group and begin rebalancing. Understanding this chain is essential for diagnosing hardware events.
Connecting to cellcli
cellcli runs only on the storage cell — you cannot run it from a database node. You must SSH to each cell individually. The management network (not the RoCE storage network) is used for cellcli SSH access.
# Connect to a storage cell via SSH using the management hostname ssh celladmin@cell01-adm # Full admin access — can modify configuration ssh cellmonitor@cell01-adm # Read-only access — monitoring only # Open the interactive cellcli prompt cellcli # You will see the prompt: # CellCLI: Release 23.x.x.x — Production on [date] # CellCLI> # Run a single command without entering interactive mode cellcli -e "LIST CELL DETAIL" # Exit the interactive session CellCLI> EXIT
cellcli syntax rules: Commands are case-insensitive. Object type keywords (CELL, CELLDISK, GRIDDISK, etc.) and attribute names are case-insensitive. String values in WHERE filters are case-sensitive and must match exactly. Use a backslash (\) as a continuation character for long commands that span multiple lines.
Cell-Level Commands
The CELL object represents the entire storage server. Start here for any health check — it gives you the overall status of the cell before drilling into disks or metrics.
Managing Physical Disks — PHYSICALDISK
Physical disks are the raw hardware inside the storage cell. The most important thing to monitor at this level is the disk status. A status of normal is expected. Any deviation — particularly warning - predictive failure — requires immediate attention.
Normal status values for PHYSICALDISK:
normal — disk is healthy.
warning - predictive failure — S.M.A.R.T. diagnostics predict this disk will fail. Replace proactively before it fails completely.
failed — disk has failed. ASM will drop the associated griddisks and rebalance.
not present — no disk in this slot — expected for empty slots.
Managing Celldisks — CELLDISK
Celldisks are the logical representation of physical disks within Exadata System Software. They are created automatically when the cell is initialised and correspond one-to-one with physical disks. The DBA rarely needs to create or drop celldisks manually — but checking their status and free space is part of routine administration.
Celldisk status values mirror physicaldisk status. If a physicaldisk enters a failed state, the corresponding celldisk will also show as failed or in an error state. When a celldisk fails, all griddisks on that celldisk become unavailable and ASM rebalance begins automatically.
Managing Griddisks — GRIDDISK
Griddisks are what Oracle ASM sees as individual disk devices. Each celldisk is partitioned into one or more griddisks — typically one for the DATA disk group and one for the RECO disk group. The griddisk is the unit that ASM adds to or drops from a disk group.
Griddisks are the most frequently administered storage object. You will work with griddisks when taking a cell offline for maintenance, restoring a replaced disk, and verifying disk group membership after a hardware event.
Taking a griddisk offline and online — for planned maintenance
When you need to take a storage cell offline for maintenance (patching, hardware work), you must first quiesce the griddisks so ASM can handle the absence gracefully. This is a celladmin-only operation.
Do not take all cells offline simultaneously. ASM needs enough cells to satisfy the disk group redundancy requirements. For a normal redundancy disk group (2-way mirroring), you can take one cell offline at a time. For high redundancy (3-way mirroring), you can take up to two cells offline simultaneously — but always confirm ASM rebalance completes between each cell.
The Exadata Alert System
Exadata System Software monitors hundreds of metrics across every cell component — disks, CPUs, temperature sensors, fans, network interfaces, flash devices. When a metric crosses a threshold, an alert is generated and recorded in the alert history. Understanding how to read and manage alerts is one of the most important Exadata administration skills.
Alert severity levels
| Severity | Meaning | Action Required |
|---|---|---|
| critical | Component has failed or is at immediate risk. Data availability may be impacted. | Immediate — escalate to Oracle Support and hardware team |
| warning | Component is degraded or approaching a failure threshold. Not yet causing data loss. | Investigate within the same business day — plan remediation |
| informational | A notable event occurred but no failure. System state changes, rebalance completions, etc. | Review — no immediate action usually required |
| clear | A previously raised alert has been resolved automatically. | Verify the root cause is genuinely resolved |
Handling Predictive Failure Warnings
A predictive failure warning is one of the most important alerts an Exadata DBA receives. It means S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology) diagnostics inside the disk have detected early signs of failure — the disk has not yet failed, but its internal health indicators predict it will fail soon. The disk must be replaced before it fails completely to avoid data loss and an emergency rebalance operation.
Step-by-step response to a predictive failure warning
Always raise an Oracle Support SR before disk replacement, even for predictive failures. Oracle Support has tools to remotely diagnose the disk and will guide the replacement procedure. For systems under Oracle-managed Platinum support, Oracle may initiate the replacement proactively before you even see the alert.
The Exadata Alert Log
In addition to the cellcli alert history, Exadata System Software maintains an alert log on each storage cell — similar in concept to the Oracle Database alert log. This log records all significant events including daemon starts and stops, disk events, configuration changes, and hardware alerts.
dcli — Running Commands Across All Cells
dcli (Distributed CLI) executes the same command on multiple cells simultaneously and collects the combined output. It is the most time-efficient way to check the health of all cells in one command rather than SSH-ing into each one individually.
Managing Cell Services — Starting and Stopping
The three cell daemons — cellsrv, MS (Management Server), and RS (Restart Server) — run as a single service called celld. RS is the watchdog process that monitors and automatically restarts cellsrv and MS if they fail. In normal operations, you should rarely need to manually start or stop cell services.
Never restart cellsrv during active database I/O without first taking the griddisks INACTIVE and allowing ASM to rebalance. An abrupt cellsrv restart while databases are actively writing will cause ASM to drop the affected griddisks from the disk group, triggering an unplanned rebalance. Always take griddisks INACTIVE first for planned maintenance.
Flash Cache Administration
Smart Flash Cache is managed by Exadata System Software automatically — it populates, evicts, and manages cache contents without DBA intervention. However, there are specific administration tasks a DBA needs to perform, particularly during troubleshooting or after hardware replacement.
Routine Health Check — Full Command Reference
This is the complete daily administration runbook for Exadata storage cell health. Run all sections at the start of each working day. Total time: under 5 minutes when run via dcli.
Complete cellcli Command Reference
| Command | What It Does | Access Required |
|---|---|---|
LIST CELL |
Cell status overview | cellmonitor |
LIST CELL DETAIL |
Full cell info including hardware model, software version, network | cellmonitor |
LIST PHYSICALDISK |
All physical disk status | cellmonitor |
LIST PHYSICALDISK WHERE status != 'normal' |
Disks with problems only | cellmonitor |
LIST CELLDISK |
All celldisk status and size | cellmonitor |
LIST CELLDISK WHERE status != 'normal' |
Celldisks with problems | cellmonitor |
LIST GRIDDISK |
All griddisk status and ASM mapping | cellmonitor |
LIST GRIDDISK WHERE status != 'active' |
Griddisks not in active state | cellmonitor |
LIST ALERTHISTORY |
All cell alerts | cellmonitor |
LIST ALERTHISTORY WHERE severity LIKE '[warning|critical]' |
Warning and critical alerts only | cellmonitor |
LIST FLASHCACHE DETAIL |
Flash cache status and configuration | cellmonitor |
LIST METRICCURRENT CL_CPUT |
Cell CPU utilisation | cellmonitor |
LIST METRICCURRENT FC_BY_USED |
Flash cache utilisation % | cellmonitor |
LIST METRICCURRENT CD_IO_RQ_R_LG |
Large I/O (Smart Scan) request rate | cellmonitor |
LIST IORMPLAN |
I/O Resource Management plan | cellmonitor |
ALTER GRIDDISK ALL INACTIVE |
Take all griddisks offline for maintenance | celladmin |
ALTER GRIDDISK ALL ACTIVE |
Bring all griddisks back online | celladmin |
ALTER ALERTHISTORY <id> examinedBy = 'name' |
Acknowledge an alert | celladmin |
ALTER CELL RESTART SERVICES cellsrv |
Restart the cellsrv daemon | celladmin |
ALTER FLASHCACHE ALL FLUSH |
Clear the flash cache | celladmin |
CREATE GRIDDISK |
Create a new griddisk on a celldisk | celladmin |
DROP GRIDDISK |
Remove a griddisk | celladmin |
Summary
- Exadata storage has three object levels: PHYSICALDISK (hardware) → CELLDISK (cell software layer) → GRIDDISK (presented to ASM). Understanding this chain is essential for hardware event diagnosis.
cellcliruns on storage cells only — SSH to each cell individually. Use cellmonitor for read-only monitoring, celladmin for configuration changes.- The most critical daily check is
LIST PHYSICALDISK WHERE status != 'normal'— awarning - predictive failurestatus means the disk must be replaced proactively before it fails completely. dcliruns any cellcli command across all cells simultaneously using a group file — it is the only practical way to check all cells every day.- The cell alert system records all significant events in ALERTHISTORY. Check for unacknowledged warning and critical alerts daily using the dcli command in the health check script.
- The Exadata alert log on each cell (
/opt/oracle/cell/log/diag/asm/cell/) provides detailed event history for deep troubleshooting. - Before any planned cell maintenance, always take griddisks INACTIVE first and wait for ASM rebalance to complete. Never restart cellsrv with griddisks in active state during database I/O.
No comments:
Post a Comment