Dell PowerScale Deploy cheat sheet

OneFS is a distributed single-namespace file system; each node runs the full stack, so every node can serve any file and there is no dedicated metadata server or head node.

Node families: F-series (all-flash, performance), H-series (hybrid, capacity plus flash), A-series (archive, high-density HDD), P-series (performance HDD, legacy) - generation suffixes (e.g., F600, H700) denote newer hardware, not protocol support.

Data is protected by Reed-Solomon erasure coding at the file level: +2d:1n tolerates one node or drive failure, +2d:2n two, +3d:3n three, +4d:4n four; protection level is set per file pool policy, not per volume.

N+M notation: N data blocks and M parity blocks per stripe; usable capacity is roughly N/(N+M) of raw, minus metadata and snapshot reserve.

Access is scale-out: clients connect to any node's IP and OneFS redirects to the data owner, so adding nodes adds CPU, cache, and network bandwidth linearly.

PowerScale runs OneFS on the node itself (no separate controller); cluster identity is the cluster name plus a shared configuration, and all nodes must run compatible OneFS versions.

SyncIQ replicates at the file level to another PowerScale cluster; Snapshots are point-in-time, copy-on-write directory views stored in the same file system.

CloudIQ/DataIQ and OneFS health checks provide telemetry; SNMP, syslog, and REST API (platform API v1-vN) are the automation surfaces.

Separate networks: internal (InfiniBand or Ethernet backend between nodes, non-routable), external/front-end (client data plus management), and IPMI/BMC (out-of-band hardware management).

External interfaces support multiple IP ranges per subnet; SmartConnect zones group interfaces and use a pool of floating IPs with policies (round-robin, connection-count, CPU usage, network throughput) plus dynamic IP rebalancing.

SmartConnect requires a delegation of DNS (subdomain) to the cluster; the cluster answers A records for the zone name and returns available node IPs, enabling client-side load balancing.

InfiniBand backend: two rails per node (fabrics A and B) for redundancy; Ethernet backend requires a dedicated, isolated VLAN/subnet with jumbo frames consistent across nodes.

Link aggregation (LACP) is configured per interface group on the external network; verify switch-side port-channel matches the node's aggregation mode to avoid asymmetric hashing.

Sizing inputs: usable capacity target, protection overhead, snapshot reserve (default 0%, often 5-20%), workflow reserve, and expected growth; usable = raw x N/(N+M) minus reserves.

Subnet and VLAN planning must include management, external data, replication (SyncIQ), and backup traffic; replication traffic can be throttled by bandwidth rules per subnet.

DNS, NTP, and a default gateway are prerequisites; NTP skew across nodes breaks cluster health, and DNS delegation is required for SmartConnect name resolution.

Installation order: rack and cable (internal, external, IPMI), power on, configure each node's serial console or OneFS install, then run the cluster join/config wizard from the first node.

The configuration wizard collects cluster name, external and internal network settings, DNS/NTP, and admin password; the cluster must reach a consistent OneFS version before it is fully operational.

Access zones partition the namespace for multi-tenancy: System zone plus custom zones, each with its own authentication providers, SMB shares, NFS exports, and IP pools.

Authentication providers: local users, Active Directory (join domain, specify OU), LDAP, and NIS; AD join requires correct DNS, time sync, and a service account with join rights.

File pool policies place data by matching criteria (path, name, type, size, timestamps) and assign protection level, storage pool (SSD/HDD tiers), and snapshot settings.

Snapshot reserve is configured per file system; snapshots are scheduled per directory (SnapshotIQ) and consume reserve before spilling into free space.

Upgrades are rolling: OneFS upgrades node by node while the cluster stays online; verify health, backup config, and read release notes for drive/firmware prerequisites.

Post-install validation: cluster health (isi status), event log review, license activation, SmartConnect zone DNS test, and a test share/export mount from a client.

SMB supports SMB1/2/3, continuous availability (CA) shares for transparent failover, and SMB signing/encryption; SMB1 is disabled by default in current OneFS releases.

NFSv3 and NFSv4/4.1 are supported; NFSv4 uses a pseudofilesystem and requires idmap configuration for UID/GID mapping between the client and OneFS.

S3 support is provided by the object API (OneFS S3 service) with buckets mapped to directories, access keys, and bucket policies; it is not a full AWS feature parity.

HDFS access uses a NameNode proxy or direct connect; OneFS presents an HDFS-compatible namespace so Hadoop clients read/write without a separate data copy.

Permissions: OneFS uses a hybrid model - POSIX mode bits plus Windows ACLs (NTFS-style) on the same file; ACL policies per share/export control which is authoritative.

Multipath and client-side load balancing: use SmartConnect zone names, not individual node IPs, so clients spread across nodes and survive node loss.

Kerberos is required for NFSv4 with krb5 security flavors and for SMB in AD environments; configure the machine account and keytab correctly.

Quotas: directory quotas, user quotas, and default quotas are enforced by OneFS; advisory vs. hard limits and notification thresholds are set per quota domain.

Protection levels map to failures tolerated: +1n (one drive), +2d:1n (one node or drive), +2d:2n (two nodes), +3d:3n (three nodes), +4d:4n (four nodes); choose based on node count and rebuild time.

Snapshots are instant and space-efficient via copy-on-write; schedule with SnapshotIQ, set expiration, and monitor snapshot reserve usage to avoid unexpected space consumption.

SyncIQ policies replicate directories to a target cluster; policies support full/initial sync, incremental sync, and failover/failback with a change list for resync.

NDMP and native backup (e.g., Dell PowerProtect, Commvault) back up via NDMP or SMB/NFS; NDMP supports 3-way and 2-way (local) topologies.

Monitoring surfaces: isi status, isi event list, isi statistics, SNMP traps, syslog forwarding, and CloudIQ; alert rules can trigger on drive, node, or protocol events.

Common failure triage: node down (check IPMI, internal network, power), drive failure (OneFS auto-rebuilds; watch for multiple simultaneous failures), and network degradation (check interface errors and LACP).

Rebuild and restripe: after a drive or node failure, OneFS rebuilds data using parity; monitor isi status for 'restriping' and avoid additional maintenance during rebuild.

Support bundles (isi_gather_info) collect logs, config, and diagnostics for support; generate before changing configuration when troubleshooting an escalation.

SMB continuous availability requires CA shares and SMB3; clients reconnect transparently after node failover, but only for CA-aware clients.

NFS export options: root squash, read-only, and security flavors (sys, krb5, krb5i, krb5p); krb5p adds privacy (encryption) at a performance cost.

SmartConnect zone names must be used in mount commands (e.g., mount zone.cluster.example.com:/ifs/data) so DNS returns a live node IP.

Identity mapping: idmap maps Windows SIDs to POSIX UIDs/GIDs; misconfigured idmap causes 'nobody' ownership on NFS or access denied on SMB.

S3 buckets are created under a base directory with an access key and secret; bucket policies and user policies control access, and versioning is not natively supported the same way as AWS.

HDFS clients connect through a NameNode proxy or direct OneFS HDFS service; the namespace is shared with the file system so data is accessible via NFS/SMB too.

Quota enforcement: hard quotas block writes at the limit; soft quotas only notify; default quotas apply to new directories under a parent.

Protocol troubleshooting: check isi smb shares list, isi nfs exports list, and client-side mount errors; verify DNS resolution of the SmartConnect zone first.

Backend network must be isolated from client traffic; InfiniBand uses two fabrics (A/B) and Ethernet backend requires dedicated switches with consistent MTU (typically 9000).

External network supports multiple subnets per interface; each subnet can have a pool of IPs used by SmartConnect for load balancing and failover.

SmartConnect policies: round-robin (connection count), CPU usage, network throughput, and connection-count; choose based on workload (many small connections vs. few large streams).

IP pool rebalancing moves floating IPs between nodes when a node goes down or when load is uneven; configure a rebalance policy and interval.

Replication network: SyncIQ traffic can be segregated to a dedicated subnet and throttled with bandwidth rules to protect client performance.

Management network carries OneFS UI/API, IPMI is separate out-of-band; do not route management through the data subnet if policy requires isolation.

DNS delegation: create an NS record for the SmartConnect subdomain pointing to the cluster's external IPs; without it, zone names do not resolve.

Capacity math example: 10 nodes x 10 TB raw with +2d:2n (N=8, M=2) yields roughly 80 TB usable before snapshot and workflow reserves.

isi status shows cluster health, node states, and restriping progress; a node in 'down' or 'smartfailed' state triggers rebuild and reduced redundancy.

isi event list shows recent events by severity; correlate drive, network, and protocol events to distinguish a single failure from a cascading issue.

isi statistics (protocol, drive, node, client) identifies hot spots; high latency with low throughput often indicates network or client-side issues.

Snapshot reserve exhaustion: if snapshots exceed reserve, they spill into free space and can fill the file system; monitor with isi snapshot settings and space reports.

SyncIQ failover requires the target policy to be in a writable state; after failover, resync uses a change list to reconcile divergent data.

NDMP backups require a DMA (backup server) and a tape or disk target; verify NDMP service is enabled and credentials match the DMA configuration.

Drive replacement: identify the failed drive in isi status, replace physically, and OneFS rebuilds automatically; verify the new drive is recognized and rebuild completes.

Escalation: capture isi_gather_info, note the OneFS version, and record the timeline of events before contacting support.

OneFS versions are released as feature and maintenance releases; all nodes must run the same version, and upgrades are rolling with no full-cluster downtime.

Node types determine performance and capacity: all-flash for IOPS-sensitive workloads, hybrid for mixed, archive for cold data; mixing types in a cluster is supported but affects policy placement.

Protection level is per file pool policy and can be changed; higher protection (e.g., +3d:3n) reduces usable capacity but tolerates more simultaneous failures.

Snapshots and SyncIQ are licensed features (SnapshotIQ, SyncIQ); base OneFS includes core file system and protocols.

The OneFS REST API (platform API) exposes configuration, statistics, and provisioning; automation tools and Dell tools use it.

Cluster identity: cluster name, GUID, and node serial numbers; node replacement requires matching or compatible hardware and OneFS version.

Data is accessed through a single namespace (/ifs), so NFS, SMB, S3, and HDFS clients see the same files without copies.

Health checks: isi healthcheck runs a suite of checks on configuration, network, and data protection; run after install and after major changes.

Before the wizard: verify all nodes power on, internal network links are up, and each node is reachable via serial or IPMI console.

The wizard's first node becomes the configuration source; other nodes join and receive the cluster config over the internal network.

Licensing: activate licenses for SnapshotIQ, SyncIQ, SmartQuotas, and other features via the OneFS UI or API before configuring dependent policies.

Access zone creation: each zone has its own IP pool, authentication providers, and SMB/NFS services; the System zone is reserved for administration.

AD join steps: configure DNS to resolve the domain, set NTP, provide a domain admin or delegated account, and specify the computer OU if required.

File pool policies are evaluated in order; the first matching policy applies, so order specific policies before general ones.

Snapshot schedules: set directory, interval, retention, and whether to use the snapshot reserve; test restore from a snapshot before relying on it.

Validation checklist: isi status healthy, DNS zone resolves, test SMB mount and NFS export, snapshot created and restored, and SyncIQ policy tested if licensed.

SMB share permissions combine share-level and file-level ACLs; the effective permission is the intersection, so both must grant access.

NFSv4 ACLs and POSIX mode bits can coexist; the 'acl' mount option and idmap determine how clients see permissions.

S3 access keys are per-user; bucket policies can grant cross-account access, and the S3 service must be enabled per access zone.

HDFS high availability uses a NameNode proxy; clients should use the proxy address, not individual node IPs, for failover.

Client multipath: use DNS round-robin via SmartConnect, or configure multiple mount paths; OneFS redirects to the data owner.

Kerberos troubleshooting: check keytab, time sync (within 5 minutes), and SPN registration; clock skew is the most common cause of krb5 failures.

Quota reporting: isi quota list and isi quota reports show usage per user/group/directory; notifications can trigger at thresholds.

Protocol performance: enable jumbo frames end-to-end, use multiple client connections, and verify no interface errors or drops on the node side.

Redundancy states: 'healthy' means full protection; 'degraded' means a drive or node is missing and rebuild is in progress; 'critical' means additional failures would cause data loss.

isi status -q gives a quick health summary; use it in scripts and monitoring checks.

Event severity levels (info, notice, warning, critical, emergency) map to SNMP traps and syslog; forward to a central collector for correlation.

Snapshot restore: restore a single file or directory from a snapshot without reverting the whole file system; use the .snapshot directory or the UI.

SyncIQ monitoring: isi sync policies list and isi sync reports show last sync status, lag, and errors; a failed policy should be investigated before failover.

NDMP log and DMA logs must be correlated; a common failure is credential or network mismatch between the DMA and the cluster.

Network troubleshooting: check interface status (isi network interfaces list), errors/drops, LACP state, and DNS resolution of SmartConnect zones.

Capacity troubleshooting: isi space reports show used, available, and snapshot usage; unexpected growth often comes from snapshots or SyncIQ staging.

OneFS is a scale-out NAS operating system; the cluster is the unit of management, not individual nodes.

Data layout: files are striped across nodes with parity; a file's blocks live on multiple nodes, so node loss does not lose data if protection allows.

Protection level notation: +Nn means N mirror copies; +Nd:Mn means N data and M parity per stripe; choose based on node count.

Usable capacity formula: raw capacity x (N / (N+M)) minus metadata, snapshot reserve, and workflow reserve.

Licensing model: base OneFS plus optional feature licenses (SnapshotIQ, SyncIQ, SmartQuotas, CloudPools, S3).

CloudPools tiers cold data to cloud object storage (e.g., AWS S3, Azure Blob) while keeping the namespace local.

API and automation: OneFS platform API supports REST; use it for provisioning, monitoring, and integration with Dell tools.

Upgrade path: check the OneFS release notes for supported upgrade paths and drive firmware; rolling upgrades keep the cluster online.

Node count affects protection choices: +2d:2n requires at least 3 nodes, +3d:3n at least 4, +4d:4n at least 5; plan node count before choosing protection.

Network throughput planning: aggregate client bandwidth should not exceed external network capacity; use multiple 25/100 GbE links and LACP where possible.

SmartConnect zone design: one zone per subnet or tenant; zone names must be delegated in DNS and should not overlap with existing records.

IP pool sizing: provide at least one floating IP per node per subnet for failover; more IPs improve load distribution.

Backend network: InfiniBand or Ethernet; Ethernet backend requires dedicated switches and consistent MTU; do not share with client traffic.

Replication planning: SyncIQ bandwidth and RPO determine link sizing; throttle replication to avoid impacting client workloads.

Security planning: separate management and data networks, enable SMB signing/encryption where required, and use Kerberos for NFSv4.

Growth planning: reserve capacity for snapshot and workflow reserves; monitor with isi space and plan expansion before 80% utilization.

Installation media and licensing: obtain the OneFS install image and license keys before starting; verify hardware compatibility with the OneFS release.

Node join: nodes must be on the same internal network and OneFS version; the wizard discovers and joins them.

Cluster config: cluster name, external IPs, DNS, NTP, and admin credentials; changing these later requires care and may affect clients.

Access zone and IP pool: create the zone, assign an IP pool, and enable SMB/NFS services per zone.

Authentication: join AD or configure LDAP/NIS per zone; test with a domain user before creating shares.

File pool policies: define storage pools (SSD/HDD/archive) and policies that place data and set protection; test with a small directory first.

Snapshot and SyncIQ: configure schedules and policies after data is placed; verify with a test restore and a test replication.

Documentation: record cluster name, IPs, zone names, and policy settings for future troubleshooting and handover.

SMB3 encryption and signing protect data in transit; enabling them may reduce throughput, so test performance impact.

NFSv4.1 supports sessions and pNFS; OneFS supports NFSv4.1 for modern clients; NFSv3 remains common for legacy workloads.

S3 compatibility: use the OneFS S3 endpoint and access keys; some AWS features (e.g., object lock) may not be supported.

HDFS access: OneFS provides an HDFS-compatible namespace; clients use the NameNode proxy for HA.

Multitenancy: access zones isolate authentication and IP pools; a client in one zone cannot see another zone's shares unless configured.

Permission troubleshooting: use isi smb shares permission and isi nfs exports to inspect; check idmap for UID/GID mismatches.

Quota enforcement: hard quotas block writes; soft quotas notify; default quotas apply to new directories.

Client tuning: increase mount read/write sizes, use multiple connections, and verify network path (jumbo frames, no drops).

Health monitoring: isi status, isi healthcheck, and CloudIQ; set up SNMP/syslog forwarding for central monitoring.

Drive failure handling: OneFS marks the drive smartfailed and rebuilds; replace the drive and verify rebuild completes with isi status.

Node failure: check IPMI and internal network; a down node triggers rebuild if protection allows; do not remove a node without following the proper procedure.

Snapshot space: monitor snapshot reserve; if snapshots exceed reserve, they consume free space and can fill the file system.

SyncIQ failover: verify target policy state, perform failover, then resync with a change list; test failover in a lab before relying on it.

NDMP: verify NDMP service, credentials, and network path; correlate DMA and cluster logs for failures.

Network issues: check interface errors, LACP state, DNS resolution, and SmartConnect zone health.

Escalation: gather isi_gather_info, record timeline, and include OneFS version and recent changes.

OneFS is a clustered file system with a single namespace; all nodes serve all data, and there is no single point of failure for data access.

Node generations: F-series (all-flash), H-series (hybrid), A-series (archive); newer generations offer higher density and performance.

Protection levels: +1n (one drive), +2d:1n (one node), +2d:2n (two nodes), +3d:3n (three nodes), +4d:4n (four nodes).

Usable capacity: raw x N/(N+M) minus reserves; plan for snapshot and workflow reserves.

Licensing: base OneFS plus optional features (SnapshotIQ, SyncIQ, SmartQuotas, CloudPools, S3).

Automation: OneFS REST API for provisioning, monitoring, and integration.

Upgrades: rolling, online; check release notes for supported paths and firmware.

Health: isi status, isi healthcheck, SNMP/syslog, CloudIQ.

Internal network: InfiniBand (two fabrics) or Ethernet (dedicated switches, jumbo frames); isolate from client traffic.

External network: multiple subnets and IP pools; SmartConnect zones for load balancing and failover.

SmartConnect policies: round-robin, CPU usage, network throughput, connection count; choose based on workload.

DNS delegation: NS record for the SmartConnect subdomain; without it, zone names do not resolve.

Replication network: separate subnet and bandwidth throttling for SyncIQ.

Management and IPMI: separate networks for security and isolation.

Sizing: node count, protection level, snapshot reserve, and growth determine usable capacity and performance.

Security: SMB signing/encryption, Kerberos for NFSv4, network segmentation.

Pre-install: rack, cable (internal, external, IPMI), power on, verify hardware compatibility.

Wizard: cluster name, external/internal network, DNS/NTP, admin password; nodes join and receive config.

Access zones: System zone plus custom zones with their own IP pools, authentication, and services.

Authentication: AD join (DNS, NTP, service account), LDAP, NIS; test with a domain user.

File pool policies: match criteria, protection level, storage pool, snapshot settings; order specific before general.

Snapshots: schedules, retention, reserve; test restore.

SyncIQ: policies, failover/failback, change list resync; test replication.

Validation: isi status, DNS resolution, test SMB/NFS mount, snapshot restore, SyncIQ test.

SMB: SMB1/2/3, CA shares, signing/encryption; SMB1 disabled by default.

NFS: v3 and v4/4.1; NFSv4 requires idmap and Kerberos for krb5 flavors.

S3: object API with buckets, access keys, bucket policies; not full AWS parity.

HDFS: NameNode proxy for HA; shared namespace with NFS/SMB.

Permissions: POSIX mode bits plus Windows ACLs; share and file permissions intersect.

SmartConnect: use zone names in mount commands for load balancing and failover.

Quotas: directory, user, default; hard vs. soft limits.

Troubleshooting: check share/export lists, idmap, DNS resolution, and client mount errors.

isi status: cluster health, node states, restriping progress.

isi event list: recent events by severity; correlate drive, network, protocol events.

isi statistics: protocol, drive, node, client; identify hot spots and latency.

Snapshot reserve: monitor usage; snapshots can spill into free space and fill the file system.

SyncIQ: monitor policy status, lag, errors; test failover.

NDMP: verify service, credentials, network; correlate DMA and cluster logs.

Drive replacement: identify failed drive, replace, verify rebuild completes.

Escalation: isi_gather_info, timeline, OneFS version, recent changes.

OneFS: scale-out NAS OS; cluster is the unit of management.

Data layout: files striped across nodes with parity; node loss tolerated per protection level.

Protection notation: +Nn (mirror), +Nd:Mn (data/parity); choose based on node count.

Usable capacity: raw x N/(N+M) minus reserves.

Licensing: base plus optional features.

CloudPools: tier cold data to cloud object storage.

API: REST for automation and monitoring.

Upgrades: rolling, online; check release notes.

Node count and protection: +2d:2n needs 3+ nodes, +3d:3n 4+, +4d:4n 5+.

Bandwidth: aggregate client bandwidth vs. external network capacity; use multiple links and LACP.

SmartConnect zones: one per subnet/tenant; DNS delegation required.

IP pools: at least one floating IP per node per subnet for failover.

Backend: InfiniBand or dedicated Ethernet; consistent MTU; isolate from clients.

Replication: SyncIQ bandwidth and RPO determine link sizing; throttle to protect clients.

Security: separate management/data networks; SMB signing/encryption; Kerberos for NFSv4.

Growth: reserve capacity; monitor and expand before 80% utilization.

Pre-install: media, licenses, hardware compatibility.

Node join: same internal network and OneFS version; wizard discovers and joins.

Cluster config: name, IPs, DNS, NTP, admin credentials.

Access zone and IP pool: create zone, assign pool, enable services.

Authentication: AD/LDAP/NIS per zone; test with domain user.

File pool policies: storage pools, protection, placement; test with small directory.

Snapshot and SyncIQ: configure after data placement; test restore and replication.

Documentation: record cluster name, IPs, zones, policies.

SMB3 encryption/signing: protect data in transit; test performance impact.

NFSv4.1: sessions and pNFS; NFSv3 for legacy.

S3: OneFS endpoint and access keys; some AWS features unsupported.

HDFS: NameNode proxy for HA; shared namespace.

Multitenancy: access zones isolate authentication and IP pools.

Permission troubleshooting: isi smb shares permission, isi nfs exports, idmap.

Quotas: hard vs. soft; default quotas.

Client tuning: mount sizes, multiple connections, jumbo frames.

Health: isi status, isi healthcheck, CloudIQ, SNMP/syslog.

Drive failure: smartfailed, rebuild, replace, verify.

Node failure: check IPMI and internal network; proper removal procedure.

Snapshot space: monitor reserve; spill into free space.

SyncIQ failover: verify target state, failover, resync with change list.

NDMP: verify service, credentials, network; correlate logs.

Network: interface errors, LACP, DNS, SmartConnect health.

Escalation: isi_gather_info, timeline, version, recent changes.

OneFS: clustered file system, single namespace, no single point of failure.

Node generations: F (all-flash), H (hybrid), A (archive).

Protection: +1n, +2d:1n, +2d:2n, +3d:3n, +4d:4n.

Usable capacity: raw x N/(N+M) minus reserves.

Licensing: base plus optional features.

Automation: REST API.

Upgrades: rolling, online.

Health: isi status, isi healthcheck, SNMP/syslog, CloudIQ.

Internal: InfiniBand or dedicated Ethernet; isolate from clients.

External: multiple subnets and IP pools; SmartConnect zones.

SmartConnect policies: round-robin, CPU, throughput, connection count.

DNS delegation: NS record for zone subdomain.

Replication: separate subnet and throttling.

Management and IPMI: separate networks.

Sizing: node count, protection, reserves, growth.

Security: SMB signing/encryption, Kerberos, segmentation.

Pre-install: rack, cable, power, compatibility.

Wizard: cluster name, network, DNS/NTP, admin password.

Access zones: System plus custom; IP pools, auth, services.

Authentication: AD join (DNS, NTP, account), LDAP, NIS.

File pool policies: match criteria, protection, storage pool; order specific first.

Snapshots: schedules, retention, reserve; test restore.

SyncIQ: policies, failover, resync; test replication.

Validation: isi status, DNS, mounts, restore, replication.

SMB: SMB1/2/3, CA shares, signing/encryption; SMB1 disabled by default.

NFS: v3 and v4/4.1; idmap and Kerberos for krb5.

S3: buckets, access keys, policies; not full AWS parity.

HDFS: NameNode proxy; shared namespace.

Permissions: POSIX plus ACLs; share and file intersect.

SmartConnect: zone names in mounts.

Quotas: directory, user, default; hard vs. soft.

Troubleshooting: share/export lists, idmap, DNS, mount errors.

isi status: health, node states, restriping.

isi event list: severity, correlation.

isi statistics: protocol, drive, node, client.

Snapshot reserve: monitor; spill into free space.

SyncIQ: status, lag, errors; test failover.

NDMP: service, credentials, network; correlate logs.

Drive replacement: identify, replace, verify rebuild.

Escalation: isi_gather_info, timeline, version, changes.