From f450776af27ac6ed853d93a7509e6d240c2a1434 Mon Sep 17 00:00:00 2001 From: Bala Harish A Date: Mon, 24 Aug 2026 12:43:26 +0530 Subject: [PATCH 1/9] docs: create new release notes for v4.6 Signed-off-by: Bala Harish A --- docs/main/releases.md | 146 ++++++++++++++++++++++++++---------------- 1 file changed, 92 insertions(+), 54 deletions(-) diff --git a/docs/main/releases.md b/docs/main/releases.md index bab9f4901..2cbf8d8ce 100644 --- a/docs/main/releases.md +++ b/docs/main/releases.md @@ -9,124 +9,162 @@ keywords: description: This page contains list of supported OpenEBS releases. --- -**Release Date: 06 June 2026** +**Release Date: TBD** OpenEBS is a collection of data engines and operators to create different types of replicated and local persistent volumes for Kubernetes Stateful workloads. Kubernetes volumes can be provisioned via CSI Drivers or using Out-of-tree Provisioners. -The status of the various components as of v4.5 are as follows: +The status of the various components as of v4.6 are as follows: | Component Type | Component | Version | Status | | :--- | :--- | :--- | :--- | -| Replicated Storage | Replicated PV Mayastor | 2.11.1 | Stable | -| Local Storage | Local PV Hostpath | 4.5.1 | Stable | -| Local Storage | Local PV LVM | 1.9.1 | Stable | -| Local Storage | Local PV ZFS | 2.10.1 | Stable | -| Local Storage | Local PV Rawfile | 0.14.1 | Experimental | -| Out-of-tree (External Storage) Provisioners | Local PV Hostpath | 4.5.1 | Stable | -| Other Components | CLI | 4.5.1 | — | +| Replicated Storage | Replicated PV Mayastor | 2.12.0 | Stable | +| Local Storage | Local PV Hostpath | 4.6.0 | Stable | +| Local Storage | Local PV LVM | 1.10.0 | Stable | +| Local Storage | Local PV ZFS | 2.11.0 | Stable | +| Local Storage | Local PV Rawfile | 0.15.0 | Experimental | +| Out-of-tree (External Storage) Provisioners | Local PV Hostpath | 4.6.0 | Stable | +| Other Components | CLI | 4.6.0 | — | ## What’s New +### General + +- **Optional CSI Snapshot Controller** + + Replicated PV Mayastor, Local PV LVM, Local PV ZFS, and Local PV Rawfile now allow you to disable the bundled CSI snapshot controller through the Helm chart. This avoids conflicts in clusters where a snapshot controller is already managed at the cluster level. + ### Replicated Storage -- **Offline Node Deletion (Node Purge)** +- **Eventing Aggregator** - Replicated PV Mayastor now supports offline node deletion (purge), allowing administrators to permanently remove an unreachable and unrecoverable node from the control plane without requiring access to the underlying host. Before performing the operation, you can review the expected impact on volumes and snapshots. This capability helps simplify recovery and cleanup workflows following permanent node or infrastructure failures. + Replicated PV Mayastor now includes the Eventing Aggregator, a new component that collects cluster events and makes them available for querying through the `get events` command in the Mayastor kubectl plugin. Events can be retrieved from Loki, directly from NATS JetStream, or from a previously collected support bundle for offline analysis, and can be filtered by category, action, node, target, component, pool, volume, replica, and state. Cluster events are also included in the support bundle produced by `kubectl openebs dump system`. The Eventing Aggregator is enabled by default and works with or without Loki deployed. -- **Offline and Online Pool Deletion** +- **TLS for Mayastor Service Endpoints** - Replicated PV Mayastor now supports deleting both offline and online Pools. Online pools can be deleted when they no longer contain replicas, while unrecoverable offline pools can be safely purged from the control plane after reviewing the impact on affected volumes and snapshots. + Replicated PV Mayastor now supports TLS for its service endpoints, starting with the REST API, along with the CSI controller, CSI node, DiskPool operator, metrics exporter, and kubectl plugin clients. Certificates can be managed in three ways: a transient self-signed certificate generated by the server at startup, self-signed certificates generated by the Helm chart, or certificates provisioned and rotated by cert-manager. Mutual TLS is supported, and the server reloads certificates without a restart when they are renewed. TLS is opt-in through `security.tls.enabled`, and plain-text communication remains the default. -- **Disk I/O Failure and Hot-Removal Handling** +- **Best-Effort Snapshot Restore Policy** - Replicated PV Mayastor now improves storage fault visibility by detecting disk I/O failures, hot-removal events, stalled I/O conditions, and runtime disk I/O errors. DiskPools automatically report updated pool states, alerts, and diagnostic information, helping you identify unhealthy storage devices and understand workload impact during disk-related failures. + A new `snapshotRestorePolicy` StorageClass parameter controls how a snapshot restore behaves when not every replica pool can host a clone. With the default `strict` policy, every requested replica must be cloned from the snapshot. With `bestEffort`, the restore proceeds as long as at least one clone succeeds; the volume comes up under-replicated and the remaining replicas are filled in through a normal rebuild. This allows a restore to succeed when a source pool is full or otherwise unable to host a clone. -- **Experimental RWX Block Volume Support for KubeVirt Live Migration** +- **DiskPool Handle Rescanning** - Replicated PV Mayastor now provides experimental support for native ReadWriteMany (RWX) block volumes to enable KubeVirt Virtual Machine (VM) live migration without requiring an intermediary NFS layer. This capability allows KubeVirt workloads to migrate between nodes while maintaining access to shared block storage. This feature is intended for evaluation and testing in non-production environments. + Replicated PV Mayastor now periodically rescans DiskPool backing device file handles to detect hot-removal on devices that require I/O to surface a removal event, such as those using the AIO and io_uring backends. Previously, removal of an idle device could go undetected. The rescan also refreshes the reported disk size, and both the behaviour and its interval are configurable through the Helm chart. -- **RDMA QoS and DSCP Marking Support** +- **DiskPool Error Clearing** - Replicated PV Mayastor now supports configuring transport-level Quality of Service (QoS) settings for RDMA connections through DSCP marking. This enables integration with network QoS policies and allows administrators to prioritize storage traffic in RDMA-enabled environments. + Building on the pool error and alert visibility introduced in the previous release, you can now clear recorded DiskPool errors using the Mayastor kubectl plugin, with the option to reset selected error categories rather than all of them. This allows a pool to be returned to normal reporting after the underlying condition has been resolved. ### Local Storage -- **Node Deployment Mode for Local PV Hostpath** +- **API Server for Local PV Rawfile** + + Local PV Rawfile now includes an API server with an OpenAPI specification and a Swagger UI, providing a documented interface for inspecting and interacting with the provisioner. + +- **Dataset Tuning Parameters for Local PV ZFS** - Local PV Hostpath now supports a node deployment mode for provisioning operations. This deployment model is designed for high-performance environments and helps reduce provisioning overhead by running provisioning workloads closer to the target node. + Local PV ZFS StorageClasses now support the `atime` and `logbias` parameters, giving you direct control over access-time updates and write-workload optimisation on the underlying ZFS datasets and volumes. -- **Quality of Service (QoS) Support for Local PV LVM** +- **Topology-Constrained StorageClasses for Local PV Hostpath** - Local PV LVM now supports Quality of Service (QoS) controls through Kubernetes VolumeAttributesClass (VAC) resources. Administrators can define and dynamically update storage performance policies, including IOPS and bandwidth limits, without recreating PersistentVolumeClaims (PVCs). This capability enables predictable storage performance, simplifies resource governance, and provides greater flexibility for managing stateful workloads. + The Local PV Hostpath Helm chart now allows you to set `allowedTopologies` on the provisioned StorageClass, so volume placement can be restricted to a defined set of nodes or zones directly from chart values. ## Enhancements ### General -**Global Helm Values Support** +- **Analytics Configuration Overrides** - Support for global Helm values has been added, simplifying configuration management and enabling more consistent deployment settings across OpenEBS components. + The OpenEBS Helm charts now accept global overrides for the analytics identifier and key values, making analytics configuration consistent across the Replicated PV Mayastor, Local PV Hostpath, Local PV LVM, Local PV ZFS, and Local PV Rawfile charts. ### Replicated Storage -- **Expanded Storage Observability** +- **Expanded NVMe-oF Transport Tuning** - New metrics are available for DiskPool capacity, maximum expandable capacity, pool health alerts, replica counts, snapshot counts, and node status. These additions provide deeper visibility into storage utilization, cluster health, and operational status. + Additional NVMe-oF transport parameters are now exposed through the Helm chart for both TCP and RDMA, including maximum I/O size, I/O unit size, in-capsule data size, and the RDMA data work-request pool size. These options allow the transport to be tuned for specific network and workload characteristics. -- **Node Shutdown State Awareness** +- **NetworkPolicy for the REST API** - Replicated PV Mayastor now exposes graceful node shutdown status through its APIs, enabling more accurate node state visibility and troubleshooting. + The Helm chart now creates a Kubernetes NetworkPolicy for the REST API service that blocks plain HTTP traffic to the API. This is enabled by default and can be disabled through `security.networkPolicy.enabled`. -- **SPDK Interrupt Mode Support** +- **Clearer Impact Reporting for Purge Operations** - Replicated PV Mayastor now supports SPDK interrupt mode and associated configuration options, providing an alternative I/O processing model that can help reduce CPU utilization in suitable environments. + Node and DiskPool purge operations that involve data loss now list the affected volumes and snapshots directly in the reported error, instead of requiring a separate `--show-impact` run. Snapshot impact is now included alongside volume impact. + +- **Pool Identification on Replica Metrics** + + Replica metrics now carry `poolname` and `pooluuid` labels, making it possible to attribute replica-level metrics to a specific DiskPool without additional correlation. ### Local Storage -- **Configurable Worker Threads and Helper Pod Timeout** +- **HTTP Health Probe for Local PV Hostpath** + + The Local PV Hostpath provisioner now serves a dedicated HTTP health endpoint, replacing the previous process-based liveness check. This provides a more accurate signal of provisioner health to Kubernetes. + +- **Configurable Kubernetes API Client Rate Limits for Local PV Hostpath** + + The Kubernetes API client QPS and burst values used by the Local PV Hostpath provisioner are now configurable, allowing provisioning throughput to be tuned in large clusters. + +- **Configurable Helper Pod Image Pull Policy for Local PV Hostpath** - Local PV Hostpath now allows administrators to configure worker thread counts and helper pod timeout values. This provides greater control over provisioning behavior and enables tuning for different workload and cluster environments. + The image pull policy for the Local PV Hostpath helper pod can now be set through the Helm chart, providing more control in air-gapped and locally mirrored registry environments. -- **Configurable DNS Policy for Local PV ZFS Node Components** +- **Updated CSI Snapshot Components for Local PV LVM** - Local PV ZFS now allows administrators to configure the Kubernetes `dnsPolicy` for ZFS node components through the Helm chart, providing greater flexibility when deploying in customized networking environments. + The bundled `csi-snapshotter` and `snapshot-controller` components have been updated to v8.2.0. ## Fixes ### Replicated Storage -- **RWX Block Volume Migration Stability** +- **Volume Expansion with Undersized Replicas** - Resolved issues that could cause repeated unpublish and republish operations during RWX block volume migrations, improving migration reliability and reducing disruption during failover events. + Resolved an issue where a volume expansion could fail to complete when one or more replicas had not yet been resized. Undersized replicas are now resized before the nexus resize is retried. -- **DiskPool Cleanup After Device Removal** +- **Pool Availability During Device Removal and Replica Deletion** - Resolved issues affecting DiskPool cleanup and recovery when underlying storage devices were unexpectedly removed or became unavailable. + Resolved several issues affecting DiskPool availability during device and replica lifecycle events, including a race between pool deregistration and reload, and a race when listing pools while replicas were being destroyed. -- **Storage Scheduling Reliability** +- **RDMA Capability Detection on Nodes** - Resolved an issue where replicas could be scheduled on pools in a critical state. Scheduling now correctly avoids unhealthy pools. + Resolved an issue where a node could be treated as RDMA-capable without the required kernel support. The CSI node driver now checks for the `nvme_rdma` kernel module before reporting RDMA capability. -- **Node Unpublish Reliability** +- **Frozen I/O During Nexus Shutdown** - Resolved an issue where node unpublish operations could fail when the target path existed as an empty file. + Resolved an issue where I/O could remain frozen when a shutdown nexus was unshared. Outstanding I/O is now aborted correctly, and devices are no longer detached on I/O submission errors. -- **Cross-Filesystem Restore Validation** +### Local Storage - Resolved an issue where restore operations could proceed between incompatible filesystem types. Restore requests are now validated to prevent unsupported cross-filesystem restores. +- **Idempotent Volume Expansion for Local PV LVM** -### Local Storage + Resolved an issue where repeating a volume expansion could behave inconsistently. Node expansion now remains required during resize, so repeated expand operations are idempotent. + +- **XFS Project Quota Cleanup for Local PV Hostpath** + + Resolved an issue where volume cleanup on XFS could attempt to reset the quota project on named pipes, causing cleanup to fail. + +- **File Permissions in Node Deployment Mode for Local PV Hostpath** + + Resolved an issue where the configured file permissions mode was not applied to provisioned volumes when the provisioner ran in node deployment mode. Additional volume manager fixes for node deployment mode are also included. -- **Capacity Reporting for Thin-Provisioned Volumes for Local PV LVM** +- **Volume Expansion for Local PV Rawfile** - Resolved an issue where available capacity calculations for thin-provisioned storage could be inaccurate. Capacity reporting now correctly considers thin pool free space when determining available storage. + Resolved an issue where volume expansion could fail because of unreliable mount output parsing. Mount information is now resolved using `findmnt`. + +- **ServiceMonitor Manifest Rendering for Local PV LVM** + + Resolved an issue where the namespace field in the generated ServiceMonitor manifest was incorrectly indented. + +## Breaking Changes + +### Local Storage -- **Improved ZFS Error Reporting** +- **Deprecated Local PV Rawfile Helm Values Removed** - Resolved issues with error reporting by providing more detailed ZFS error messages directly from the underlying system. This simplifies troubleshooting and improves visibility into storage-related failures. + The top-level `dataDirPath` and `reservedCapacity` Helm chart values have been removed. Use the equivalent storage pool specific values instead. -- **Graceful Filesystem Shutdown Handling for Local PV ZFS** +- **Filesystem-Level Snapshots Removed from Local PV Rawfile** - Resolved issues affecting volume publish and unpublish operations when a filesystem entered a shutdown state. These improvements help ensure more reliable volume lifecycle operations during filesystem failure scenarios. + Filesystem-level (Btrfs) snapshot support has been removed. Existing snapshots are not deleted, but they are no longer accessible after the upgrade. Remove any filesystem-level snapshots before upgrading. ## Known Issues @@ -161,7 +199,7 @@ This issue is not caused by Mayastor but is triggered more frequently because of ## Related Information -OpenEBS Release notes are maintained in the GitHub repositories alongside the code and releases. For release summaries and full version-level notes, see [OpenEBS Release 4.5](https://github.com/openebs/openebs/releases). +OpenEBS Release notes are maintained in the GitHub repositories alongside the code and releases. For release summaries and full version-level notes, see [OpenEBS Release 4.6](https://github.com/openebs/openebs/releases). See version specific Releases to view the legacy OpenEBS Releases. @@ -173,4 +211,4 @@ See version specific Releases to view the legacy OpenEBS Releases. - [OpenEBS Local Storage](./concepts/data-engines/local-storage.md) - [OpenEBS Replicated Storage](./concepts/data-engines/replicated-storage.md) - [Community](community.md) -- [Commercial Support](commercial-support.md) \ No newline at end of file +- [Commercial Support](commercial-support.md) From 7b41b463438198245bcdb943215527870b6c4b00 Mon Sep 17 00:00:00 2001 From: Bala Harish A Date: Mon, 24 Aug 2026 14:57:01 +0530 Subject: [PATCH 2/9] docs: update zfs docs Signed-off-by: Bala Harish A --- docs/main/quickstart-guide/installation.md | 2 +- docs/main/releases.md | 4 +- .../zfs-raw-block-volume.md | 4 + .../advanced-operations/zfs-resize.md | 2 + .../configuration/zfs-create-storageclass.md | 13 +- .../zfs-storageclass-parameters.md | 174 ++++++++++++++++-- .../local-pv-zfs/configuration/zfs-usage.md | 2 + docs/main/user-guides/upgrades.md | 14 +- docs/versions.json | 1 + 9 files changed, 193 insertions(+), 23 deletions(-) diff --git a/docs/main/quickstart-guide/installation.md b/docs/main/quickstart-guide/installation.md index c3945bcb6..65050ae17 100644 --- a/docs/main/quickstart-guide/installation.md +++ b/docs/main/quickstart-guide/installation.md @@ -112,7 +112,7 @@ Verify helm is installed and helm repo is updated. You need helm 3.2 or more. ``` NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION - openebs openebs 1 2026-06-06 06:18:45.787893124 +0000 UTC deployed openebs-4.5.1 4.5.1 + openebs openebs 1 2026-08-28 06:18:45.787893124 +0000 UTC deployed openebs-4.6.0 4.6.0 ``` ## Verifying OpenEBS Installation diff --git a/docs/main/releases.md b/docs/main/releases.md index 2cbf8d8ce..76a6c46ac 100644 --- a/docs/main/releases.md +++ b/docs/main/releases.md @@ -52,7 +52,7 @@ The status of the various components as of v4.6 are as follows: - **DiskPool Error Clearing** - Building on the pool error and alert visibility introduced in the previous release, you can now clear recorded DiskPool errors using the Mayastor kubectl plugin, with the option to reset selected error categories rather than all of them. This allows a pool to be returned to normal reporting after the underlying condition has been resolved. + Building on the pool error and alert visibility introduced in the previous release, you can now clear recorded DiskPool errors using the Mayastor kubectl plugin. This allows a pool to be returned to normal reporting after the underlying condition has been resolved. ### Local Storage @@ -92,7 +92,7 @@ The status of the various components as of v4.6 are as follows: - **Pool Identification on Replica Metrics** - Replica metrics now carry `poolname` and `pooluuid` labels, making it possible to attribute replica-level metrics to a specific DiskPool without additional correlation. + Replica metrics now carry `pool_name` and `pool_uuid` labels, making it possible to attribute replica-level metrics to a specific DiskPool without additional correlation. ### Local Storage diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-raw-block-volume.md b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-raw-block-volume.md index d7968900d..706b67847 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-raw-block-volume.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-raw-block-volume.md @@ -14,6 +14,10 @@ As it becomes more common to run database software and storage infrastructure so To provision the raw block volume, we should create a storageclass without any fstype as Raw block volume does not have any fstype. +:::note +Since a raw block volume is attached to the pod as a block device instead of being mounted with a filesystem, the [mountOptions](../configuration/zfs-storageclass-parameters.md#mountoptions-optional) specified in the storage class are not applied to it. +::: + ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-resize.md b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-resize.md index 7189327c5..31663e8dc 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-resize.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-resize.md @@ -10,6 +10,8 @@ description: This section talks about the advanced operations that can be perfor We can resize the volume by updating the PVC yaml to the desired size and apply it. The ZFS Driver will take care of updating the quota in case of dataset. If we are using a Zvol and have mounted it as ext2/3/4 or xfs file system, the driver will take care of expanding the volume via reize2fs/xfs_growfs binaries. +For a dataset, the property that the driver updates depends on the [quotatype](../configuration/zfs-storageclass-parameters.md#quotatype-optional-parameter) parameter of the storage class. With the default `quota`, the driver updates the `quota` property, and with `refquota`, it updates the `refquota` property. For a thick provisioned dataset, the corresponding `reservation` or `refreservation` property is updated as well. + For resize, storageclass that provisions the pvc must support resize. We should have allowVolumeExpansion as true in storageclass. :::note diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-create-storageclass.md b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-create-storageclass.md index b4b6c2dee..f072c6294 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-create-storageclass.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-create-storageclass.md @@ -124,7 +124,18 @@ The provisioner name for ZFS driver is "zfs.csi.openebs.io", we have to use this ## Scheduler -The ZFS driver has its own scheduler which will try to distribute the PV across the nodes so that one node should not be loaded with all the volumes. Currently the driver supports two scheduling algorithms: VolumeWeighted and CapacityWeighted, in which it will try to find a ZFS pool which has less number of volumes provisioned in it or less capacity of volume provisioned out of a pool respectively, from all the nodes where the ZFS pools are available. Refer [StorageClass With K8s Scheduler](https://github.com/openebs/zfs-localpv/blob/HEAD/docs/storageclasses.md#storageclass-with-k8s-scheduler) to learn how to select a scheduler via storage class. +The ZFS driver has its own scheduler which will try to distribute the PV across the nodes so that one node should not be loaded with all the volumes. Currently the driver supports two scheduling algorithms: VolumeWeighted and CapacityWeighted, in which it will try to find a ZFS pool which has less number of volumes provisioned in it or less capacity of volume provisioned out of a pool respectively, from all the nodes where the ZFS pools are available. + +To select a scheduler, add the `scheduler` parameter in the storage class and give its value accordingly: + +``` +parameters: + scheduler: "VolumeWeighted" + fstype: "zfs" + poolname: "zfspv-pool" +``` + +If the `scheduler` parameter is not provided in the storage class, `CapacityWeighted` is used. Refer to [StorageClass Parameters](zfs-storageclass-parameters.md#scheduler-optional-parameter) and [StorageClass with k8s Scheduler](zfs-usage.md#storageclass-with-k8s-scheduler) for more details. Once it can find the node, it will create a PV for that node and also create a ZFSVolume custom resource for the volume with the NODE information. The watcher for this ZFSVolume CR will get all the information for this object and creates a ZFS dataset (zvol) with the given ZFS property on the mentioned node. diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-storageclass-parameters.md b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-storageclass-parameters.md index 755db9e57..522b873a7 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-storageclass-parameters.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-storageclass-parameters.md @@ -12,20 +12,46 @@ description: This guide will help you to create Local PV ZFS StorageClass. # StorageClass Parameters -This document outlines the key parameters used in configuring a StorageClass for Local PV ZFS. The document categorizes parameters into mandatory and optional, emphasizing critical settings such as poolname, while detailing optional features like FsType, recordsize, compression, and deduplication. +This document describes the StorageClass fields and parameters supported by Local PV ZFS. It covers the standard Kubernetes StorageClass fields, such as volume expansion, mount options, and binding mode, along with the ZFS-specific parameters that control the properties of the underlying dataset or ZVOL, such as poolname, recordsize, compression, and quotatype. + +## Standard StorageClass Fields + +These fields are set at the top level of the StorageClass, alongside `provisioner`, and not under `parameters`. + +| Field | Values | +|-------|--------| +| `allowVolumeExpansion` | `true`, `false` | +| `mountOptions` | Mount options supported by the filesystem | +| `volumeBindingMode` | `Immediate`, `WaitForFirstConsumer` | +| `reclaimPolicy` | `Delete`, `Retain` | +| `allowedTopologies` | Node label key and the values to match | ## Supported StorageClass Parameters -| Parameter | Requirement | Allowed Values | -|-----------|-------------|----------------| -| `poolname` | Required | Existing ZFS pool or child dataset (for example, `zfspv-pool`, `zfspv-pool/child`) | -| `fstype` | Optional | `zfs`, `ext2`, `ext3`, `ext4`, `xfs`, `btrfs` | -| `recordsize` | Optional | Any power of 2 from 512 bytes to 128 KiB | -| `volblocksize` | Optional | Any power of 2 from 512 bytes to 128 KiB | -| `compression` | Optional | `on`, `off`, `lzjb`, `lz4`, `zle`, `gzip`, `gzip-1` through `gzip-9`, `zstd`, `zstd-1` through `zstd-19` | -| `dedup` | Optional | `on`, `off` | -| `thinProvision` | Optional | `yes`, `no` | -| `shared` | Optional | `yes`, `no` | +These parameters are set under `parameters` in the StorageClass. + +| Parameter | Requirement | Allowed Values | Applies To | +|-----------|-------------|----------------|------------| +| `poolname` | Required | Existing ZFS pool or child dataset (for example, `zfspv-pool`, `zfspv-pool/child`) | Both | +| `fstype` | Optional | `zfs`, `ext2`, `ext3`, `ext4`, `xfs`, `btrfs` | Both | +| `recordsize` | Optional | Any power of 2 from 512 bytes to 128 KiB | Dataset | +| `volblocksize` | Optional | Any power of 2 from 512 bytes to 128 KiB | ZVOL | +| `compression` | Optional | `on`, `off`, `lzjb`, `lz4`, `zle`, `gzip`, `gzip-1` through `gzip-9`, `zstd`, `zstd-fast`, `zstd-1` through `zstd-19` | Both | +| `dedup` | Optional | `on`, `off` | Both | +| `atime` | Optional | `on`, `off` | Dataset | +| `logbias` | Optional | `latency`, `throughput` | Both | +| `thinProvision` | Optional | `yes`, `no` | Both | +| `quotatype` | Optional | `quota`, `refquota` | Dataset | +| `shared` | Optional | `yes`, `no` | Both | +| `scheduler` | Optional | `CapacityWeighted`, `VolumeWeighted` | Both | + +In the **Applies To** column: + +- **Dataset** means that the parameter takes effect only when `fstype` is "zfs". +- **ZVOL** means that the parameter takes effect only when `fstype` is a filesystem other than "zfs". +- **Both** means that the parameter takes effect in either case. + +A parameter that does not apply to the type of volume being created is ignored. ## Poolname (Must Parameter) @@ -61,14 +87,138 @@ Compression specifies the block-level compression algorithm to be applied to the Deduplication is the process for removing redundant data at the block level, reducing the total amount of data stored. +## Atime (Optional Parameter) + +Atime controls whether the access time of a file is updated when the file is read. Setting it to "off" avoids the write traffic that is otherwise generated by reading files, which can improve performance for read-heavy workloads. + +This parameter is applicable if fstype provided is "zfs". For any other fstype the driver creates a ZVOL, where atime does not apply, and the value is ignored. + +If atime is not provided in the storageclass, the volume inherits the value from the parent ZFS pool or dataset. + +## Logbias (Optional Parameter) + +Logbias provides a hint to ZFS about how to handle synchronous requests for the volume. With "latency", ZFS uses the separate log devices (SLOG) of the pool, if any, to handle these requests at low latency. With "throughput", ZFS does not use the separate log devices and instead optimizes synchronous operations for overall pool throughput. + +This parameter applies to both ZFS datasets and ZVOLs. + +If logbias is not provided in the storageclass, the volume inherits the value from the parent ZFS pool or dataset. + ## Thinprovision (Optional Parameter) ThinProvision describes whether space reservation for the source volume is required or not. The value "yes" indicates that volume should be thin provisioned and "no" means thick provisioning of the volume. If thinProvision is set to "yes" then volume can be provisioned even if the ZPOOL does not have enough capacity. If thinProvision is set to "no" then volume can be provisioned only if the ZPOOL has enough capacity and capacity required by volume can be reserved. Omitting this parameter lets ZFS default behavior prevail: `thin` provisioning for filesystems and `thick` provisioning (through refreservation) for volumes. +## Quotatype (Optional Parameter) + +Quotatype selects the ZFS property that is used to enforce the size of the volume. With "quota", the limit applies to the dataset together with everything it contains, including its snapshots and clones. With "refquota", the limit applies only to the data that the dataset itself references, so snapshots and clones are not counted against it. + +Quotatype also determines the property that is used to reserve space when `thinProvision` is set to "no". With "quota", the space is reserved using `reservation`, and with "refquota", it is reserved using `refreservation`. + +This parameter is applicable if fstype provided is "zfs" otherwise it will be ignored. Quotatype can not be modified once volume has been provisioned. If quotatype is not provided in the storageclass, the driver uses "quota". + ## Shared (Optional Parameter) Shared specifies whether the volume can be shared among multiple pods. If it is not set to "yes", then the ZFS-LocalPV Driver will not allow the volumes to be mounted by more than one pods. The default value is "no" if shared is not provided in the storageclass. +## Scheduler (Optional Parameter) + +Scheduler selects the algorithm that the Local PV ZFS driver uses to pick the node on which a volume is provisioned: + +- `CapacityWeighted` picks the node whose matching ZFS pool has the least capacity already provisioned from it. +- `VolumeWeighted` picks the node whose matching ZFS pool has the fewest volumes provisioned on it. + +If scheduler is not provided in the storageclass, `CapacityWeighted` is used. Refer to [StorageClass with k8s Scheduler](zfs-usage.md#storageclass-with-k8s-scheduler) for more details. + +## AllowVolumeExpansion (Optional) + +Volumes provisioned by a StorageClass can be expanded only when `allowVolumeExpansion` is set to `true`. If this field is not specified, volume expansion is not supported. + +Local PV ZFS supports online volume expansion, which means that the application does not need to be scaled down for the volume to be resized. Refer to [Resize](../advanced-operations/zfs-resize.md) for more details. + +:::note +btrfs does not support online volume resize, so we can not resize the btrfs volumes. +::: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: openebs-zfspv +allowVolumeExpansion: true +parameters: + fstype: "zfs" + poolname: "zfspv-pool" +provisioner: zfs.csi.openebs.io +``` + +## MountOptions (Optional) + +Volumes provisioned by Local PV ZFS are mounted using the mount options specified in the storageclass. + +Mount options apply to volumes that are mounted with a filesystem, which covers both ZFS datasets and formatted ZVOLs. They are not applied to [raw block volumes](../advanced-operations/zfs-raw-block-volume.md), because a raw block volume is attached to the pod as a block device instead of being mounted with a filesystem. + +:::note +Mount options are not validated. If the mount options are invalid, then the volume mount fails. +::: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: openebs-zfspv +parameters: + fstype: "zfs" + poolname: "zfspv-pool" +provisioner: zfs.csi.openebs.io +mountOptions: + - nosuid +``` + +## VolumeBindingMode (Optional) + +Local PV ZFS supports both the volume binding modes that are `Immediate` and `WaitForFirstConsumer`. + +- `Immediate` indicates that volume binding and dynamic provisioning occur once the PersistentVolumeClaim is created. +- `WaitForFirstConsumer` is also known as late binding, which delays the binding and provisioning of a PersistentVolumeClaim until a pod using it is created. + +Use `WaitForFirstConsumer` when the application pod has node selector or affinity rules, or CPU and memory constraints. In this case, Kubernetes schedules the pod first and the driver then provisions the volume on the selected node. + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: openebs-zfspv +parameters: + fstype: "zfs" + poolname: "zfspv-pool" +provisioner: zfs.csi.openebs.io +volumeBindingMode: WaitForFirstConsumer +``` + +## ReclaimPolicy (Optional) + +Local PV ZFS supports both the reclaim policies that are `Delete` and `Retain`. If it is not specified, it defaults to `Delete`. + +- `Delete` indicates that the backend volume resources are deleted along with the PersistentVolumeClaim. +- `Retain` indicates that the backend volume resources are retained in the cluster after the PersistentVolumeClaim is deleted. + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: openebs-zfspv +parameters: + fstype: "zfs" + poolname: "zfspv-pool" +provisioner: zfs.csi.openebs.io +reclaimPolicy: Delete +``` + +## AllowedTopologies (Optional) + +If the ZFS pool is available on certain nodes only, use `allowedTopologies` to list the nodes where the pool is present. The driver will create volumes on those nodes only. + +Refer to [ZFS Pool Availability](zfs-create-storageclass.md#zfs-pool-availability) and [StorageClass with Custom Node Labels](zfs-usage.md#storageclass-with-custom-node-labels) for more details and examples. + ## Support If you encounter issues or have a question, file a [Github issue](https://github.com/openebs/openebs/issues/new), or talk to us on the [#openebs channel on the Kubernetes Slack server](https://kubernetes.slack.com/messages/openebs/). @@ -79,4 +229,4 @@ If you encounter issues or have a question, file a [Github issue](https://github - [Create StorageClass(s)](zfs-create-storageclass.md) - [Create PVC](zfs-create-pvc.md) - [Usage](zfs-usage.md) -- [Deploy an Application](zfs-deployment.md) \ No newline at end of file +- [Deploy an Application](zfs-deployment.md) diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-usage.md b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-usage.md index 7083ee61f..59ea90e57 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-usage.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-usage.md @@ -112,6 +112,8 @@ parameters: CapacityWeighted is the default scheduler in zfs-localpv driver, so even if we don't use scheduler parameter in storage-class, driver will pick the node where total provisioned volumes have occupied less capacity from the given pool. On the other hand for using VolumeWeighted scheduler, we have to specify it under scheduler parameter in storage-class. Then driver will pick the node to create volume where ZFS Pool is less loaded with the volumes. Here, it just checks the volume count and creates the volume where less volume is configured in a given ZFS Pool. It does not account for other factors like available CPU or memory while making scheduling decisions. +Refer to [StorageClass Parameters](zfs-storageclass-parameters.md#scheduler-optional-parameter) for the supported values of the `scheduler` parameter. + In case where you want to use node selector/affinity rules on the application pod or have CPU/Memory constraints, the Kubernetes scheduler should be used. To make use of Kubernetes scheduler, we can set the volumeBindingMode as WaitForFirstConsumer in the storage class: ```yaml diff --git a/docs/main/user-guides/upgrades.md b/docs/main/user-guides/upgrades.md index d6d681cc4..7926a79f6 100644 --- a/docs/main/user-guides/upgrades.md +++ b/docs/main/user-guides/upgrades.md @@ -10,9 +10,9 @@ description: Upgrade to the latest OpenEBS version. ## Overview -This document describes the procedure for upgrading an existing OpenEBS deployment to OpenEBS 4.5. +This document describes the procedure for upgrading an existing OpenEBS deployment to OpenEBS 4.6. -This upgrade process allows you to upgrade to the latest OpenEBS version 4.5 which is a unified installer for three Local Storages (a.k.a Local Engines): +This upgrade process allows you to upgrade to the latest OpenEBS version 4.6 which is a unified installer for three Local Storages (a.k.a Local Engines): - Local PV HostPath - Local PV LVM - Local PV ZFS @@ -21,7 +21,7 @@ This upgrade process allows you to upgrade to the latest OpenEBS version 4.5 whi and one Replicated Storage (a.k.a Replicated Engine): - Replicated PV Mayastor -As a part of the upgrade to OpenEBS 4.5, the Helm chart will install all four engines regardless of the engine you used before the upgrade. +As a part of the upgrade to OpenEBS 4.6, the Helm chart will install all four engines regardless of the engine you used before the upgrade. :::info During the upgrade, if you are only interested in Local PV Storage, you can disable Replicated PV Mayastor by using the below option: @@ -49,9 +49,9 @@ Refer to the [Migration documentation](../user-guides/data-migration/migration-o - When deploying with the OpenEBS Helm chart, use the `kubectl openebs upgrade` command to upgrade all storages. -## Upgrade from 3.x to 4.5 +## Upgrade from 3.x to 4.6 -Follow these steps to upgrade OpenEBS from version 3.x to 4.5: +Follow these steps to upgrade OpenEBS from version 3.x to 4.6: 1. Update the helm repository: The OpenEBS Helm chart repository URL has changed. The repository target URL needs to be updated. @@ -86,9 +86,9 @@ Follow these steps to upgrade OpenEBS from version 3.x to 4.5: ::: -## Upgrade from 4.x to 4.5 +## Upgrade from 4.x to 4.6 -Follow these steps to upgrade OpenEBS from version 4.x to 4.5: +Follow these steps to upgrade OpenEBS from version 4.x to 4.6: 1. Download the `kubectl openebs` binary from the [OpenEBS Release repository](https://github.com/openebs/openebs/releases) on GitHub. diff --git a/docs/versions.json b/docs/versions.json index 2b362fac4..2fd22d723 100644 --- a/docs/versions.json +++ b/docs/versions.json @@ -1,4 +1,5 @@ [ + "4.6.x", "4.5.x", "4.4.x", "4.3.x", From 1e518efe7a633be31a4c12d92783caf36ea19612 Mon Sep 17 00:00:00 2001 From: Bala Harish A Date: Mon, 24 Aug 2026 15:39:53 +0530 Subject: [PATCH 3/9] docs: update docs Signed-off-by: Bala Harish A --- .../hostpath-create-storageclass.md | 53 ++++++++++++++++++- .../advanced-operations/lvm-snapshot.md | 9 ++++ .../advanced-operations/rawfile-snapshot.md | 21 ++++++++ .../advanced-operations/zfs-snapshot.md | 20 +++++++ .../advanced-operations/delete-nodes.md | 4 ++ .../diskpool-io-failure-handling.md | 11 ++++ .../advanced-operations/snapshot-restore.md | 6 +++ .../advanced-operations/snapshot.md | 6 ++- .../rs-storage-class-parameters.md | 12 +++++ 9 files changed, 139 insertions(+), 3 deletions(-) diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-storageclass.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-storageclass.md index 77cbec038..19feebc30 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-storageclass.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-storageclass.md @@ -60,7 +60,7 @@ The default Storage Class is called `openebs-hostpath` and its `BasePath` is con ``` :::note - Using NodeAffinityLabels does not influence the scheduling of the application Pod. Use Kubernetes [Allowed Topologies](https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/tutorials/hostpath/allowedtopologies.md) to configure scheduling options. + Using NodeAffinityLabels does not influence the scheduling of the application Pod. Use Kubernetes [Allowed Topologies](#restrict-volume-placement-using-allowed-topologies) to configure scheduling options. ::: 2. Edit `local-hostpath-sc.yaml` and update with your desired values for `metadata.name` and `cas.openebs.io/config.BasePath`. @@ -78,7 +78,56 @@ The default Storage Class is called `openebs-hostpath` and its `BasePath` is con ``` kubectl get sc local-hostpath -o yaml ``` - + +## Restrict Volume Placement Using Allowed Topologies + +By default, a Local PV Hostpath volume can be provisioned on any node in the cluster. If the `BasePath` is present on certain nodes only, then make use of topology to tell the list of nodes where the path is available. As shown in the below storage class, we can use `allowedTopologies` to provision volumes on the specified nodes only. + +``` +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: local-hostpath + annotations: + openebs.io/cas-type: local + cas.openebs.io/config: | + - name: StorageType + value: "hostpath" + - name: BasePath + value: "/var/local-hostpath" +provisioner: openebs.io/local +volumeBindingMode: WaitForFirstConsumer +allowedTopologies: +- matchLabelExpressions: + - key: kubernetes.io/hostname + values: + - worker-2 + - worker-3 +``` + +The above storage class tells that the `BasePath` is available on nodes worker-2 and worker-3 only. Volumes of this StorageClass will be provisioned and scheduled on those nodes only. + +Unlike `NodeAffinityLabels`, `allowedTopologies` also influences the scheduling of the application Pod. + +To set `allowedTopologies` on the `openebs-hostpath` StorageClass created by the Helm chart, use the `localpv-provisioner.hostpathClass.allowedTopologies` value: + +```yaml +localpv-provisioner: + hostpathClass: + allowedTopologies: + - matchLabelExpressions: + - key: kubernetes.io/hostname + values: + - worker-2 + - worker-3 +``` + +Apply the updated values: + +```bash +helm upgrade openebs openebs/openebs --namespace openebs -f values.yaml +``` + ## Support If you encounter issues or have a question, file a [Github issue](https://github.com/openebs/openebs/issues/new), or talk to us on the [#openebs channel on the Kubernetes Slack server](https://kubernetes.slack.com/messages/openebs/). diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md index 8da2acc14..6b5292f09 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md @@ -14,6 +14,15 @@ The LVM driver supports creating snapshots of the LVM volumes. This requires the - Snapshots created by the LVM driver are ReadOnly by default as opposed to the ReadWrite snapshots created by default by `lvcreate` command. - The size of snapshot will be set to the size of the origin volume. +## Using an Existing Snapshot Controller + +Local PV LVM runs a `snapshot-controller` container by default. A cluster should run only one snapshot controller, so if one is already managed at the cluster level, disable the bundled one: + +```bash +helm upgrade openebs openebs/openebs \ + --namespace openebs \ + --set lvm-localpv.lvmController.snapshotController.enabled=false +``` ## Default SnapshotClass without SnapSize Parameter diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-rawfile/advanced-operations/rawfile-snapshot.md b/docs/main/user-guides/local-storage-user-guide/local-pv-rawfile/advanced-operations/rawfile-snapshot.md index 94a32ba2b..4587b1690 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-rawfile/advanced-operations/rawfile-snapshot.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-rawfile/advanced-operations/rawfile-snapshot.md @@ -17,6 +17,27 @@ Local PV Rawfile supports block-level volume snapshots using the standard Kubern - Snapshots must be enabled in the chart (default): `capabilities.snapshots.enabled=true` - Kubernetes VolumeSnapshot CRDs must be installed in the cluster - A `VolumeSnapshotClass` must exist for the rawfile driver +- A snapshot controller must be running in the cluster. Local PV Rawfile deploys one by default. Refer to [Using an Existing Snapshot Controller](#using-an-existing-snapshot-controller) if the cluster already manages its own. + +## Using an Existing Snapshot Controller + +Local PV Rawfile runs a `snapshot-controller` container by default. A cluster should run only one snapshot controller, so if one is already managed at the cluster level, disable the bundled one: + +```yaml +node: + snapshotController: + enabled: false +``` + +Apply the updated values: + +```bash +helm upgrade rawfile-localpv rawfile-localpv/rawfile-localpv -n openebs -f values.yaml +``` + +:::warning +The snapshot controller that remains in the cluster must be started with `--enable-distributed-snapshotting=true`. Without this flag, Local PV Rawfile snapshots are never provisioned, and no error is reported at install time. +::: ## Create a VolumeSnapshotClass diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-snapshot.md b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-snapshot.md index 486f215bc..fd72949d4 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-snapshot.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/advanced-operations/zfs-snapshot.md @@ -10,6 +10,18 @@ description: This section talks about the advanced operations that can be perfor We can create a snapshot of a volume that can be used further for creating a clone and for taking a backup. To create a snapshot, we have to first create a SnapshotClass just like a storage class where you can provide deletionPolicy as Retain or Delete. +## Using an Existing Snapshot Controller + +Local PV ZFS runs a `snapshot-controller` container by default. A cluster should run only one snapshot controller, so if one is already managed at the cluster level, disable the bundled one: + +```bash +helm upgrade openebs openebs/openebs \ + --namespace openebs \ + --set zfs-localpv.zfsController.snapshotController.enabled=false +``` + +## Create a VolumeSnapshotClass + ```yaml $ cat snapshotclass.yaml kind: VolumeSnapshotClass @@ -29,6 +41,8 @@ $ kubectl apply -f snapshotclass.yaml volumesnapshotclass.snapshot.storage.k8s.io/zfspv-snapclass created ``` +## Create a Snapshot + Find a PVC for which snapshot has to be created. ``` @@ -62,6 +76,8 @@ volumesnapshot.snapshot.storage.k8s.io/zfspv-snap created Create the snapshot in the same namespace where the PVC is created. Check the created snapshot resource, make sure readyToUsefield is true, before using this snapshot for any purpose. ::: +## Verify the Snapshot + ``` $ kubectl get volumesnapshot.snapshot NAME AGE @@ -98,6 +114,8 @@ status: restoreSize: "0" ``` +### Check the ZFSSnapshot Resource + Check the OpenEBS resource for the created snapshot. The status should be "Ready". ``` @@ -132,6 +150,8 @@ status: state: Ready ``` +### Verify the Snapshot on the Node + We can go to the node and confirm that snapshot has been created: ``` diff --git a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/delete-nodes.md b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/delete-nodes.md index 62e923225..c0ad8d9ae 100644 --- a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/delete-nodes.md +++ b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/delete-nodes.md @@ -101,6 +101,10 @@ NODE STATUS REPLICAS VOLUMES READY node-0-469923 Unknown 1 1 true ``` +:::note +When a purge operation would result in data loss, the affected volumes and snapshots are also listed in the reported error. Running `--show-impact` beforehand is therefore no longer required to identify them, though it remains useful as an explicit pre-check. +::: + ## Purge a Node Use the following command to purge an irrecoverable node from the Replicated PV Mayastor control plane. diff --git a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/diskpool-io-failure-handling.md b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/diskpool-io-failure-handling.md index 97cac5d0b..19d329ff1 100644 --- a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/diskpool-io-failure-handling.md +++ b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/diskpool-io-failure-handling.md @@ -59,6 +59,17 @@ As part of this process: The control plane probes the io-engine to detect whether the backing device has been reattached to the node. If the device becomes available again, Replicated PV Mayastor import the pool and restore normal operations. +### Hot-Removal Detection on Idle Devices + +Devices accessed over PCIe report hot-removal events directly, so the affected pool is unloaded as soon as the device is detached. Devices accessed through `aio` or `io_uring` surface a removal only once I/O is submitted to them, which means the removal of an idle device can go undetected. + +To handle this, Replicated PV Mayastor periodically rescans the file handles of the DiskPool backing devices and triggers a hot-removal event when a handle is no longer valid. The rescan also refreshes the reported disk size. + +| Parameter | Default Value | Description | Helm Path | +| :--- | :--- | :--- | :--- | +| `enabled` | `true` | Enables periodic rescanning of DiskPool backing device file handles | `mayastor.io_engine.pool.diskHandleRescan.enabled` | +| `period` | `5m` | Interval between successive rescans | `mayastor.io_engine.pool.diskHandleRescan.period` | + ## DiskPool I/O Error Handling Replicated PV Mayastor tracks runtime disk I/O errors and exposes alert information for affected DiskPools. diff --git a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/snapshot-restore.md b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/snapshot-restore.md index 3867cdd91..3604a27f2 100644 --- a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/snapshot-restore.md +++ b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/snapshot-restore.md @@ -108,3 +108,9 @@ spec: ``` By running this command, you create a new PVC named `restore-pvc` based on the specified snapshot. The restored volume will have the same data and configuration as the original volume had at the time of the snapshot. + +## Restore Policy + +By default, a restore requires every replica of the new volume to be cloned from the source snapshot. If any of the snapshot's replica pools cannot host a clone, for example because a source pool has run out of space, the restore fails even though the data is available in another reachable pool. + +To allow the restore to proceed in this situation, set the `snapshotRestorePolicy` parameter to `bestEffort` in the StorageClass used for the restore. The restore then succeeds as long as at least one clone succeeds, and the remaining replicas are created through a normal rebuild. Refer to the [Storage Class Parameters](../configuration/rs-storage-class-parameters.md) documentation for more details. diff --git a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/snapshot.md b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/snapshot.md index 2e7de2c66..e15691240 100644 --- a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/snapshot.md +++ b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/advanced-operations/snapshot.md @@ -78,7 +78,11 @@ Copy the PVC name, for example, `ms-volume-claim`. ::: Create an application by following the instructions provided in the [Deploy an Application documentation](../configuration/rs-deployment.md). - + +## Using an Existing Snapshot Controller + +Replicated PV Mayastor runs a `csi-snapshot-controller` container by default. A cluster should run only one snapshot controller, so if one is already managed at the cluster level, disable the bundled one by setting `mayastor.csi.controller.snapshotController.enabled` to `false` in the Helm chart values. + ## Create a Snapshot You can create a snapshot (with or without an application) using the PVC. diff --git a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/configuration/rs-storage-class-parameters.md b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/configuration/rs-storage-class-parameters.md index 058725ac4..f2654aac3 100644 --- a/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/configuration/rs-storage-class-parameters.md +++ b/docs/main/user-guides/replicated-storage-user-guide/replicated-pv-mayastor/configuration/rs-storage-class-parameters.md @@ -78,6 +78,18 @@ formatOptions: "-m bigtime=1 -m inobtcount=1" Enables volume encryption when set to `true`. The volume will be provisioned only if the number of available encrypted pools meets or exceeds the `repl` defined in the StorageClass. +## "snapshotRestorePolicy" + +Controls how a volume restore from a snapshot behaves when not every replica pool can host a clone of the source snapshot. The supported values are `strict` and `bestEffort`, and the default when not specified is `strict`. + +With `strict`, every requested replica must be cloned from the snapshot. If any of the snapshot's replica pools cannot host a clone, for example because a source pool has run out of space, the restore fails. + +With `bestEffort`, the restore proceeds as long as at least one clone succeeds. The volume comes up under-replicated, and the remaining replicas are created through a normal rebuild. Refer to the [Snapshot Restore documentation](../advanced-operations/snapshot-restore.md) for more details. + +:::note +A volume restored using `bestEffort` does not tolerate the number of node failures implied by its `repl` value until the outstanding rebuilds are complete. +::: + ## See Also - [Installation](../../../../quickstart-guide/installation.md) From 8e7f5b6fda7d9cf3ac24c7fc501a03567a15c372 Mon Sep 17 00:00:00 2001 From: Bala Harish A Date: Mon, 24 Aug 2026 16:03:34 +0530 Subject: [PATCH 4/9] docs: revert premature 4.6.x entry in versions.json The 4.6.x version was added to versions.json without a corresponding versioned_docs/version-4.6.x directory, which fails the Docusaurus build: The docs folder does not exist for version "4.6.x". A docs folder is expected to be found at versioned_docs/version-4.6.x. Remove the entry until the 4.6.x docs version is actually cut. Signed-off-by: Bala Harish A --- docs/versions.json | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/versions.json b/docs/versions.json index 2fd22d723..2b362fac4 100644 --- a/docs/versions.json +++ b/docs/versions.json @@ -1,5 +1,4 @@ [ - "4.6.x", "4.5.x", "4.4.x", "4.3.x", From 5057053f672b7f454577697ceaf044add7970c4a Mon Sep 17 00:00:00 2001 From: Bala Harish A Date: Tue, 25 Aug 2026 12:14:49 +0530 Subject: [PATCH 5/9] docs: document missing Local PV LVM storageclass, VAC, and snapshot parameters Adds formatOptions and scheduler to the StorageClass parameters page, corrects the field casing and allowed values in the conformance matrix, documents how unified and directional VAC QoS parameters combine, states the snapSize bounds and the cap at the origin volume size, and notes that mount and format options do not apply to raw block volumes. Signed-off-by: Bala Harish A --- .../lvm-raw-block-volume.md | 4 ++ .../advanced-operations/lvm-snapshot.md | 9 +++ .../configuration/lvm-create-storageclass.md | 11 ++-- .../lvm-storageclass-parameters.md | 57 ++++++++++++++++++- .../configuration/lvm-vac-parameters.md | 34 +++++++++++ 5 files changed, 108 insertions(+), 7 deletions(-) diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-raw-block-volume.md b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-raw-block-volume.md index 6d4dbe68f..424338051 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-raw-block-volume.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-raw-block-volume.md @@ -15,6 +15,10 @@ As it becomes more common to run database software and storage infrastructure so To provision the raw block volume, we should create a storageclass without any fstype as Raw block volume does not have any fstype. +:::note +Since a raw block volume is attached to the pod as a block device instead of being mounted with a filesystem, the [mountOptions](../configuration/lvm-storageclass-parameters.md#mountoptions-optional) and [formatOptions](../configuration/lvm-storageclass-parameters.md#formatoptions-optional) specified in the storage class are not applied to it. +::: + ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md index 6b5292f09..10bbc958f 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md @@ -47,6 +47,15 @@ Using the SnapshotClass `snapSize` parameter we can configure the snapshot size In this case, whether the volume is thin provisioned or thick provisioned, the driver will create a thick snapshot with size as `snapSize` mentioned in the `SnapshotClass`. +The accepted values are: + +- **Percentage**, written with a trailing `%`, for example `50%`. The value must be between 1 and 100, and it is applied as a percentage of the capacity of the origin volume. A value outside this range is rejected and the snapshot is not created. +- **Absolute size**, written as a quantity such as `10G` or `500Mi`. The value must be greater than 0. + +:::note +If an absolute `snapSize` is larger than the origin volume, the driver reduces it to the size of the origin volume. The snapshot is created successfully, so the resulting snapshot can be smaller than the value set in the SnapshotClass. +::: + - SnapshotClass with `snapSize` parameter as a percentage(%) value: ```yaml diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-create-storageclass.md b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-create-storageclass.md index cfddbaaec..48d300a15 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-create-storageclass.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-create-storageclass.md @@ -38,9 +38,9 @@ The following matrix shows standard StorageClass parameters for Local PV LVM. | Parameter | Values | | -------------------------------------------- | :---: | | `allowVolumeExpansion` | `true` / `false` | -| `MountOptions` | Filesystem-supported mount options | -| `VolumeBindingMode` | `Immediate` / `WaitForFirstConsumer` | -| `Reclaim Policy` | `Retain`/ `Delete` | +| `mountOptions` | Filesystem-supported mount options | +| `volumeBindingMode` | `Immediate` / `WaitForFirstConsumer` | +| `reclaimPolicy` | `Retain` / `Delete` | | `allowedTopologies` | - | | `fsType` | `ext2`, `ext3`, `ext4`, `xfs` and `btrfs` | @@ -48,10 +48,11 @@ The following matrix shows standard StorageClass parameters for Local PV LVM. | Parameter | Values | |-----------|--------| -| `shared` | `yes` | +| `shared` | `yes` / `no` | | `vgpattern` | Regular expression of the volume group name | | `volgroup` | Volume group name | -| `thinProvision` | `yes` | +| `thinProvision` | `yes` / `no` | +| `formatOptions` | Extra `mkfs` options as a space-separated string | | `scheduler` | `SpaceWeighted`, `CapacityWeighted`, `VolumeWeighted` | ### StorageClass with Scheduler Parameters diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-storageclass-parameters.md b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-storageclass-parameters.md index 95c669302..2926145ff 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-storageclass-parameters.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-storageclass-parameters.md @@ -13,7 +13,7 @@ description: This guide will help you to create Local PV LVM StorageClass. # StorageClass Parameters -This document describes the supported StorageClass parameters for Local PV LVM and explains how to configure them. These parameters allow you to control volume expansion, mount options, filesystem selection, volume sharing, provisioning behavior, topology-aware scheduling, and other storage features. +This document describes the supported StorageClass parameters for Local PV LVM and explains how to configure them. These parameters allow you to control volume expansion, mount options, filesystem selection and formatting, volume sharing, provisioning behavior, node scheduling, topology-aware placement, and other storage features. ## AllowVolumeExpansion (Optional) @@ -36,6 +36,8 @@ Volumes that are provisioned via Local PV LVM will use the mount options specifi :::note Mount options are not validated. If mount options are invalid, then volume mount fails. +::: + ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass @@ -48,7 +50,8 @@ parameters: mountOptions: ## Various mount options of volume can be specified here - debug ``` -::: + +Mount options are not applied to [raw block volumes](../advanced-operations/lvm-raw-block-volume.md), because a raw block volume is attached to the pod as a block device instead of being mounted with a filesystem. ## FsType (Optional) @@ -67,6 +70,32 @@ mountOptions: ## Various mount options of volume can be specified here fsType: xfs ## Supported filesystems are ext2, ext3, ext4, xfs & btrfs ``` +## FormatOptions (Optional) + + Use the `formatOptions` parameter to pass extra options to the `mkfs` command that formats the volume with the filesystem specified by `fsType`. Provide the options as a single space-separated string. + + ```yaml + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: openebs-lvm + provisioner: local.csi.openebs.io + parameters: + storage: "lvm" + volgroup: "lvmvg" + formatOptions: "-b 4096 -N 5000000" ## Extra mkfs options for filesystem volumes + ``` + + The options are applied only while the volume is being formatted, which happens the first time the volume is mounted. Changing `formatOptions` in the storage class has no effect on volumes that have already been formatted. + + Refer to the documentation of the filesystem you are using to know which format options it supports. + +:::note +Format options are not validated by the driver. If the options are not valid for the chosen filesystem, then formatting fails and the volume does not mount. + +Format options are also not applied to [raw block volumes](../advanced-operations/lvm-raw-block-volume.md), because a raw block volume is not formatted with a filesystem. +::: + ## Shared (Optional) Local PV LVM volume mount points can be shared among the multiple pods on the same node. Applications that can share the volume can set the value of `shared` parameter to yes. Refer [Shared Volume](https://github.com/openebs/lvm-localpv/blob/develop/design/lvm/storageclass-parameters/shared.md) for more information about workflow of shared volume. @@ -150,6 +179,30 @@ It is recommended to use vgpattern since volumegroup will be deprecated in futur $ modprobe dm_thin_pool ``` +## Scheduler (Optional) + + The `scheduler` parameter selects the algorithm that the Local PV LVM driver uses to pick the node on which a volume is provisioned. Only the volume groups matching the `volgroup` or `vgpattern` parameter are considered. + + - `SpaceWeighted` picks the node that has a volume group with the highest free space. + - `CapacityWeighted` picks the node containing a volume group that has the least allocated storage in terms of capacity. + - `VolumeWeighted` picks the node containing a volume group that has the least number of volumes provisioned on it. + + If the scheduler parameter is not provided in the storage class, `SpaceWeighted` is used. + + ```yaml + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: openebs-lvm + provisioner: local.csi.openebs.io + parameters: + storage: "lvm" + volgroup: "lvmvg" + scheduler: "CapacityWeighted" ## or "SpaceWeighted" or "VolumeWeighted" + ``` + + The scheduling algorithm accounts only for the volume groups and does not consider other factors such as available CPU or memory. If the application pod has node selector or affinity rules, or CPU and memory constraints, use the Kubernetes scheduler instead by setting `volumeBindingMode` to `WaitForFirstConsumer`. + ## VolumeBindingMode (Optional) Local PV LVM supports two types of volume binding modes that are `Immediate` and `late binding`. diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-vac-parameters.md b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-vac-parameters.md index 4e2e885e4..d1e5dd8c1 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-vac-parameters.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/configuration/lvm-vac-parameters.md @@ -34,6 +34,8 @@ The following VAC parameters are supported for Local PV LVM volumes. :::note Setting an IOPS parameter to `max` disables IOPS limiting for that parameter on the volume. A value of `0` is not supported for IOPS or bandwidth parameters. + +VAC parameter names are case-sensitive and must be written exactly as shown in the table above. A misspelled or differently cased name is rejected as an unsupported parameter rather than being ignored. Values must not have leading or trailing spaces. ::: **Example VAC Configuration** @@ -68,6 +70,17 @@ parameters: qosBandwithWritePerSec: "8000Mi" ``` +## Combining Unified and Directional Parameters + +`qosIopsLimit` and `qosBandwithPerSec` are unified parameters. Each one applies to both the read and the write direction, unless the matching directional parameter is also specified: + +- `qosIopsLimit: "500"` on its own limits both reads and writes to 500 IOPS. +- `qosIopsLimit: "500"` together with `qosIopsWriteLimit: "200"` limits reads to 500 IOPS and writes to 200 IOPS. + +The same applies to `qosBandwithPerSec` with `qosBandwithReadPerSec` and `qosBandwithWritePerSec`. + +A unified parameter and a directional parameter of the same kind can be used together only when they do not contradict each other. If both are specified with different values, the VAC is rejected. Specifying the same value in both is accepted. + ## Update VAC QoS Parameters You can modify the QoS policy applied to a PVC by updating the `volumeAttributesClassName` field to reference a different VAC. @@ -110,6 +123,12 @@ kubectl apply -f pvc.yaml After the PVC is updated, OpenEBS Local PV LVM automatically reconciles and applies the new QoS settings associated with the updated `VolumeAttributesClass`. +:::note +When a volume is moved to a different VAC, only the parameters present in the new VAC are applied. A parameter that the new VAC does not specify keeps the value that is already in effect on the volume and is not reset. + +To remove a limit that was applied earlier, set that parameter to `max` in the new VAC instead of omitting it. +::: + ### Invalid VAC Parameter Example **Conflicting QoS Parameters** @@ -127,6 +146,21 @@ parameters: qosIopsReadLimit: "1000" ``` +The unified parameter `qosIopsLimit` removes the IOPS limit for both directions, while `qosIopsReadLimit` sets a read limit of 1000 IOPS. As the two values contradict each other, the VAC is rejected and the QoS settings of the volume are left unchanged. + +To apply these limits, use only the directional parameters: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: VolumeAttributesClass +metadata: + name: lvm-qos-read-limited +driverName: local.csi.openebs.io +parameters: + qosIopsReadLimit: "1000" + qosIopsWriteLimit: "max" +``` + ## Verify VAC QoS Configuration Use the following command to verify the PVC configuration. From 22626428e379c0afd9892c9496fca97b99b81306 Mon Sep 17 00:00:00 2001 From: Bala Harish A Date: Tue, 25 Aug 2026 12:54:01 +0530 Subject: [PATCH 6/9] docs: add Local PV Hostpath storageclass parameters, create PVC, and EXT4 quota docs Adds StorageClass Parameters and Create PVC pages for Local PV Hostpath, documents FilePermissions and PVC cas-config annotations, and adds a four-page EXT4 Quota set mirroring the XFS Quota docs. Also records the PVC-level BasePath override restriction in the v4.6 release notes and fixes two See Also links in the XFS Quota pages. Signed-off-by: Bala Harish A --- docs/main/releases.md | 4 + .../ext4-quota/enable-ext4-quota.md | 183 +++++++++++++ .../ext4-quota/ext4-quota-pre.md | 250 ++++++++++++++++++ .../ext4-quota/loop-device-ext4-quota.md | 88 ++++++ .../ext4-quota/modify-ext4-quota.md | 189 +++++++++++++ .../xfs-quota/enable-xfs-quota.md | 2 +- .../xfs-quota/xfs-quota-pre.md | 2 +- .../configuration/hostpath-create-pvc.md | 125 +++++++++ .../hostpath-create-storageclass.md | 34 +++ .../configuration/hostpath-deployment.md | 4 +- .../hostpath-storageclass-parameters.md | 164 ++++++++++++ docs/sidebars.ts | 43 +++ 12 files changed, 1085 insertions(+), 3 deletions(-) create mode 100644 docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/enable-ext4-quota.md create mode 100644 docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/ext4-quota-pre.md create mode 100644 docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/loop-device-ext4-quota.md create mode 100644 docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/modify-ext4-quota.md create mode 100644 docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-pvc.md create mode 100644 docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-storageclass-parameters.md diff --git a/docs/main/releases.md b/docs/main/releases.md index 76a6c46ac..060ef46b6 100644 --- a/docs/main/releases.md +++ b/docs/main/releases.md @@ -158,6 +158,10 @@ The status of the various components as of v4.6 are as follows: ### Local Storage +- **PVC-Level BasePath Override Disabled for Local PV Hostpath** + + As a security hardening, a `BasePath` supplied through the `cas.openebs.io/config` annotation on a PersistentVolumeClaim is now ignored. This prevents a user who can create PersistentVolumeClaims from choosing the directory on the node where the volume is created. Set `BasePath` on the StorageClass instead. Deployments that depend on the earlier behaviour can restore it with the `localpv-provisioner.localpv.allowInsecurePvcBasePathOverride` Helm value, which is disabled by default. + - **Deprecated Local PV Rawfile Helm Values Removed** The top-level `dataDirPath` and `reservedCapacity` Helm chart values have been removed. Use the equivalent storage pool specific values instead. diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/enable-ext4-quota.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/enable-ext4-quota.md new file mode 100644 index 000000000..3af22f01d --- /dev/null +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/enable-ext4-quota.md @@ -0,0 +1,183 @@ +--- +id: enable-ext4-quota +title: Enable EXT4 Quota on LocalPV Hostpath +keywords: + - OpenEBS LocalPV Hostpath Enable EXT4 Quota + - EXT4 Quota + - Enable EXT4 Quota + - Advanced Operations +description: This section describes about enabling EXT4 quotas for OpenEBS LocalPV Hostpath. +--- + +# Enable EXT4 Quota on LocalPV Hostpath + +This document provides the necessary steps to enable and configure EXT4 Quota on OpenEBS LocalPV Hostpath. By following these instructions, you will install the OpenEBS LocalPV provisioner, create a StorageClass with EXT4 Quota support, and set up a PersistentVolumeClaim (PVC) to apply project quotas on the local volumes. It also includes the process for mounting the volume to an application pod and verifying that the quota is successfully applied. + +:::important +Complete the [EXT4 Quota Prerequisites](ext4-quota-pre.md) before proceeding. The filesystem holding the `BasePath` must have the `project` and `quota` features enabled and must be mounted with the `prjquota` option. +::: + +## Install the OpenEBS Dynamic LocalPV Provisioner + +Refer to the [OpenEBS Installation documentation](../../../../../quickstart-guide/installation.md) to install the OpenEBS LocalPV Hostpath Provisioner. + +## Create StorageClass + +1. To create a hostpath StorageClass with the EXT4Quota configuration option, use the following YAML definition. This configuration will enable the EXT4Quota for the specified path and storage type. + +``` +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: openebs-hostpath-ext4 + annotations: + openebs.io/cas-type: local + cas.openebs.io/config: | + - name: StorageType + value: "hostpath" + - name: BasePath + value: "/var/openebs/local/" + - name: EXT4Quota + enabled: "true" +provisioner: openebs.io/local +volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Delete +``` + +2. For advanced configuration of EXT4Quota, you may also set the `softLimitGrace` and `hardLimitGrace` parameters, which define the storage capacity limits beyond the Persistent Volume (PV) storage request. The updated YAML definition is as follows: + +``` +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: openebs-hostpath-ext4 + annotations: + openebs.io/cas-type: local + cas.openebs.io/config: | + - name: StorageType + value: "hostpath" + - name: BasePath + value: "/var/openebs/local/" + - name: EXT4Quota + enabled: "true" + data: + softLimitGrace: "0%" + hardLimitGrace: "0%" +provisioner: openebs.io/local +volumeBindingMode: WaitForFirstConsumer +reclaimPolicy: Delete +``` + +:::note +- `softLimitGrace` and `hardLimitGrace` are used in conjunction with the PV storage request to determine the soft and hard limits of the quota. + +- The size of these limits is calculated as **"Size of PV storage request * (1 + LimitGrace%)"** + +- If no values are specified, the default is **softLimitGrace: "0%" / hardLimitGrace: "0%"**, meaning the storage capacity is limited to the PV storage request value. + + For example, with a PV of 100Gi capacity and values **softLimitGrace: "90%" / hardLimitGrace: "100%"**, the soft limit will be set to 190Gi, and the hard limit will be set to 200Gi. + You can select to use either `softLimitGrace` or `hardLimitGrace` independently based on your requirements. + + Refer to the [setquota documentation](https://man7.org/linux/man-pages/man8/setquota.8.html) for more detailed information regarding soft and hard limits. +::: + +## Create a PVC + +1. To create a PVC using the StorageClass's name, use the following definition: + +``` +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: local-hostpath-ext4 +spec: + storageClassName: openebs-hostpath-ext4 + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi +``` + +At this stage, the PVC will remain in the 'Pending' state until the volume is successfully mounted. + +2. Verify the PVC status. + +``` +$ kubectl get pvc +``` + +**Example Output** + +``` +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +local-hostpath-ext4 Pending openebs-hostpath-ext4 21s +``` + +## Mount the Volume + +1. Mount the volume to the application pod container. A sample BusyBox Pod template is as follows: + +``` +apiVersion: v1 +kind: Pod +metadata: + name: busybox +spec: + volumes: + - name: local-storage + persistentVolumeClaim: + claimName: local-hostpath-ext4 + containers: + - name: busybox + image: busybox + command: + - sh + - -c + - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done' + volumeMounts: + - mountPath: /mnt/store + name: local-storage +``` + +The PVC status will change to 'Bound' once the volume is successfully mounted and the quota will be applied. + +2. Verify that the EXT4 project quota is applied. Run the command on the node where the volume was provisioned. + +``` +$ sudo repquota -P /var/openebs/local +``` + +**Example Output** + +``` +*** Report for project quotas on device /dev/nvme1n1 +Block grace time: 7days; Inode grace time: 7days + Block limits File limits +Project used soft hard grace used soft hard grace +---------------------------------------------------------------------- +#0 -- 20 0 0 2 0 0 +#1 -- 0 5242880 5242880 1 0 0 +``` + +3. You can also confirm the project ID that was assigned to the volume directory. + +``` +$ sudo lsattr -pd /var/openebs/local/pvc-864a5ac8-dd3f-416b-9f4b-ffd7d285b425 +``` + +**Example Output** + +``` + 1 --------------P------ /var/openebs/local/pvc-864a5ac8-dd3f-416b-9f4b-ffd7d285b425 +``` + +## Limitation + +Resizing of quota is not supported. + +## See Also + +- [EXT4 Quota Prerequisites](ext4-quota-pre.md) +- [Modify EXT4 Quota on LocalPV Hostpath](modify-ext4-quota.md) +- [EXT4 Quota with Loop Device](loop-device-ext4-quota.md) diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/ext4-quota-pre.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/ext4-quota-pre.md new file mode 100644 index 000000000..50a53737b --- /dev/null +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/ext4-quota-pre.md @@ -0,0 +1,250 @@ +--- +id: ext4-quota-pre +title: EXT4 Quota Prerequisites +keywords: + - OpenEBS LocalPV Hostpath Enable EXT4 Quota + - EXT4 Quota + - EXT4 Quota Prerequisites + - Advanced Operations +description: This section talks about the prerequisites of EXT4 quotas for OpenEBS LocalPV Hostpath. +--- + +# EXT4 Quota Prerequisites + +To enable EXT4 Quota on LocalPV Hostpath, certain prerequisites must be met to ensure proper configuration and functionality. This involves installing the `quota` and `e2fsprogs` packages, verifying the filesystem type, enabling the project quota feature on the filesystem, and mounting it with the `prjquota` mount option. + +The following steps outline the installation and configuration procedures for both root and data disk filesystems on Ubuntu, Debian, RHEL, and CentOS systems. By completing these steps, you will be ready to enable and manage EXT4 Quotas on your OpenEBS LocalPV Hostpath setup. + +## Install the `quota` and `e2fsprogs` Packages + +The `quota` package provides the `repquota` and `setquota` commands, and `e2fsprogs` provides the `tune2fs`, `chattr`, and `lsattr` commands. + +**For Ubuntu/Debian Systems** + +To install the packages on Ubuntu and Debian systems, execute the following command: + +``` +$ sudo apt-get update +$ sudo apt-get install -y quota e2fsprogs +``` + +**For RHEL/CentOS Systems** + +To install the packages on RHEL/CentOS systems, execute the following command: + +``` +$ sudo yum install -y quota e2fsprogs +``` + +**For Fedora** + +To install the packages on Fedora, execute the following command: + +``` +$ sudo dnf install -y quota e2fsprogs +``` + +## Check the Filesystem Type + +Verify whether the filesystem of the hostPath directory is EXT4. The default hostPath directory is `/var/openebs/local`. Execute the following command to check the filesystem type and to identify the device where the filesystem is stored: + +``` +$ df -Th /var/openebs/local +``` + +**Example Output** + +``` +Filesystem Type Size Used Avail Use% Mounted on +/dev/nvme1n1 ext4 8.0G 959M 7.1G 12% /mnt/data +``` + +If the above command fails because the path does not exist yet, execute the following script to check the filesystem type and host device name of the closest existing parent directory: + +``` +BASEPATH="/var/openebs/local" + +until OUTPUT=$(df -Th $BASEPATH 2> /dev/null) +do +BASEPATH=$(echo "$BASEPATH" | sed 's|\(.*\)/.*|\1|') +done + +echo "PATH=${BASEPATH}" +#Final output +echo "$OUTPUT" +``` + +## Enable the Project Quota Feature on the Filesystem + +Unlike XFS, an EXT4 filesystem must have the `project` and `quota` features enabled before project quotas can be used. Filesystems created by older versions of `mkfs.ext4` may not have them. + +1. Check the features that are currently enabled on the device. + +``` +$ sudo tune2fs -l /dev/nvme1n1 | grep -i "filesystem features" +``` + +**Example Output** + +``` +Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum +``` + +In the example above, `project` and `quota` are not present, so they have to be enabled. + +2. Unmount the filesystem. The features cannot be changed while the filesystem is mounted. + +``` +$ sudo umount /dev/nvme1n1 +``` + +3. Check the filesystem for errors, which is recommended before changing its features. + +``` +$ sudo e2fsck -f /dev/nvme1n1 +``` + +4. Enable the `project` and `quota` features. + +``` +$ sudo tune2fs -O project,quota /dev/nvme1n1 +``` + +5. Verify that the features are now enabled. + +``` +$ sudo tune2fs -l /dev/nvme1n1 | grep -i "filesystem features" +``` + +**Expected Output** + +``` +Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum quota project +``` + +:::note +The `project` feature requires the filesystem to have an inode size of 256 bytes or more. This is the default for `mkfs.ext4`, but filesystems created with a smaller inode size cannot be converted, and have to be recreated. You can check the inode size with `sudo tune2fs -l /dev/nvme1n1 | grep -i "inode size"`. +::: + +## Mount the Filesystem using the `prjquota` Mount Option + +1. **Check Existing Mount Options** + +Ensure that the mount options for the device include `prjquota`. Execute the following command to verify the mount options for the device (Example: `/dev/nvme1n1`). + +``` +$ sudo mount | grep "^/dev/nvme1n1" +``` + +**Example Output** + +``` +/dev/nvme1n1 on /mnt/data type ext4 (rw,relatime) +``` + +If the mount options already include `prjquota`, you can proceed to the next section to [Enable EXT4 Quota](enable-ext4-quota.md). If not, continue with the steps below. + +2. **Mount the Device with the `prjquota` Option** + +If the filesystem is the root filesystem (`/`), follow the instructions below. If the filesystem is located on a data disk, proceed to the [Filesystem on Data Disk](#filesystem-on-data-disk) section. + +**Root Filesystem:** + +To enable `prjquota` for the root filesystem, modify the `GRUB_CMDLINE_LINUX` option in the `/etc/default/grub` file. + +- Edit the file `/etc/default/grub`. + +``` +$ sudo vi /etc/default/grub +``` + +- Locate the line containing the variable `GRUB_CMDLINE_LINUX`. + +``` +GRUB_CMDLINE_LINUX="console=tty0 crashkernel=auto net.ifnames=0 console=ttyS0" +``` + +- Add `rootflags=prjquota` at the end of the string. If the `rootflags` option is already present, append `prjquota` to the list of options. + +``` +GRUB_CMDLINE_LINUX="console=tty0 crashkernel=auto net.ifnames=0 console=ttyS0 rootflags=prjquota" +``` + +- Locate the `grub.cfg` file. The file path may vary based on your OS. + + - /boot/grub2/grub.cfg + - /boot/efi/EFI/ubuntu/grub.cfg + - /boot/efi/EFI/debian/grub.cfg + - /boot/efi/EFI/redhat/grub.cfg + - /boot/efi/EFI/centos/grub.cfg + - /boot/efi/EFI/fedora/grub.cfg + +- Create a backup copy of the existing `grub.cfg`. The sample commands below use the path `/boot/grub2/grub.cfg`. Replace the paths with your `grub.cfg` path. + +``` +$ sudo cp /boot/grub2/grub.cfg /boot/grub2/grub.cfg.backup +``` + +- Generate a new `grub.cfg` that includes the changes. + +``` +$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg +``` + +- Reboot the system. + +``` +$ sudo reboot +``` + +- After rebooting, check the mount options again to confirm the changes. + +``` +$ sudo mount | grep " / " +``` + +**Expected Output** + +``` +/dev/nvme0n1p1 on / type ext4 (rw,relatime,prjquota) +``` + +### Filesystem on Data Disk + +If the filesystem is located on a data disk, follow these steps: + +1. Unmount the filesystem on the data disk (Replace `/dev/nvme1n1` and `/mnt/data` with your device and mount path). + +``` +$ sudo umount /dev/nvme1n1 +``` + +2. Mount the disk using the `prjquota` mount option. + +``` +$ sudo mount -o rw,prjquota /dev/nvme1n1 /mnt/data +``` + +3. Verify the mount options. + +``` +$ sudo mount | grep "^/dev/nvme1n1" +``` + +**Expected Output** + +``` +/dev/nvme1n1 on /mnt/data type ext4 (rw,relatime,prjquota) +``` + +4. Add the `prjquota` option to the `/etc/fstab` file for the data disk to make the changes persistent across reboots. + +``` +UUID=9cff3d69-3769-4ad9-8460-9c54050583f9 /mnt/data ext4 defaults,prjquota 0 0 +``` + +## See Also + +- [Enable EXT4 Quota on LocalPV Hostpath](enable-ext4-quota.md) +- [Modify EXT4 Quota on LocalPV Hostpath](modify-ext4-quota.md) +- [EXT4 Quota with Loop Device](loop-device-ext4-quota.md) diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/loop-device-ext4-quota.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/loop-device-ext4-quota.md new file mode 100644 index 000000000..f99849cd8 --- /dev/null +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/loop-device-ext4-quota.md @@ -0,0 +1,88 @@ +--- +id: loop-device-ext4-quota +title: EXT4 Quota with Loop Device +keywords: + - OpenEBS LocalPV Hostpath EXT4 Quota with Loop Device + - EXT4 Quota + - EXT4 Quota with Loop Device + - Advanced Operations +description: This section talks about creating an EXT4 filesystem at the basepath as a loop device. +--- + +# EXT4 Quota with Loop Device + +In scenarios where you do not have an existing device formatted with an EXT4 filesystem that has project quota enabled, you can create one on a loop device. This process is particularly useful when the root filesystem cannot be remounted with `prjquota`, and it allows you to try out project quota enforcement without repartitioning a disk. + +This document outlines the steps to create a sparse file, format it with the EXT4 filesystem with the project quota features enabled, and mount it as a loop device at the specified directory, `/var/openebs/local`. + +## Create an EXT4 Filesystem at the Basepath as a Loop Device + +1. **Ensure the Required Utilities Are Installed** + +Before proceeding, ensure that `e2fsprogs` and `quota` are installed on your system. + +**For Ubuntu/Debian-based Systems** + +``` +sudo apt update +sudo apt-get install -y quota e2fsprogs +``` + +**For RHEL/CentOS-based Systems** + +``` +sudo yum install -y quota e2fsprogs +``` + +2. **Create the Mount Directory** + +Create the directory where the filesystem will be mounted. + +``` +sudo mkdir -p /var/openebs/local +cd /var/openebs +``` + +3. **Create a Sparse File** + +Create a sparse file of maximum size 1GiB. Use a size that can accommodate the volumes you intend to provision. + +``` +sudo dd if=/dev/zero of=ext4.1G bs=1 count=0 seek=1G +``` + +4. **Format the Sparse File with EXT4** + +Format the newly created sparse file with the EXT4 filesystem, enabling the `quota` and `project` features that project quotas require. + +``` +sudo mkfs.ext4 -F -O quota,project ext4.1G +``` + +5. **Mount the Sparse File** + +Mount the sparse file as a loop device with project quota enabled. This will make the file accessible as the directory `/var/openebs/local`. + +``` +sudo mount -o loop,rw,prjquota ext4.1G /var/openebs/local +``` + +6. **Verify the Mount Options** + +``` +sudo mount | grep "/var/openebs/local" +``` + +**Expected Output** + +``` +/var/openebs/ext4.1G on /var/openebs/local type ext4 (rw,relatime,prjquota) +``` + +You can now proceed to [Enable EXT4 Quota on LocalPV Hostpath](enable-ext4-quota.md). + +## See Also + +- [EXT4 Quota Prerequisites](ext4-quota-pre.md) +- [Enable EXT4 Quota on LocalPV Hostpath](enable-ext4-quota.md) +- [Modify EXT4 Quota on LocalPV Hostpath](modify-ext4-quota.md) diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/modify-ext4-quota.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/modify-ext4-quota.md new file mode 100644 index 000000000..c900ac75d --- /dev/null +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/modify-ext4-quota.md @@ -0,0 +1,189 @@ +--- +id: modify-ext4-quota +title: Modify EXT4 Quota on LocalPV Hostpath +keywords: + - OpenEBS LocalPV Hostpath Modify EXT4 Quota + - EXT4 Quota + - Modify EXT4 Quota + - Advanced Operations +description: This section talks about modifying EXT4 quotas for OpenEBS LocalPV Hostpath. +--- + +# Modify EXT4 Quota on LocalPV Hostpath + +This document provides the necessary steps to modify or remove the EXT4 project quota enforcement for existing OpenEBS LocalPV Hostpath volumes. EXT4 quotas help in managing storage utilization by enforcing soft and hard limits for allocated volumes. + +## Identify the BasePath Directory + +Make a note of the BasePath directory used for the hostpath volume. The default BasePath is `/var/openebs/local`. You can retrieve the BasePath from the StorageClass by executing the following command: + +``` +$ kubectl describe sc +``` + +## Locate the Node + +1. Log in to the node where the volume exists. You can determine the node by describing the Persistent Volume (PV) resource. To retrieve information about the PV, use the following command: + +``` +$ kubectl get pvc --namespace demo +``` + +**Example Output** + +``` +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +demo-vol-demo-0 Bound pvc-0365904e-0add-45ec-9b4e-f4080929d6cd 2Gi RWO openebs-hostpath-ext4 21s +``` + +2. Describe the PV. + +``` +$ kubectl describe pv pvc-0365904e-0add-45ec-9b4e-f4080929d6cd +``` + +**Example Output** + +``` +Name: pvc-0365904e-0add-45ec-9b4e-f4080929d6cd +Labels: openebs.io/cas-type=local-hostpath +Annotations: pv.kubernetes.io/provisioned-by: openebs.io/local +Finalizers: [kubernetes.io/pv-protection] +StorageClass: openebs-hostpath-ext4 +Status: Bound +Claim: demo/demo-vol-demo-0 +Reclaim Policy: Delete +Access Modes: RWO +VolumeMode: Filesystem +Capacity: 2Gi +Node Affinity: + Required Terms: + Term 0: kubernetes.io/hostname in [storage-node-2] +Message: +Source: + Type: LocalVolume (a persistent volume backed by local storage on a node) + Path: /var/openebs/local/pvc-0365904e-0add-45ec-9b4e-f4080929d6cd +Events: +``` + +3. Identify the node name. + +``` +$ kubectl get node -l 'kubernetes.io/hostname in (storage-node-2)' +``` + +**Example Output** + +``` +NAME STATUS ROLES AGE VERSION +storage-node-2 Ready worker 10m v1.22.1 +``` + +## Modify the EXT4 Quota Limits + +You can change the soft and/or hard limit of an existing hostpath volume with EXT4 project quota enabled by following the steps below. If you wish to remove the EXT4 project quota entirely, refer to the [Remove Project](#remove-project) section. + +### Change Quota Limits + +Execute the following commands on the node where the hostpath volume exists: + +- Make a note of the Project ID. + +``` +$ sudo repquota -P /var/openebs/local +``` + +**Example Output** + +``` +*** Report for project quotas on device /dev/nvme1n1 +Block grace time: 7days; Inode grace time: 7days + Block limits File limits +Project used soft hard grace used soft hard grace +---------------------------------------------------------------------- +#0 -- 20 0 0 2 0 0 +#1 -- 1048576 2097152 2097152 1 0 0 +``` + +You can also read the project ID directly from the volume directory: + +``` +$ sudo lsattr -pd /var/openebs/local/pvc-0365904e-0add-45ec-9b4e-f4080929d6cd +``` + +- Modify the quota limits using the following command. The arguments are the project ID, the block soft limit, the block hard limit, the inode soft limit, and the inode hard limit, followed by the filesystem. The block limits are expressed in kilobytes. + +``` +$ sudo setquota -P 1 3145728 5242880 0 0 /var/openebs/local +``` + +:::note +The command above sets a soft limit of 3 GiB (3145728 KB) and a hard limit of 5 GiB (5242880 KB) for project ID 1. The inode limits are set to 0, which means they are unlimited. +::: + +- Verify the updated limits. + +``` +$ sudo repquota -P /var/openebs/local +``` + +**Example Output** + +``` +*** Report for project quotas on device /dev/nvme1n1 +Block grace time: 7days; Inode grace time: 7days + Block limits File limits +Project used soft hard grace used soft hard grace +---------------------------------------------------------------------- +#0 -- 20 0 0 2 0 0 +#1 -- 1048576 3145728 5242880 1 0 0 +``` + +## Remove Project + +To completely remove the EXT4 project quota from a volume, follow these steps: + +1. Make a note of the Project ID. + +``` +$ sudo repquota -P /var/openebs/local +``` + +2. Set the project limits to 0, effectively removing the quota limits. + +``` +$ sudo setquota -P 1 0 0 0 0 /var/openebs/local +``` + +:::note +The command is for a project ID=1 at directory path `/var/openebs/local`. +::: + +3. Clear the project ID and the project inheritance attribute from the volume directory. + +``` +$ sudo chattr -P -p 0 /var/openebs/local/pvc-0365904e-0add-45ec-9b4e-f4080929d6cd +``` + +4. Verify the changes. + +``` +$ sudo repquota -P /var/openebs/local +``` + +**Expected Output** + +``` +*** Report for project quotas on device /dev/nvme1n1 +Block grace time: 7days; Inode grace time: 7days + Block limits File limits +Project used soft hard grace used soft hard grace +---------------------------------------------------------------------- +#0 -- 1048596 0 0 3 0 0 +``` + +## See Also + +- [EXT4 Quota Prerequisites](ext4-quota-pre.md) +- [Enable EXT4 Quota on LocalPV Hostpath](enable-ext4-quota.md) +- [EXT4 Quota with Loop Device](loop-device-ext4-quota.md) diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/xfs-quota/enable-xfs-quota.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/xfs-quota/enable-xfs-quota.md index 9ee0d434b..45cef1038 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/xfs-quota/enable-xfs-quota.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/xfs-quota/enable-xfs-quota.md @@ -163,4 +163,4 @@ Resizing of quota is not supported. - [XFS Quota Prerequisites](xfs-quota-pre.md) - [Modify XFS Quota on LocalPV Hostpath](modify-xfs-quota.md) -- [XFS Quota with Loop Device](xfs-quota-pre.md) \ No newline at end of file +- [XFS Quota with Loop Device](loop-device-xfs-quota.md) \ No newline at end of file diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/xfs-quota/xfs-quota-pre.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/xfs-quota/xfs-quota-pre.md index 3c543d5be..d61a30517 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/xfs-quota/xfs-quota-pre.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/xfs-quota/xfs-quota-pre.md @@ -197,4 +197,4 @@ UUID=9cff3d69-3769-4ad9-8460-9c54050583f9 /mnt/data xfs defaul - [Enable XFS Quota on LocalPV Hostpath](enable-xfs-quota.md) - [Modify XFS Quota on LocalPV Hostpath](modify-xfs-quota.md) -- [XFS Quota with Loop Device](xfs-quota-pre.md) \ No newline at end of file +- [XFS Quota with Loop Device](loop-device-xfs-quota.md) \ No newline at end of file diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-pvc.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-pvc.md new file mode 100644 index 000000000..4fce35762 --- /dev/null +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-pvc.md @@ -0,0 +1,125 @@ +--- +id: hostpath-create-pvc +title: Create PersistentVolumeClaim +keywords: + - OpenEBS Local PV Hostpath + - Local PV Hostpath Configuration + - Configuration + - Create PersistentVolumeClaim + - Local PV Hostpath PVC +description: This guide will help you to create a PersistentVolumeClaim for Local PV Hostpath. +--- + +# Create PersistentVolumeClaim + +This document provides step-by-step instructions to create a PersistentVolumeClaim (PVC) using a Local PV Hostpath StorageClass. It also explains how a PVC can carry its own configuration through an annotation. + +## Create a PVC + +Save the following as `local-hostpath-pvc.yaml`, updating `storageClassName` if you are using a StorageClass other than the default `openebs-hostpath`: + +```yaml +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: local-hostpath-pvc +spec: + storageClassName: openebs-hostpath + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi +``` + +Create the PVC: + +``` +kubectl apply -f local-hostpath-pvc.yaml +``` + +:::note +Local PV Hostpath supports the `ReadWriteOnce` access mode only. A PVC that requests any other access mode is not provisioned. +::: + +## Verify the PVC + +``` +kubectl get pvc local-hostpath-pvc +``` + +The PVC remains in `Pending` state until an application pod that uses it is scheduled: + +``` +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +local-hostpath-pvc Pending openebs-hostpath 21s +``` + +This is expected. Local PV Hostpath StorageClasses use `volumeBindingMode: WaitForFirstConsumer`, because the volume directory has to be created on the node where the application pod runs. Kubernetes selects that node first, and the provisioner then creates the volume there. + +Once a pod using the PVC is scheduled, the PVC is bound: + +``` +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +local-hostpath-pvc Bound pvc-864a5ac8-dd3f-416b-9f4b-ffd7d285b425 5Gi RWO openebs-hostpath 3m +``` + +Refer to [Deploy an Application](hostpath-deployment.md) for deploying a pod that uses this PVC. + +## Configure a Volume Using PVC Annotations + +In addition to the settings on the StorageClass, an individual PVC can supply configuration through the `cas.openebs.io/config` annotation. This is useful when one volume needs a setting that differs from the rest of the volumes provisioned by the same StorageClass. + +The annotation takes the same entries as the StorageClass annotation. For example, to create the volume directory of this one volume with `0770` permissions: + +```yaml +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: local-hostpath-pvc + annotations: + cas.openebs.io/config: | + - name: FilePermissions + data: + mode: "0770" +spec: + storageClassName: openebs-hostpath + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 5Gi +``` + +:::note +The configuration on the StorageClass takes precedence. A PVC annotation only applies for the parameters that the StorageClass does not already set, so it cannot be used to override a value that the StorageClass defines. + +In the example above, the `0770` mode is applied only if the `openebs-hostpath` StorageClass does not set `FilePermissions` itself. +::: + +### BasePath in a PVC Annotation + +`BasePath` supplied through a PVC annotation is ignored by default. Because a PVC can be created by any user with access to a namespace, allowing it to choose a directory on the node would let that user place volumes outside the location the cluster administrator intended. + +If you need the earlier behaviour, the provisioner can be started with the `--allow-insecure-pvc-basepath-override` flag, which is exposed through the Helm chart: + +```yaml +localpv-provisioner: + localpv: + allowInsecurePvcBasePathOverride: true +``` + +:::warning +Enabling this option allows a user who can create PVCs in any namespace to choose the directory on the node where the volume is created. Leave it disabled unless you specifically need it, and set `BasePath` on the StorageClass instead. +::: + +## Support + +If you encounter issues or have a question, file a [Github issue](https://github.com/openebs/openebs/issues/new), or talk to us on the [#openebs channel on the Kubernetes Slack server](https://kubernetes.slack.com/messages/openebs/). + +## See Also + +- [Installation](../../../../quickstart-guide/installation.md) +- [Create StorageClass(s)](hostpath-create-storageclass.md) +- [StorageClass Parameters](hostpath-storageclass-parameters.md) +- [Deploy an Application](hostpath-deployment.md) diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-storageclass.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-storageclass.md index 19feebc30..83ff1ae4c 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-storageclass.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-storageclass.md @@ -63,6 +63,38 @@ The default Storage Class is called `openebs-hostpath` and its `BasePath` is con Using NodeAffinityLabels does not influence the scheduling of the application Pod. Use Kubernetes [Allowed Topologies](#restrict-volume-placement-using-allowed-topologies) to configure scheduling options. ::: + ## (Optional) File Permissions + + By default, Local PV Hostpath creates the volume directory with `0777` permissions. For some workloads these permissions are wider than necessary. Use the `FilePermissions` config to set the permissions that the volume directory is created with: + + ``` + apiVersion: storage.k8s.io/v1 + kind: StorageClass + metadata: + name: local-hostpath + annotations: + openebs.io/cas-type: local + cas.openebs.io/config: | + - name: StorageType + value: "hostpath" + - name: BasePath + value: "/var/local-hostpath" + - name: FilePermissions + data: + mode: "0770" + provisioner: openebs.io/local + reclaimPolicy: Delete + volumeBindingMode: WaitForFirstConsumer + ``` + + With the above StorageClass, the directory of every volume provisioned by it is created with `0770` permissions. + + :::note + The permissions are applied when the volume directory is created, so changing `FilePermissions` later does not affect volumes that already exist. + + `FilePermissions` cannot be set through the Helm chart values for the default `openebs-hostpath` StorageClass. To use it, create a custom StorageClass as shown above, or set it for an individual volume through a [PVC annotation](hostpath-create-pvc.md#configure-a-volume-using-pvc-annotations). + ::: + 2. Edit `local-hostpath-sc.yaml` and update with your desired values for `metadata.name` and `cas.openebs.io/config.BasePath`. :::note @@ -135,4 +167,6 @@ If you encounter issues or have a question, file a [Github issue](https://github ## See Also - [Installation](../../../../quickstart-guide/installation.md) +- [StorageClass Parameters](hostpath-storageclass-parameters.md) +- [Create PersistentVolumeClaim](hostpath-create-pvc.md) - [Deploy an Application](hostpath-deployment.md) \ No newline at end of file diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-deployment.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-deployment.md index aa5bc5b06..d9aac7a22 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-deployment.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-deployment.md @@ -34,4 +34,6 @@ If you encounter issues or have a question, file a [Github issue](https://github ## See Also - [Installation](../../../../quickstart-guide/installation.md) -- [Create StorageClass(s)](hostpath-create-storageclass.md) \ No newline at end of file +- [Create StorageClass(s)](hostpath-create-storageclass.md) +- [StorageClass Parameters](hostpath-storageclass-parameters.md) +- [Create PersistentVolumeClaim](hostpath-create-pvc.md) \ No newline at end of file diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-storageclass-parameters.md b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-storageclass-parameters.md new file mode 100644 index 000000000..2d258b346 --- /dev/null +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-storageclass-parameters.md @@ -0,0 +1,164 @@ +--- +id: hostpath-storageclass-parameters +title: StorageClass Parameters +keywords: + - OpenEBS Local PV Hostpath + - Local PV Hostpath Configuration + - Configuration + - StorageClass Parameters + - Local PV Hostpath StorageClass Parameters +description: This document describes the supported StorageClass parameters for Local PV Hostpath. +--- + +# StorageClass Parameters + +This document describes the StorageClass fields and parameters supported by Local PV Hostpath and explains how to configure them. These settings control where the volume directory is created on the node, the permissions it is created with, how volumes are placed across nodes, and whether a quota is enforced on the directory. + +Unlike the other Local Storage engines, Local PV Hostpath is provisioned by an out-of-tree provisioner and does not use the `parameters` field of the StorageClass. Its settings are supplied through the `cas.openebs.io/config` annotation, alongside the `openebs.io/cas-type: local` annotation: + +```yaml +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: local-hostpath + annotations: + openebs.io/cas-type: local + cas.openebs.io/config: | + - name: StorageType + value: "hostpath" + - name: BasePath + value: "/var/local-hostpath" +provisioner: openebs.io/local +reclaimPolicy: Delete +volumeBindingMode: WaitForFirstConsumer +``` + +Each entry under `cas.openebs.io/config` has a `name`, and supplies its setting through `value`, `data`, or `list`, depending on the parameter. + +## Standard StorageClass Fields + +| Field | Values | +|-------|--------| +| `provisioner` | Must be `openebs.io/local` | +| `volumeBindingMode` | Must be `WaitForFirstConsumer` | +| `reclaimPolicy` | `Delete`, `Retain` | +| `allowedTopologies` | Node label key and the values to match | + +## Supported cas-config Parameters + +| Parameter | Form | Values | +|-----------|------|--------| +| `StorageType` | `value` | `hostpath` | +| `BasePath` | `value` | Absolute path on the node, for example `/var/local-hostpath` | +| `NodeAffinityLabels` | `list` | Node label keys used to identify a node instead of its hostname | +| `FilePermissions` | `data` | `mode`, the permissions used to create the volume directory | +| `XFSQuota` | `data` | `softLimitGrace` and `hardLimitGrace` | +| `EXT4Quota` | `data` | `softLimitGrace` and `hardLimitGrace` | + +## StorageType + +`StorageType` selects the kind of storage that backs the volume. For Local PV Hostpath, set it to `hostpath`, which provisions the volume as a subdirectory under `BasePath` on the node where the application pod is scheduled. + +## BasePath + +`BasePath` is the directory on the node under which volume directories are created. Each volume gets its own subdirectory, named after the persistent volume. + +If `BasePath` is not specified, the provisioner uses the base path it was installed with, which is `/var/openebs/local` by default. The default `openebs-hostpath` StorageClass also uses `/var/openebs/local`. + +:::note +The value must be a valid absolute path. If the directory does not exist on the node, the provisioner creates it when the first volume of this StorageClass is scheduled onto that node. +::: + +## NodeAffinityLabels + +`NodeAffinityLabels` lists the node label keys that are used to identify the node in the node affinity of the provisioned volume, instead of the default `kubernetes.io/hostname` label. This is useful when the hostname of a node can change, for example when a node is removed and added back with its disks intact. + +```yaml + - name: NodeAffinityLabels + list: + - "openebs.io/custom-node-unique-id" +``` + +:::note +`NodeAffinityLabels` does not influence the scheduling of the application pod. Use [allowedTopologies](#allowedtopologies) to constrain scheduling. +::: + +Refer to [Custom Node Labeling](hostpath-create-storageclass.md#optional-custom-node-labeling) for a complete example. + +## FilePermissions + +`FilePermissions` sets the permissions that the volume directory is created with. The permissions are given through the `mode` key: + +```yaml + - name: FilePermissions + data: + mode: "0770" +``` + +If `FilePermissions` is not specified, the volume directory is created with `0777`. + +The permissions are applied when the directory is created, so changing this setting does not affect volumes that already exist. + +Refer to [File Permissions](hostpath-create-storageclass.md#optional-file-permissions) for more details, including how to set the permissions for an individual volume. + +## XFSQuota + +`XFSQuota` enforces a quota on the volume directory, so that an application cannot write more data than the capacity requested in the PersistentVolumeClaim. It requires the filesystem holding `BasePath` to be XFS and to be mounted with project quota enabled. + +```yaml + - name: XFSQuota + enabled: "true" + data: + softLimitGrace: "80%" + hardLimitGrace: "85%" +``` + +`softLimitGrace` and `hardLimitGrace` are expressed as a percentage of the requested capacity of the volume. + +Refer to [XFS Quota](../advanced-operations/xfs-quota/xfs-quota-pre.md) for the prerequisites and the full configuration workflow. + +## EXT4Quota + +`EXT4Quota` enforces a quota on the volume directory in the same way as `XFSQuota`, for a `BasePath` that is held on an ext4 filesystem mounted with project quota enabled. It takes the same `softLimitGrace` and `hardLimitGrace` keys. + +```yaml + - name: EXT4Quota + enabled: "true" + data: + softLimitGrace: "80%" + hardLimitGrace: "85%" +``` + +An ext4 filesystem additionally requires the `project` and `quota` features to be enabled on it before project quotas can be used. Refer to [EXT4 Quota](../advanced-operations/ext4-quota/ext4-quota-pre.md) for the prerequisites and the full configuration workflow. + +## VolumeBindingMode + +Local PV Hostpath requires `volumeBindingMode` to be set to `WaitForFirstConsumer`. The volume directory is created on the node where the application pod is scheduled, so the provisioner needs Kubernetes to select that node first. + +With `Immediate`, no node is selected at provisioning time and the PersistentVolumeClaim remains pending. + +## ReclaimPolicy + +Local PV Hostpath supports both the reclaim policies that are `Delete` and `Retain`. If it is not specified, it defaults to `Delete`. + +- `Delete` indicates that the volume directory on the node is removed when the PersistentVolumeClaim is deleted. +- `Retain` indicates that the volume directory and its data are kept on the node after the PersistentVolumeClaim is deleted. + +## AllowedTopologies + +By default, a Local PV Hostpath volume can be provisioned on any node in the cluster. If `BasePath` is present on certain nodes only, use `allowedTopologies` to restrict provisioning to those nodes. + +Unlike `NodeAffinityLabels`, `allowedTopologies` also influences the scheduling of the application pod. + +Refer to [Restrict Volume Placement Using Allowed Topologies](hostpath-create-storageclass.md#restrict-volume-placement-using-allowed-topologies) for examples, including how to set it on the default `openebs-hostpath` StorageClass through the Helm chart. + +## Support + +If you encounter issues or have a question, file a [Github issue](https://github.com/openebs/openebs/issues/new), or talk to us on the [#openebs channel on the Kubernetes Slack server](https://kubernetes.slack.com/messages/openebs/). + +## See Also + +- [Installation](../../../../quickstart-guide/installation.md) +- [Create StorageClass(s)](hostpath-create-storageclass.md) +- [Create PersistentVolumeClaim](hostpath-create-pvc.md) +- [Deploy an Application](hostpath-deployment.md) diff --git a/docs/sidebars.ts b/docs/sidebars.ts index e92127688..99b22661f 100644 --- a/docs/sidebars.ts +++ b/docs/sidebars.ts @@ -160,6 +160,18 @@ const sidebars: SidebarsConfig = label: "Create StorageClass(s)", key: "hostpath-create-storageclass", }, + { + type: "doc", + id: "user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-storageclass-parameters", + label: "StorageClass Parameters", + key: "hostpath-storageclass-parameters", + }, + { + type: "doc", + id: "user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-create-pvc", + label: "Create PVC", + key: "hostpath-create-pvc", + }, { type: "doc", id: "user-guides/local-storage-user-guide/local-pv-hostpath/configuration/hostpath-deployment", @@ -208,6 +220,37 @@ const sidebars: SidebarsConfig = } ] }, + { + collapsed: true, + type: "category", + label: "EXT4 Quota", + customProps: { + icon: "Book" + }, + items: [ + { + type: "doc", + id: "user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/ext4-quota-pre", + label: "Prerequisites", + key: "EXT4_Prerequisites", + }, + { + type: "doc", + id: "user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/enable-ext4-quota", + label: "Enable EXT4 Quota" + }, + { + type: "doc", + id: "user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/modify-ext4-quota", + label: "Modify EXT4 Quota" + }, + { + type: "doc", + id: "user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/ext4-quota/loop-device-ext4-quota", + label: "EXT4 Quota with Loop Device" + } + ] + }, { type: "doc", id: "user-guides/local-storage-user-guide/local-pv-hostpath/advanced-operations/node-deployment-mode", From 4c292366e485d1b77de856657637a35a042adcfa Mon Sep 17 00:00:00 2001 From: Bala Harish A Date: Tue, 25 Aug 2026 14:00:50 +0530 Subject: [PATCH 7/9] docs: clarify Local PV LVM volume restore thin-provisioning constraints Adds a warning on the snapshot page that volume restore is supported only for thin volumes from OpenEBS v4.4.0 onwards, and records on the restore page that the StorageClass used for the restore must also be thin provisioned. Signed-off-by: Bala Harish A --- .../local-pv-lvm/advanced-operations/lvm-snapshot.md | 6 ++++++ .../local-pv-lvm/advanced-operations/lvm-volume-restore.md | 1 + 2 files changed, 7 insertions(+) diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md index 10bbc958f..68ee279b3 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-snapshot.md @@ -14,6 +14,12 @@ The LVM driver supports creating snapshots of the LVM volumes. This requires the - Snapshots created by the LVM driver are ReadOnly by default as opposed to the ReadWrite snapshots created by default by `lvcreate` command. - The size of snapshot will be set to the size of the origin volume. +:::warning +Volume Restore is supported only for thin volumes created from snapshots, using OpenEBS v4.4.0 or later. Restore from a snapshot of a thick-provisioned volume is not supported. + +Refer to [Volume Restore](lvm-volume-restore.md) for the complete list of requirements. +::: + ## Using an Existing Snapshot Controller Local PV LVM runs a `snapshot-controller` container by default. A cluster should run only one snapshot controller, so if one is already managed at the cluster level, disable the bundled one: diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-volume-restore.md b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-volume-restore.md index c9e1eb46b..cdcd23edb 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-volume-restore.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-lvm/advanced-operations/lvm-volume-restore.md @@ -22,6 +22,7 @@ Before performing the restore operation, ensure the following requirements are m - A compatible StorageClass is available for the restore operation. - A snapshot has been created for the source volume. Follow the steps outlined in the [Volume Snapshots Documentation](lvm-snapshot.md) to create a volume snapshot. - Only thin snapshot restores are supported. To verify whether a snapshot is thin or thick, describe the LVM snapshot Custom Resource (CR) and check the `spec.thinProvision` field. +- The StorageClass used for the restore must also set `thinProvision: "yes"`. A restore into a thick provisioned StorageClass is rejected, even when the snapshot itself is thin. - The restore volume request capacity must match the snapshot logical volume (LV) size. To verify snapshot LV size, describe LVM snapshot CR and check the snapshot LV size in `status.lvSize` field. - The Local PV LVM volume group name must match the snapshot’s volume group (`spec.volGroup`). From 87b0262106cc0515b04b722ff1042d6ed74cec96 Mon Sep 17 00:00:00 2001 From: Bala Harish A Date: Tue, 25 Aug 2026 14:53:00 +0530 Subject: [PATCH 8/9] docs: address review feedback on v4.6 release notes and ZFS parameters Removes the ZFS scheduler parameter documentation, drops the duplicate out-of-tree provisioner row from the component table, reworks the Replicated PV Mayastor entries to match the v2.12.0 release notes including the Volume and Nexus Label V2 layout, and adds the Local PV ZFS image URL rendering fix. Signed-off-by: Bala Harish A --- docs/main/releases.md | 61 ++++++++++++++----- .../configuration/zfs-create-storageclass.md | 13 +--- .../zfs-storageclass-parameters.md | 10 --- .../local-pv-zfs/configuration/zfs-usage.md | 2 - 4 files changed, 48 insertions(+), 38 deletions(-) diff --git a/docs/main/releases.md b/docs/main/releases.md index 060ef46b6..5e861c252 100644 --- a/docs/main/releases.md +++ b/docs/main/releases.md @@ -17,11 +17,10 @@ The status of the various components as of v4.6 are as follows: | Component Type | Component | Version | Status | | :--- | :--- | :--- | :--- | | Replicated Storage | Replicated PV Mayastor | 2.12.0 | Stable | -| Local Storage | Local PV Hostpath | 4.6.0 | Stable | +| Local Storage (non-CSI) | Local PV Hostpath | 4.6.0 | Stable | | Local Storage | Local PV LVM | 1.10.0 | Stable | | Local Storage | Local PV ZFS | 2.11.0 | Stable | | Local Storage | Local PV Rawfile | 0.15.0 | Experimental | -| Out-of-tree (External Storage) Provisioners | Local PV Hostpath | 4.6.0 | Stable | | Other Components | CLI | 4.6.0 | — | ## What’s New @@ -38,10 +37,6 @@ The status of the various components as of v4.6 are as follows: Replicated PV Mayastor now includes the Eventing Aggregator, a new component that collects cluster events and makes them available for querying through the `get events` command in the Mayastor kubectl plugin. Events can be retrieved from Loki, directly from NATS JetStream, or from a previously collected support bundle for offline analysis, and can be filtered by category, action, node, target, component, pool, volume, replica, and state. Cluster events are also included in the support bundle produced by `kubectl openebs dump system`. The Eventing Aggregator is enabled by default and works with or without Loki deployed. -- **TLS for Mayastor Service Endpoints** - - Replicated PV Mayastor now supports TLS for its service endpoints, starting with the REST API, along with the CSI controller, CSI node, DiskPool operator, metrics exporter, and kubectl plugin clients. Certificates can be managed in three ways: a transient self-signed certificate generated by the server at startup, self-signed certificates generated by the Helm chart, or certificates provisioned and rotated by cert-manager. Mutual TLS is supported, and the server reloads certificates without a restart when they are renewed. TLS is opt-in through `security.tls.enabled`, and plain-text communication remains the default. - - **Best-Effort Snapshot Restore Policy** A new `snapshotRestorePolicy` StorageClass parameter controls how a snapshot restore behaves when not every replica pool can host a clone. With the default `strict` policy, every requested replica must be cloned from the snapshot. With `bestEffort`, the restore proceeds as long as at least one clone succeeds; the volume comes up under-replicated and the remaining replicas are filled in through a normal rebuild. This allows a restore to succeed when a source pool is full or otherwise unable to host a clone. @@ -54,6 +49,16 @@ The status of the various components as of v4.6 are as follows: Building on the pool error and alert visibility introduced in the previous release, you can now clear recorded DiskPool errors using the Mayastor kubectl plugin. This allows a pool to be returned to normal reporting after the underlying condition has been resolved. +- **Volume and Nexus Label V2** + + A new versioned on-disk label layout, V2, is introduced alongside the existing V1 layout. V2 reduces the metadata partition reserved at the front of every replica from 4 MiB to 3 MiB and aligns volume capacity to 1 MiB boundaries. It also resolves block-mode backup compatibility with Kasten K10. + + For a V2 volume, the requested size is rounded up to the next 1 MiB boundary. A size that is already a multiple of 1 MiB is unchanged, so a request for `10Gi` stays `10Gi`, while a decimal quantity such as `10G` is not MiB-aligned and is rounded up. Each replica additionally reserves 8 MiB for the metadata partition, and the replica total is then aligned up to the cluster size of the pool, which is 4 MiB by default. For example, a 10 MiB volume on a pool with the default cluster size is exposed as a 10 MiB device while each replica consumes 20 MiB of pool space. + + A volume created with a size that is not 1 MiB aligned reports a slightly larger size than requested, which is expected behaviour. Because V2 also adds the 8 MiB metadata reservation for every replica, a V2 volume consumes more pool space than the raw requested size, so size your pools accordingly. + + The cluster-wide label version is negotiated automatically as the lowest version supported across all registered io-engines, and only ever moves up, so the control plane never creates a label that a node cannot understand. In a cluster where some nodes do not yet support V2, the negotiated version stays at V1 until every node supports it. Existing volumes remain on V1 and require no migration, and resize and snapshot operations preserve the label version of the volume. + ### Local Storage - **API Server for Local PV Rawfile** @@ -78,13 +83,21 @@ The status of the various components as of v4.6 are as follows: ### Replicated Storage -- **Expanded NVMe-oF Transport Tuning** +- **TLS Hardening and Certificate Auto-Reload** + + Replicated PV Mayastor now supports TLS for its service endpoints, starting with the REST API, along with the CSI controller, CSI node, DiskPool operator, metrics exporter, and kubectl plugin clients. Certificates can be managed in three ways: a transient self-signed certificate generated by the server at startup, self-signed certificates generated by the Helm chart, or certificates provisioned and rotated by cert-manager. Certificates are hot-reloaded on rotation without a restart, TLS discovery is enabled by default, plain HTTP on the REST service is restricted to health probes only, and the Helm chart exposes the TLS configuration for the public API. + +- **RDMA Capability Detection** - Additional NVMe-oF transport parameters are now exposed through the Helm chart for both TCP and RDMA, including maximum I/O size, I/O unit size, in-capsule data size, and the RDMA data work-request pool size. These options allow the transport to be tuned for specific network and workload characteristics. + The CSI node now checks for the `nvme_rdma` kernel module before reporting RDMA capability, and node transport capabilities are propagated through registration. Previously a node could be treated as RDMA-capable without the required kernel support. -- **NetworkPolicy for the REST API** +- **Asynchronous Bdev Destruction** - The Helm chart now creates a Kubernetes NetworkPolicy for the REST API service that blocks plain HTTP traffic to the API. This is enabled by default and can be disabled through `security.networkPolicy.enabled`. + Block device destruction in the io-engine is now asynchronous, improving the responsiveness of pool and replica teardown. + +- **Additional Helm Chart Options** + + Additional environment variables are exposed for component customisation, the api-rest health probes use an `initialDelaySeconds` of 1 for faster readiness, and the maximum Loki ingestion limits have been increased. - **Clearer Impact Reporting for Purge Operations** @@ -124,13 +137,29 @@ The status of the various components as of v4.6 are as follows: Resolved several issues affecting DiskPool availability during device and replica lifecycle events, including a race between pool deregistration and reload, and a race when listing pools while replicas were being destroyed. -- **RDMA Capability Detection on Nodes** +- **Frozen I/O During Nexus Shutdown** - Resolved an issue where a node could be treated as RDMA-capable without the required kernel support. The CSI node driver now checks for the `nvme_rdma` kernel module before reporting RDMA capability. + Resolved an issue where I/O could remain frozen when a shutdown nexus was unshared. Outstanding I/O is now aborted correctly, child device closure is awaited, and devices are no longer detached on transient I/O submission errors. -- **Frozen I/O During Nexus Shutdown** +- **Nexus Size Miscalculation** + + Resolved an issue where the nexus block device size could be off by one block. Volume sizing now accounts for label metadata, so the usable capacity always meets the requested size. + +- **Pool Capacity Accounting for Snapshot Clones** + + Resolved an issue where pool capacity tallying did not account for the creation of snapshot clones, which could lead to over-commitment of pool space. + +- **Readiness Probe Startup Race** + + Resolved an issue where a readiness failure could be cached before the first successful probe, causing api-rest to be incorrectly reported as not ready. - Resolved an issue where I/O could remain frozen when a shutdown nexus was unshared. Outstanding I/O is now aborted correctly, and devices are no longer detached on I/O submission errors. +- **Node Rebuild Count Accuracy** + + Resolved an issue where the node rebuild count was not refreshed on single nexus updates, improving the accuracy of rebuild throttling. + +- **SPDK Fixes** + + Updated SPDK with fixes for a null pointer dereference and IPv6 transport handling, and resolved an interrupt-mode reactor teardown issue. ### Local Storage @@ -154,6 +183,10 @@ The status of the various components as of v4.6 are as follows: Resolved an issue where the namespace field in the generated ServiceMonitor manifest was incorrectly indented. +- **Image URL Rendering for Local PV ZFS** + + Resolved an issue where image URLs in rendered manifests were not quoted, which could break rendering for registries whose URLs contain characters that YAML treats specially. + ## Breaking Changes ### Local Storage diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-create-storageclass.md b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-create-storageclass.md index f072c6294..b4b6c2dee 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-create-storageclass.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-create-storageclass.md @@ -124,18 +124,7 @@ The provisioner name for ZFS driver is "zfs.csi.openebs.io", we have to use this ## Scheduler -The ZFS driver has its own scheduler which will try to distribute the PV across the nodes so that one node should not be loaded with all the volumes. Currently the driver supports two scheduling algorithms: VolumeWeighted and CapacityWeighted, in which it will try to find a ZFS pool which has less number of volumes provisioned in it or less capacity of volume provisioned out of a pool respectively, from all the nodes where the ZFS pools are available. - -To select a scheduler, add the `scheduler` parameter in the storage class and give its value accordingly: - -``` -parameters: - scheduler: "VolumeWeighted" - fstype: "zfs" - poolname: "zfspv-pool" -``` - -If the `scheduler` parameter is not provided in the storage class, `CapacityWeighted` is used. Refer to [StorageClass Parameters](zfs-storageclass-parameters.md#scheduler-optional-parameter) and [StorageClass with k8s Scheduler](zfs-usage.md#storageclass-with-k8s-scheduler) for more details. +The ZFS driver has its own scheduler which will try to distribute the PV across the nodes so that one node should not be loaded with all the volumes. Currently the driver supports two scheduling algorithms: VolumeWeighted and CapacityWeighted, in which it will try to find a ZFS pool which has less number of volumes provisioned in it or less capacity of volume provisioned out of a pool respectively, from all the nodes where the ZFS pools are available. Refer [StorageClass With K8s Scheduler](https://github.com/openebs/zfs-localpv/blob/HEAD/docs/storageclasses.md#storageclass-with-k8s-scheduler) to learn how to select a scheduler via storage class. Once it can find the node, it will create a PV for that node and also create a ZFSVolume custom resource for the volume with the NODE information. The watcher for this ZFSVolume CR will get all the information for this object and creates a ZFS dataset (zvol) with the given ZFS property on the mentioned node. diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-storageclass-parameters.md b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-storageclass-parameters.md index 522b873a7..7a31092a0 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-storageclass-parameters.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-storageclass-parameters.md @@ -43,7 +43,6 @@ These parameters are set under `parameters` in the StorageClass. | `thinProvision` | Optional | `yes`, `no` | Both | | `quotatype` | Optional | `quota`, `refquota` | Dataset | | `shared` | Optional | `yes`, `no` | Both | -| `scheduler` | Optional | `CapacityWeighted`, `VolumeWeighted` | Both | In the **Applies To** column: @@ -119,15 +118,6 @@ This parameter is applicable if fstype provided is "zfs" otherwise it will be ig Shared specifies whether the volume can be shared among multiple pods. If it is not set to "yes", then the ZFS-LocalPV Driver will not allow the volumes to be mounted by more than one pods. The default value is "no" if shared is not provided in the storageclass. -## Scheduler (Optional Parameter) - -Scheduler selects the algorithm that the Local PV ZFS driver uses to pick the node on which a volume is provisioned: - -- `CapacityWeighted` picks the node whose matching ZFS pool has the least capacity already provisioned from it. -- `VolumeWeighted` picks the node whose matching ZFS pool has the fewest volumes provisioned on it. - -If scheduler is not provided in the storageclass, `CapacityWeighted` is used. Refer to [StorageClass with k8s Scheduler](zfs-usage.md#storageclass-with-k8s-scheduler) for more details. - ## AllowVolumeExpansion (Optional) Volumes provisioned by a StorageClass can be expanded only when `allowVolumeExpansion` is set to `true`. If this field is not specified, volume expansion is not supported. diff --git a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-usage.md b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-usage.md index 59ea90e57..7083ee61f 100644 --- a/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-usage.md +++ b/docs/main/user-guides/local-storage-user-guide/local-pv-zfs/configuration/zfs-usage.md @@ -112,8 +112,6 @@ parameters: CapacityWeighted is the default scheduler in zfs-localpv driver, so even if we don't use scheduler parameter in storage-class, driver will pick the node where total provisioned volumes have occupied less capacity from the given pool. On the other hand for using VolumeWeighted scheduler, we have to specify it under scheduler parameter in storage-class. Then driver will pick the node to create volume where ZFS Pool is less loaded with the volumes. Here, it just checks the volume count and creates the volume where less volume is configured in a given ZFS Pool. It does not account for other factors like available CPU or memory while making scheduling decisions. -Refer to [StorageClass Parameters](zfs-storageclass-parameters.md#scheduler-optional-parameter) for the supported values of the `scheduler` parameter. - In case where you want to use node selector/affinity rules on the application pod or have CPU/Memory constraints, the Kubernetes scheduler should be used. To make use of Kubernetes scheduler, we can set the volumeBindingMode as WaitForFirstConsumer in the storage class: ```yaml From 8dc0b26a928b02ef608f6bf12529f026d41bf25f Mon Sep 17 00:00:00 2001 From: Bala Harish A Date: Tue, 25 Aug 2026 15:16:53 +0530 Subject: [PATCH 9/9] docs: reword the version banner shown on older documentation versions The banner used the Docusaurus default text, which stated that older versions are no longer actively maintained. Overrides the two theme translation strings in i18n/en/code.json so the banner instead notes that an older version is being viewed and links to the latest documentation. Signed-off-by: Bala Harish A --- docs/i18n/en/code.json | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 docs/i18n/en/code.json diff --git a/docs/i18n/en/code.json b/docs/i18n/en/code.json new file mode 100644 index 000000000..70d0a6219 --- /dev/null +++ b/docs/i18n/en/code.json @@ -0,0 +1,10 @@ +{ + "theme.docs.versions.unmaintainedVersionLabel": { + "message": "You are viewing an older version of the OpenEBS documentation.", + "description": "The label used to tell the user that he's browsing an unmaintained doc version" + }, + "theme.docs.versions.latestVersionSuggestionLabel": { + "message": "For the most up-to-date content, see the {latestVersionLink}.", + "description": "The label used to tell the user to check the latest version" + } +}