Using Alibaba Cloud Distributed Storage in Self-built K8S Clusters

Tags:
Categories:

Introduction

This article was written on 2024.06.14, introducing how to use Alibaba Cloud distributed storage in self-built clusters on Alibaba Cloud. Relevant documentation links are provided at the end. The official Alibaba Cloud documentation is in Chinese, but the Alibaba Cloud storage plugin installation is on GitHub and currently only has English documentation. Readers who can are advised to read the original text.

Storage Plugin Installation

  1. Create a custom permission policy: https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/docs/ram-policies/disk.json
  2. Create a RAM role, grant the custom permission policy, and temporarily store the accesskey and secret
    1. kubectl create secret -n kube-system generic csi-access-key --from-literal=id='{id}' --from-literal=secret='{secret}'
  3. Install the CSI driver. There is no helm chart available; it must be installed locally (as of 20240613).
    1. git clone https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver.git
    2. cd alibaba-cloud-csi-driver/deploy
    3. If installing on a self-built cluster on Alibaba Cloud ECS, you can directly execute the next command. If not, please read: https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/docs/install.md
    4. helm upgrade --install alibaba-cloud-csi-driver ./chart --values chart/values-ecs.yaml --namespace kube-system
  4. Verify: watch kubectl get pods -n kube-system -l app=csi-plugin

Storage Type Selection Reference

  • The minimum capacity for creating an ECS cloud disk is 20GB, with 3000 IOPS. This capacity is relatively large and not very cost-effective.
    • Cloud Disk Dynamic Persistent Volume
      • Official Documentation:
        • Cloud disks do not support cross-zone usage, are non-shared storage, and can only be mounted by one Pod at a time. (Testing shows they can be mounted by multiple pods of the same deployment)
        • The cloud disk type and ECS type must match for successful mounting; otherwise, mounting will fail. For the matching relationship between cloud disk types and ECS types, see Instance Type Families.
        • When deploying applications, automatically create a PV to purchase a cloud disk through StorageClass. If you have already purchased a cloud disk, it is recommended to use a cloud disk static persistent volume.
        • The requested cloud disk size cannot exceed the single disk capacity range.
        • When a Pod is recreated, it will remount the original cloud disk. If scheduling to the original availability zone is not possible due to other constraints, the Pod will remain in a Pending state.
        • Dynamically created cloud disks are pay-as-you-go.
      • Other Test Summaries:
        • Although cloud disks can be mounted by multiple pods, only one pod can read and write; other pods cannot. Therefore, accessModes in the PVC can only be set to ReadWriteOnce; modifying it will not yield the correct result.
        • If the StorageClass’s reclaimPolicy is set to Delete, the cloud disk can also be automatically deleted when the PVC is deleted.
        • If the StorageClass’s reclaimPolicy is set to Retain, the cloud disk will not be automatically deleted when the PVC is deleted and must be manually deleted in the cluster and the Alibaba Cloud console.
      • Difficult to find suitable use cases.
    • Cloud Disk Static Persistent Volume
      • Official Documentation:
        • Manually create PV and PVC.
        • Cloud disks do not support cross-zone usage, are non-shared storage, and can only be mounted by one Pod at a time.
        • The cloud disk type and ECS type must match for successful mounting.
        • You can select a cloud disk in the same region and availability zone as the cluster that is in the pending mount state.
  • NAS has relatively high operation latency, with the best performance at 2ms and deep storage at 10ms. It is billed on a pay-as-you-go basis, and its read/write performance is higher than that of Object Storage Service (OSS).
  • OSS Persistent Volume, https://help.aliyun.com/zh/ack/ack-managed-and-ack-dedicated/user-guide/oss-volume-overview-1?spm=a2c4g.11186623.0.0.43166a351NbtvU
    • OSS is shared storage and can provide shared storage services to multiple Pods simultaneously.
    • (As of 20240613) Currently supports CentOS, Alibaba Cloud Linux, ContainerOS, and Anolis OS.
    • When using a data volume, each application uses an independent PV name.
    • The OSS data volume is a FUSE file system mounted using the ossfs file.
      • Suitable for file reading scenarios, such as reading configuration files, videos, image files, etc.
      • Not suitable for application scenarios that involve writing files. If file writing is needed, it is recommended to use the SDK to implement write operations or use the NAS storage volume service.
    • ossfs can optimize its performance in caching, permissions, and other aspects by adjusting configuration parameters.
    • ossfs Usage Limitations:
      • Random or append write file operations will cause the entire file to be rewritten.
      • Metadata operations like listing a directory have poor performance because they require remote access to the OSS server.
      • File and directory rename operations are not atomic.
      • When multiple clients mount the same OSS Bucket, users must coordinate the behavior of each client themselves, for example, avoiding multiple clients writing to the same file.
      • Hard links are not supported.
      • When the CSI plugin version is below v1.20.7, it only detects local modifications and cannot detect external modifications made by other clients or tools.
      • To avoid increasing system load, do not use it in high-concurrency read/write scenarios.
  • For hybrid clusters (where some nodes do not belong to Alibaba Cloud), only NAS and OSS static volumes can be used.
  • Cloud disks, NAS, and OSS all have regional restrictions.

Summary: Cloud disks are requested and mounted as entire hard disks, making sharing inconvenient. OSS operates at the file granularity, has performance issues with high-concurrency reads/writes, and has limited supported systems.

  • Cloud disks are suitable for scenarios like databases that require large capacity and high performance.
  • NAS can be chosen for other scenarios with lower performance requirements.
  • OSS is not suitable for high-concurrency write scenarios in Alibaba Cloud clusters but can be used for concurrent read scenarios.

Alibaba Cloud’s official documentation has issues with inconsistent locations and contradictions. Readers need to judge based on the document date themselves, as some unsupported features may have become supported with version updates, requiring some experimentation.

Operational Steps

This is the official Alibaba Cloud guidance documentation. After installing the Alibaba Cloud storage plugin as instructed above, you can proceed with deployment testing according to Using NAS Static Persistent Volumes.

Note: k3s users may encounter issues with local-path-storage, with possible error messages such as:

  • failed to provision volume with StorageClass “local-path”: claim.Spec.Selector is not supported
  • Waiting for a volume to be created either by the external provisioner ’localplugin.csi.alibabacloud.com’ or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

You need to set the storageClassName in the persistentVolumeClaim to empty to avoid using k3s’s default local-path-storage.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-nas
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
  selector:
    matchLabels:
      alicloud-pvname: pv-nas
  storageClassName: ""

References