# External Ceph Cluster for Kubernetes

Create an external Ceph Cluster for all your Kubernetes clusters.

**INSTALL CEPHADM**


- Prerequisites:
    - 3 VMs, with unused disks 
    - Python 3, Cephadm is a Python-based utility
    - Systemd, runs Ceph daemons like mons, mgrs, mds
    - Podman or Docker some Cephadmn tasks are executed in containers
    - Time synchronization is important for distributed systems ensure Chrony or ntpd
    - LVM2 Ceph for provisioning and managing storage devices

- Install Cephadm on first server which will be the Ceph manager
- Set Ceph version and download the cephadm binary
    
    ```
    CEPH_RELEASE=19.2.2
    curl --silent --remote-name --location https://download.ceph.com/rpm-${CEPH_RELEASE}/el9/noarch/cephadm
    ```
    
- Add the Ceph repo and install Cephadm to the system and verify version
    
    ```bash
    ./cephadm add-repo --release squid
    ./cephadm install
    cephadm --version
    ```
    

**CREATE CEPH CLUSTER**

- Bootstrap a new Ceph cluster
    
    ```bash
    cephadm bootstrap --mon-ip 172.30.0.61 --ssh-user debian
    
    ===Output Truncated===
    Ceph Dashboard is now available at:
    
                 URL: https://ceph-1:8443/
                User: admin
            Password: ek1sfbgrl2
    ===Output Truncated===
    ```
    

- Login to Ceph dashboard via https://ceph-1:8443/

- Install Ceph tools to manage cluster
    
    ```
    cephadm install ceph-common
    
    ceph status
    
      cluster:
        id:     e376a9b4-4696-11f0-8594-2264c6d97862
        health: HEALTH_WARN
                OSD count 0 < osd_pool_default_size 3
    
      services:
        mon: 1 daemons, quorum ceph-1 (age 10m)
        mgr: ceph-1.jbqokr(active, since 9m)
        osd: 0 osds: 0 up, 0 in
    
      data:
        pools:   0 pools, 0 pgs
        objects: 0 objects, 0 B
        usage:   0 B used, 0 B / 0 B avail
        pgs:
    ```
    
- Allow Ceph admin node to SSH into other nodes without a password
    
    ```
    ssh-copy-id -f -i /etc/ceph/ceph.pub debian@172.30.0.62
    ssh-copy-id -f -i /etc/ceph/ceph.pub debian@172.30.0.63
    ```
    

- Add new hosts to cluster, admin label ensures node can run ceph mgt taks, also allows for HA and fault tolerance.
    
    ```
    ceph orch host add ceph-3 172.30.0.62 --labels _admin
    ceph orch host add ceph-3 172.30.0.63 --labels _admin
    
    ceph status
    ```

- Add OSDs, to fix the error with OSDs

    ```
    ceph status
    ceph orch apply osd --all-available-devices
    
    ceph osd ls
    ceph osd df
    ```

**ENABLE BLOCK STORAGE**

- Create a pool in the Ceph dashboards, from which to provision block storage
- Create a pool(name rbd) set replication size to 3 and application to RBD or RADOS Block Device which is Ceph's block storage application

**ENABLE FS STORAGE**

- To create Filesystem storage MDS or Metadata Server is required
- MDS is a Ceph daemon that manages metadata for CephFS, Ceph's distributed file system
- First set MDS labels on servers that can be used to run mds daemons
- Create a cephfs filesystem, which will auto create the cephfs data and metadata pools as well as the mds daemons
    
    ```
    ceph fs volume create cephfs --placement="label:mds"
    
    ceph -s
      cluster:
        id:     e376a9b4-4696-11f0-8594-2264c6d97862
        health: HEALTH_OK
    
      services:
        mon: 3 daemons, quorum ceph-1,ceph-2,ceph-3 (age 17m)
        mgr: ceph-1.jbqokr(active, since 33m), standbys: ceph-2.vjlwim
        mds: 1/1 daemons up, 1 standby
        osd: 3 osds: 3 up (since 14m), 3 in (since 14m)
    
      data:
        volumes: 1/1 healthy
        pools:   4 pools, 35 pgs
        objects: 24 objects, 579 KiB
        usage:   82 MiB used, 120 GiB / 120 GiB avail
        pgs:     35 active+clean
    
    ceph osd pool ls
    .mgr
    rbd
    cephfs.cephfs.meta
    cephfs.cephfs.data
    ```
    

**ENABLE OBJECT STORAGE**

- Create a RADOS Gateway (radosgw or RGW)
- A RADOS Gateway is Ceph's object storage gateway that provides RESTful APIs compatible with Amazon S3
- Create and use S3 buckets with the Ceph cluster
- Add 'rgw' label to GW eligible nodes from the dashboard
- Create a RADOS Gateway on selected ceph nodes
    
    ```
    ceph orch apply rgw radosgw '--placement=label:rgw count-per-host:1' --port=8001
    
    ceph osd pool ls
    .mgr
    rbd
    cephfs.cephfs.meta
    cephfs.cephfs.data
    .rgw.root
    default.rgw.log
    default.rgw.control
    default.rgw.meta
    ```
    

**INSTALL ROOK IN KUBERNETES**

- Rook is Open-Source, Cloud-Native Storage for Kubernetes and is typically how you install Ceph in a Kubernetes cluster.
- Clone the Rook repository to your local machine, node ceph node
    
    ```
    git clone --single-branch --branch master https://github.com/rook/rook.git
    ```
    
- Install the Rook CRDs and Rook Operator
    
    ```
    cd rook/deploy/examples
    kubectl create -f crds.yaml -f common.yaml -f operator.yaml
    
    kubectl get pods -n rook-ceph
    
    NAME                                           READY   STATUS    RESTARTS   AGE
    rook-ceph-operator-59dcf6d55b-p7hk5            1/1     Running   0          66m
    ```
    
- Install a Ceph cluster of type external, meaning Rook connects to an external ceph cluster
    
    ```
    cd ../charts/rook-ceph-cluster
    helm repo add rook-release https://charts.rook.io/release
    helm install --namespace rook-ceph rook-ceph-cluster \
    --set operatorNamespace=rook-ceph rook-release/rook-ceph-cluster -f values-external.yaml
    
    kubectl get cephcluster
    
    kubectl --namespace rook-ceph get cephcluster
    NAME        DATADIRHOSTPATH   MONCOUNT   AGE   PHASE        MESSAGE                                             HEALTH   EXTERNAL   FSID
    rook-ceph   /var/lib/rook     3          27s   Connecting   Attempting to connect to an external Ceph cluster            true
    
    kubectl get pods -n rook-ceph
    NAME                                           READY   STATUS    RESTARTS   AGE
    csi-cephfsplugin-4vb4w                         3/3     Running   0          58m
    csi-cephfsplugin-provisioner-57467b74d-df826   6/6     Running   0          58m
    csi-rbdplugin-7wsx9                            3/3     Running   0          58m
    csi-rbdplugin-provisioner-7c47cc896b-tl2zj     6/6     Running   0          58m
    rook-ceph-operator-59dcf6d55b-p7hk5            1/1     Running   0          66m
    ```
    
- Import provider cluster data that will initiate the connection of the ceph cluster with Rook
- We need to extract some information form the ceph cluster
- On ceph manager node clone the Rook git repo

```
git clone --single-branch --branch master https://github.com/rook/rook.git

cd rook/deploy/examples/external

ls
cluster-external.yaml
common-external.yaml
create-external-cluster-resources-tests.py
create-external-cluster-resources.py
dashboard-external-http.yaml
dashboard-external-https.yaml
import-external-cluster.sh
object-bucket-claim-delete.yaml
object-external.yaml
storageclass-bucket-delete.yaml

python3 create-external-cluster-resources.py --rbd-data-pool-name rbd --cephfs-filesystem-name cephfs --rgw-endpoint 172.30.0.61:8000 --namespace rook-ceph --format bash
```

- Copy the output and save it in a ceph-cluster.env file
- Import the data into Rook
- On your local machine with kuberentes cluster context
    
    ```
    cd rook/deploy/examples/external
    
    ls
    cluster-external.yaml                      dashboard-external-https.yaml
    common-external.yaml                       import-external-cluster.sh
    create-external-cluster-resources-tests.py object-bucket-claim-delete.yaml
    create-external-cluster-resources.py       object-external.yaml
    dashboard-external-http.yaml               storageclass-bucket-delete.yaml
    
    source ceph-cluster.env
    
    ./import-external-cluster.sh
    cluster namespace rook-ceph already exists
    secret/rook-ceph-mon created
    configmap/rook-ceph-mon-endpoints created
    configmap/external-cluster-user-command created
    secret/rook-csi-rbd-node created
    secret/rook-csi-rbd-provisioner created
    secret/rgw-admin-ops-user created
    secret/rook-csi-cephfs-node created
    secret/rook-csi-cephfs-provisioner created
    storageclass.storage.k8s.io/ceph-rbd created
    storageclass.storage.k8s.io/cephfs created
    
    kubectl -n rook-ceph  get CephCluster
    NAME        DATADIRHOSTPATH   MONCOUNT   AGE   PHASE       MESSAGE                          HEALTH      EXTERNAL   FSID
    rook-ceph   /var/lib/rook     3          10m   Connected   Cluster connected successfully   HEALTH_OK   true       e376a9b4-4696-11f0-8594-2264c6d97862
    
    
    kubectl -n rook-ceph get sc
    NAME                   PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
    ceph-rbd               rook-ceph.rbd.csi.ceph.com      Delete          Immediate              true                   56s
    cephfs                 rook-ceph.cephfs.csi.ceph.com   Delete          Immediate              true                   56s
    local-path (default)   rancher.io/local-path           Delete          WaitForFirstConsumer   false                  28m
    ```
    

**USE STORAGE**

- Deploy statefulset to consume block storage
    
    ```
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: test-deploy
      namespace: default
    spec:
      selector:
        matchLabels:
          app: test-deploy 
      serviceName: "test-deploy"
      replicas: 3
      template:
        metadata:
          labels:
            app: test-deploy
        spec:
          terminationGracePeriodSeconds: 10
          containers:
          - name: test-deploy
            image: nginx
            ports:
            - containerPort: 80
              name: web
            volumeMounts:
            - name: www
              mountPath: /usr/share/nginx/html
      volumeClaimTemplates:
      - metadata:
          name: www
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: "ceph-rbd"
          resources:
            requests:
              storage: 2Gi
    
    kubectl apply -f k8s-test-deploy.yml -n default
    
    kubectl get pods -n default
    kubectl get pvc -n default
    ```
    
- Create a CephObjectStore connect to rados gateway and Storage Class to provision object storage in the k8s cluster
- Create ObjectBucketClaim to create an S3 bucket
    
    ```
    apiVersion: ceph.rook.io/v1
    kind: CephObjectStore
    metadata:
      name: external-object-store
      namespace: rook-ceph
    spec:
      gateway:
        port: 8001
        externalRgwEndpoints:
          - ip: 172.30.0.61
          - ip: 172.30.0.63
    
    ---
    
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
       name: rook-ceph-bucket
    provisioner: rook-ceph.ceph.rook.io/bucket
    reclaimPolicy: Delete
    parameters:
      objectStoreName: external-object-store
      objectStoreNamespace: rook-ceph
    
    ---
    apiVersion: objectbucket.io/v1alpha1
    kind: ObjectBucketClaim
    metadata:
      name: ceph-bucket
    spec:
      generateBucketName: ceph-bkt
      storageClassName: rook-ceph-bucket
    
    
    k get sc,ObjectBucketClaim
    NAME                                               PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
    storageclass.storage.k8s.io/ceph-rbd               rook-ceph.rbd.csi.ceph.com      Delete          Immediate              true                   6m26s
    storageclass.storage.k8s.io/cephfs                 rook-ceph.cephfs.csi.ceph.com   Delete          Immediate              true                   6m26s
    storageclass.storage.k8s.io/local-path (default)   rancher.io/local-path           Delete          WaitForFirstConsumer   false                  33m
    storageclass.storage.k8s.io/rook-ceph-bucket       rook-ceph.ceph.rook.io/bucket   Delete          Immediate              false                  63s
    
    NAME                                            AGE
    objectbucketclaim.objectbucket.io/ceph-bucket   63s
    ```
    

- Test bucket
    
    ```
    export AWS_HOST=$(kubectl -n default get cm ceph-bucket -o jsonpath='{.data.BUCKET_HOST}')
    export PORT=$(kubectl -n default get cm ceph-bucket -o jsonpath='{.data.BUCKET_PORT}')
    export BUCKET_NAME=$(kubectl -n default get cm ceph-bucket -o jsonpath='{.data.BUCKET_NAME}')
    export AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-bucket -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 --decode)
    export AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-bucket -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 --decode)
    
    
    ❯ mc alias set ceph-rgw http://$AWS_HOST:$PORT $AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY
    Added `ceph-rgw` successfully.
    ❯ mc ls ceph-rgw
    [2025-06-11 10:50:06 CEST]     0B ceph-bkt-33aaa50b-cfa0-4d7b-94a8-fb62cca60ca0/
    ❯ mc cp ceph-cluster.env ceph-rgw/ceph-bkt-33aaa50b-cfa0-4d7b-94a8-fb62cca60ca0/
    .../ceph-cluster.env: 1.51 KiB / 1.51 KiB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 506 B/s 3s❯ mc ls ceph-rgw/ceph-bkt-33aaa50b-cfa0-4d7b-94a8-fb62cca60ca0/
    [2025-06-11 10:54:51 CEST] 1.5KiB STANDARD ceph-cluster.env
    ```