kubectl logs -f rook-ceph-mgr-a-76ff44c7fc-frmm2
2018-11-02 11:07:59.012068 I | ceph-mgr: 2018-11-02 11:07:59.011920 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:01.014374 I | ceph-mgr: 2018-11-02 11:08:01.014234 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:03.016552 I | ceph-mgr: 2018-11-02 11:08:03.016406 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:05.019184 I | ceph-mgr: 2018-11-02 11:08:05.019045 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:07.021337 I | ceph-mgr: 2018-11-02 11:08:07.021195 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:09.023518 I | ceph-mgr: 2018-11-02 11:08:09.023404 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:09.702423 I | ceph-mgr: 2018-11-02 11:08:09.702303 7fbfe3433700 0 -- 10.233.98.6:6800/15 >> 10.233.98.4:0/4019921133 conn(0x5646ad6bf000 :6800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer
2018-11-02 11:08:10.391671 I | ceph-mgr: 2018-11-02 11:08:10.391562 7fbfe2431700 0 -- 10.233.98.6:6800/15 >> 10.233.115.0:0/1294457323 conn(0x5646ac719000 :6800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer
2018-11-02 11:08:11.026142 I | ceph-mgr: 2018-11-02 11:08:11.026003 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:13.027979 I | ceph-mgr: 2018-11-02 11:08:13.027854 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:15.030524 I | ceph-mgr: 2018-11-02 11:08:15.030381 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:17.032671 I | ceph-mgr: 2018-11-02 11:08:17.032524 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:19.034784 I | ceph-mgr: 2018-11-02 11:08:19.034636 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:21.037393 I | ceph-mgr: 2018-11-02 11:08:21.037241 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:23.039654 I | ceph-mgr: 2018-11-02 11:08:23.039510 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:23.405036 I | ceph-mgr: 2018-11-02 11:08:23.404884 7fbfe2431700 0 -- 10.233.98.6:6800/15 >> 10.233.98.4:0/104616375 conn(0x5646add62800 :6800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer
2018-11-02 11:08:25.042001 I | ceph-mgr: 2018-11-02 11:08:25.041862 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:27.044121 I | ceph-mgr: 2018-11-02 11:08:27.043975 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:29.046263 I | ceph-mgr: 2018-11-02 11:08:29.046135 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:31.048681 I | ceph-mgr: 2018-11-02 11:08:31.048536 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:33.050808 I | ceph-mgr: 2018-11-02 11:08:33.050669 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:35.053481 I | ceph-mgr: 2018-11-02 11:08:35.053330 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:37.055637 I | ceph-mgr: 2018-11-02 11:08:37.055493 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:39.057774 I | ceph-mgr: 2018-11-02 11:08:39.057623 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:41.060498 I | ceph-mgr: 2018-11-02 11:08:41.060354 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:43.062571 I | ceph-mgr: 2018-11-02 11:08:43.062431 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:45.065186 I | ceph-mgr: 2018-11-02 11:08:45.065043 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:47.067328 I | ceph-mgr: 2018-11-02 11:08:47.067180 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:49.069516 I | ceph-mgr: 2018-11-02 11:08:49.069378 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:51.071688 I | ceph-mgr: 2018-11-02 11:08:51.071538 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:53.073262 I | ceph-mgr: 2018-11-02 11:08:53.073119 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:55.075462 I | ceph-mgr: 2018-11-02 11:08:55.075319 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:57.077066 I | ceph-mgr: 2018-11-02 11:08:57.076934 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:08:59.078606 I | ceph-mgr: 2018-11-02 11:08:59.078479 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:01.080754 I | ceph-mgr: 2018-11-02 11:09:01.080629 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:03.082339 I | ceph-mgr: 2018-11-02 11:09:03.082203 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:03.504180 I | ceph-mgr: 2018-11-02 11:09:03.504050 7fbfe2431700 0 -- 10.233.98.6:6800/15 >> 10.233.98.4:0/1454377778 conn(0x5646add44800 :6800 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer
2018-11-02 11:09:05.084580 I | ceph-mgr: 2018-11-02 11:09:05.084436 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:07.086328 I | ceph-mgr: 2018-11-02 11:09:07.086192 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:09.087983 I | ceph-mgr: 2018-11-02 11:09:09.087848 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:11.090057 I | ceph-mgr: 2018-11-02 11:09:11.089915 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:13.091829 I | ceph-mgr: 2018-11-02 11:09:13.091684 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:15.094156 I | ceph-mgr: 2018-11-02 11:09:15.094014 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:17.095906 I | ceph-mgr: 2018-11-02 11:09:17.095762 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:19.097673 I | ceph-mgr: 2018-11-02 11:09:19.097531 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:21.099951 I | ceph-mgr: 2018-11-02 11:09:21.099810 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:23.101653 I | ceph-mgr: 2018-11-02 11:09:23.101509 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:25.103962 I | ceph-mgr: 2018-11-02 11:09:25.103823 7fbfdd427700 1 mgr send_beacon active
2018-11-02 11:09:27.105664 I | ceph-mgr: 2018-11-02 11:09:27.105518 7fbfdd427700 1 mgr send_beacon active
kubectl -n rook-ceph exec -ti rook-ceph-tools-5d9ccf6fd7-jdk7j -- ceph -s
cluster:
id: 0319846d-08d1-437c-8799-81e81522fd9e
health: HEALTH_WARN
Reduced data availability: 32 pgs stale
1 slow requests are blocked > 32 sec. Implicated osds 1
1/3 mons down, quorum rook-ceph-mon2,rook-ceph-mon1
services:
mon: 3 daemons, quorum rook-ceph-mon2,rook-ceph-mon1, out of quorum: rook-ceph-mon0
mgr: a(active)
osd: 3 osds: 2 up, 2 in
data:
pools: 2 pools, 200 pgs
objects: 4 objects, 35 bytes
usage: 2050 MB used, 181 GB / 183 GB avail
pgs: 168 active+clean
32 stale+active+clean
kubectl -n default describe pvc pvc0001
Name: pvc0001
Namespace: default
StorageClass: default
Status: Pending
Volume:
Labels: <none>
Annotations: control-plane.alpha.kubernetes.io/leader:
{"holderIdentity":"24f5bfc9-dd45-11e8-861b-de21cb4e814d","leaseDurationSeconds":15,"acquireTime":"2018-11-02T11:06:44Z","renewTime":"2018-...
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"pvc0001","namespace":"default"},"spec":{"accessMode...
volume.beta.kubernetes.io/storage-provisioner: ceph.rook.io/block
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 3m1s ceph.rook.io/block rook-ceph-operator-8684f89d6f-pcnfg 24f5bfc9-dd45-11e8-861b-de21cb4e814d External provisioner is provisioning volume for claim "default/pvc0001"
Normal ExternalProvisioning 2m2s (x25 over 3m1s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "ceph.rook.io/block" or manually created by system administrator
Mounted By: <none>
kubectl get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE
rook-ceph-mgr-a-76ff44c7fc-frmm2 1/1 Running 0 1d 10.233.98.6 node2
rook-ceph-mon0-9jqp2 1/1 Unknown 0 1d 10.233.109.6 node0
rook-ceph-mon0-jb7z8 0/1 Pending 0 59m <none> <none>
rook-ceph-mon1-z8n7c 1/1 Running 0 1d 10.233.115.6 node1
rook-ceph-mon2-hmxvm 1/1 Running 0 1d 10.233.98.5 node2
rook-ceph-osd-id-0-8645ffc685-gr65t 0/1 Pending 0 59m <none> <none>
rook-ceph-osd-id-0-8645ffc685-zvl7j 1/1 Unknown 0 1d 10.233.109.8 node0
rook-ceph-osd-id-1-55b5cc5d57-qpg92 1/1 Running 0 1d 10.233.115.8 node1
rook-ceph-osd-id-2-6cb64bdb8b-kf52x 1/1 Running 0 1d 10.233.98.8 node2
rook-ceph-osd-prepare-node1-9c4ds 0/1 Completed 0 1d 10.233.115.7 node1
rook-ceph-osd-prepare-node2-s6xfx 0/1 Completed 0 1d 10.233.98.7 node2
rook-ceph-tools-5d9ccf6fd7-jdk7j 1/1 Running 0 1d 172.18.1.21 node1
kubectl -n rook-ceph-system logs -f rook-ceph-operator-8684f89d6f-pcnfg
2018-11-02 12:18:57.415206 I | rookcmd: starting Rook v0.8.3 with arguments '/usr/local/bin/rook ceph operator'
2018-11-02 12:18:57.415420 I | rookcmd: flag values: --help=false, --log-level=INFO, --mon-healthcheck-interval=45s, --mon-out-timeout=5m0s
2018-11-02 12:18:57.498322 I | cephcmd: starting operator
2018-11-02 12:18:58.962348 I | op-agent: getting flexvolume dir path from FLEXVOLUME_DIR_PATH env var
2018-11-02 12:18:58.962388 I | op-agent: discovered flexvolume dir path from source env var. value: /var/lib/kubelet/volume-plugins
2018-11-02 12:18:59.033527 I | op-agent: rook-ceph-agent daemonset already exists, updating ...
2018-11-02 12:18:59.219377 I | op-discover: rook-discover daemonset already exists, updating ...
2018-11-02 12:18:59.238966 I | operator: rook-provisioner ceph.rook.io/block started using ceph.rook.io flex vendor dir
2018-11-02 12:18:59.297845 I | operator: rook-provisioner rook.io/block started using rook.io flex vendor dir
2018-11-02 12:18:59.297898 I | op-cluster: start watching clusters in all namespaces
2018-11-02 12:18:59.696614 I | op-cluster: skipping watching for legacy rook cluster events (legacy cluster CRD probably doesn't exist): the server could not find the requested resource (get clusters.rook.io)
2018-11-02 12:19:00.900719 I | op-cluster: starting cluster in namespace rook-ceph
2018-11-02 12:19:18.506891 I | op-provisioner: creating volume with configuration {pool:defaultpool clusterNamespace:rook-ceph fstype:}
2018-11-02 12:19:18.506945 I | exec: Running command: rbd create defaultpool/pvc-6263d19a-de8f-11e8-a7e6-de1a241f8003 --size 2048 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring
2018-11-02 12:21:16.104499 W | op-mon: mon rook-ceph-mon0 not found in quorum, still in mon out timeout
2018-11-02 12:21:16.509337 I | op-mgr: start running mgr
2018-11-02 12:21:16.525426 I | op-mgr: the mgr keyring was already generated
2018-11-02 12:21:16.557307 I | op-mgr: rook-ceph-mgr-a deployment already exists
2018-11-02 12:21:16.557486 I | exec: Running command: ceph mgr module enable prometheus --force --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/625799751
2018-11-02 12:21:26.020101 I | op-mgr: mgr metrics service already exists
2018-11-02 12:21:26.020293 I | exec: Running command: ceph mgr module enable dashboard --force --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/300172538
2018-11-02 12:21:35.959841 I | op-mgr: dashboard service already exists
2018-11-02 12:21:35.959894 I | op-osd: start running osds in namespace rook-ceph
2018-11-02 12:21:35.961323 I | exec: Running command: ceph osd set noscrub --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/111773713
2018-11-02 12:21:44.895878 I | exec: noscrub is set
2018-11-02 12:21:44.896250 I | exec: Running command: ceph osd set nodeep-scrub --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/594708028
2018-11-02 12:21:53.805648 I | exec: nodeep-scrub is set
2018-11-02 12:21:53.903163 I | op-osd: 3 of the 3 storage nodes are valid
2018-11-02 12:21:53.903204 I | op-osd: checking if orchestration is still in progress
2018-11-02 12:21:54.097692 I | op-osd: start provisioning the osds on nodes, if needed
2018-11-02 12:21:54.250374 I | op-osd: avail devices for node node0: [{Name:nbd2 FullPath: Config:map[]}]
2018-11-02 12:21:54.268425 I | op-osd: Removing previous provision job for node node0 to start a new one
2018-11-02 12:21:54.305793 I | op-osd: batch job rook-ceph-osd-prepare-node0 still exists
2018-11-02 12:21:56.316093 I | op-osd: batch job rook-ceph-osd-prepare-node0 deleted
2018-11-02 12:21:56.396480 I | op-osd: osd provision job started for node node0
2018-11-02 12:21:56.586505 I | op-osd: avail devices for node node1: [{Name:nbd2 FullPath: Config:map[]}]
2018-11-02 12:21:56.596968 I | op-osd: Removing previous provision job for node node1 to start a new one
2018-11-02 12:21:56.633995 I | op-osd: batch job rook-ceph-osd-prepare-node1 still exists
2018-11-02 12:21:58.656244 I | op-osd: batch job rook-ceph-osd-prepare-node1 deleted
2018-11-02 12:21:58.696232 I | op-osd: osd provision job started for node node1
2018-11-02 12:21:58.837632 I | op-osd: avail devices for node node2: [{Name:nbd2 FullPath: Config:map[]}]
2018-11-02 12:21:58.846184 I | op-osd: Removing previous provision job for node node2 to start a new one
2018-11-02 12:21:58.909264 I | op-osd: batch job rook-ceph-osd-prepare-node2 still exists
2018-11-02 12:22:00.996071 I | op-osd: batch job rook-ceph-osd-prepare-node2 deleted
2018-11-02 12:22:01.022136 I | op-osd: osd provision job started for node node2
2018-11-02 12:22:01.022173 I | op-osd: start osds after provisioning is completed, if needed
2018-11-02 12:22:01.044307 I | op-osd: osd orchestration status for node node0 is starting
2018-11-02 12:22:01.044361 I | op-osd: osd orchestration status for node node1 is starting
2018-11-02 12:22:01.044393 I | op-osd: osd orchestration status for node node2 is starting
2018-11-02 12:22:01.044410 I | op-osd: 0/3 node(s) completed osd provisioning, resource version 1698798
2018-11-02 12:22:01.200793 I | op-osd: osd orchestration status for node node1 is computingDiff
2018-11-02 12:22:01.460396 I | op-osd: osd orchestration status for node node1 is orchestrating
2018-11-02 12:22:01.511517 I | op-osd: osd orchestration status for node node1 is completed
2018-11-02 12:22:01.511569 I | op-osd: starting 1 osd daemons on node node1
2018-11-02 12:22:01.596362 I | op-osd: deployment for osd 1 already exists. updating if needed
2018-11-02 12:22:01.614094 I | op-k8sutil: updating deployment rook-ceph-osd-id-1
2018-11-02 12:22:03.746189 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-id-1
2018-11-02 12:22:03.746230 I | op-osd: started deployment for osd 1 (dir=false, type=bluestore)
2018-11-02 12:22:03.796043 I | op-osd: osd orchestration status for node node2 is computingDiff
2018-11-02 12:22:03.796530 I | op-osd: osd orchestration status for node node2 is orchestrating
2018-11-02 12:22:03.796715 I | op-osd: osd orchestration status for node node2 is completed
2018-11-02 12:22:03.796736 I | op-osd: starting 1 osd daemons on node node2
2018-11-02 12:22:03.819353 I | op-osd: deployment for osd 2 already exists. updating if needed
2018-11-02 12:22:03.830298 I | op-k8sutil: updating deployment rook-ceph-osd-id-2
2018-11-02 12:22:05.880560 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-id-2
2018-11-02 12:22:05.880596 I | op-osd: started deployment for osd 2 (dir=false, type=bluestore)
2018-11-02 12:23:05.911672 I | op-osd: waiting on orchestration status update from 1 remaining nodes
2018-11-02 12:24:05.911888 I | op-osd: waiting on orchestration status update from 1 remaining nodes
2018-11-02 12:25:05.912094 I | op-osd: waiting on orchestration status update from 1 remaining nodes
2018-11-02 12:26:05.912303 I | op-osd: waiting on orchestration status update from 1 remaining nodes
2018-11-02 12:27:05.912494 I | op-osd: waiting on orchestration status update from 1 remaining nodes
2018-11-02 12:28:05.912690 I | op-osd: waiting on orchestration status update from 1 remaining nodes
2018-11-02 12:29:05.912866 I | op-osd: waiting on orchestration status update from 1 remaining nodes
2018-11-02 12:30:05.913130 I | op-osd: waiting on orchestration status update from 1 remaining nodes
2018-11-02 12:31:05.913326 I | op-osd: waiting on orchestration status update from 1 remaining nodes
2018-11-02 12:32:05.913559 E | op-osd: timed out waiting for 1 nodes: &{values:map[node0:{}]}
2018-11-02 12:32:05.913761 I | op-osd: checking if any nodes were removed
2018-11-02 12:32:05.951075 I | op-osd: processing 0 removed nodes
2018-11-02 12:32:05.951119 I | op-osd: done processing removed nodes
2018-11-02 12:32:05.951291 I | exec: Running command: ceph osd unset noscrub --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/245396075
2018-11-02 12:32:14.796107 I | exec: noscrub is unset
2018-11-02 12:32:14.796452 I | exec: Running command: ceph osd unset nodeep-scrub --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/345924814
2018-11-02 12:32:24.595838 I | exec: nodeep-scrub is unset
2018-11-02 12:32:24.596068 E | op-cluster: failed to create cluster in namespace rook-ceph. failed to start the osds. 1 failures encountered while running osds in namespace rook-ceph: timed out waiting for 1 nodes: &{values:map[node0:{}]}
2018-11-02 12:32:24.665342 I | op-mon: start running mons
2018-11-02 12:32:24.692906 I | cephmon: parsing mon endpoints: rook-ceph-mon1=10.233.32.100:6790,rook-ceph-mon2=10.233.17.182:6790,rook-ceph-mon0=10.233.38.19:6790
2018-11-02 12:32:24.693026 I | op-mon: loaded: maxMonID=2, mons=map[rook-ceph-mon1:0xc42093c1a0 rook-ceph-mon2:0xc42093c1e0 rook-ceph-mon0:0xc42093c3c0], mapping=&{Node:map[rook-ceph-mon0:0xc420039200 rook-ceph-mon1:0xc4200393e0 rook-ceph-mon2:0xc420039470] Port:map[]}
2018-11-02 12:32:24.740073 I | op-mon: saved mon endpoints to config map map[data:rook-ceph-mon1=10.233.32.100:6790,rook-ceph-mon2=10.233.17.182:6790,rook-ceph-mon0=10.233.38.19:6790 maxMonId:2 mapping:{"node":{"rook-ceph-mon0":{"Name":"node0","Hostname":"node0","Address":"172.18.1.20"},"rook-ceph-mon1":{"Name":"node1","Hostname":"node1","Address":"172.18.1.21"},"rook-ceph-mon2":{"Name":"node2","Hostname":"node2","Address":"172.18.1.22"}},"port":{}}]
2018-11-02 12:32:24.741076 I | cephmon: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2018-11-02 12:32:24.741427 I | cephmon: copying config to /etc/ceph/ceph.conf
2018-11-02 12:32:24.741810 I | cephmon: generated admin config in /var/lib/rook/rook-ceph
2018-11-02 12:32:33.502328 W | op-mon: mon rook-ceph-mon0 not found in quorum, still in mon out timeout
2018-11-02 12:32:33.640744 I | op-mgr: start running mgr
2018-11-02 12:32:33.649975 I | op-mgr: the mgr keyring was already generated
2018-11-02 12:32:33.673542 I | op-mgr: rook-ceph-mgr-a deployment already exists
2018-11-02 12:32:33.673763 I | exec: Running command: ceph mgr module enable prometheus --force --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/596636464
2018-11-02 12:32:42.696120 I | op-mgr: mgr metrics service already exists
2018-11-02 12:32:42.696304 I | exec: Running command: ceph mgr module enable dashboard --force --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/460724943
2018-11-02 12:32:52.574725 I | op-mgr: dashboard service already exists
2018-11-02 12:32:52.574769 I | op-osd: start running osds in namespace rook-ceph
2018-11-02 12:32:52.595934 I | exec: Running command: ceph osd set noscrub --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/688147938
2018-11-02 12:33:01.301327 I | exec: noscrub is set
2018-11-02 12:33:01.301611 I | exec: Running command: ceph osd set nodeep-scrub --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/747331545
2018-11-02 12:33:10.603333 I | exec: nodeep-scrub is set
2018-11-02 12:33:10.698789 I | op-osd: 3 of the 3 storage nodes are valid
2018-11-02 12:33:10.698828 I | op-osd: checking if orchestration is still in progress
2018-11-02 12:33:10.711654 I | op-osd: osd orchestration status for node node0 is starting
2018-11-02 12:33:10.711692 I | op-osd: 0/1 node(s) completed osd provisioning, resource version 1700637
2018-11-02 12:34:10.716844 I | op-osd: waiting on orchestration status update from 1 remaining nodes
2018-11-02 12:35:10.717105 E | op-osd: timed out waiting for 1 nodes: &{values:map[node0:{}]}
2018-11-02 12:35:10.717297 I | op-osd: start provisioning the osds on nodes, if needed
2018-11-02 12:35:10.882832 I | op-osd: avail devices for node node0: [{Name:nbd2 FullPath: Config:map[]}]
2018-11-02 12:35:10.892299 I | op-osd: Found previous provision job for node node0. Status={Conditions:[] StartTime:2018-11-02 12:21:56 +0000 UTC CompletionTime:<nil> Active:1 Succeeded:0 Failed:0}
2018-11-02 12:35:11.509312 I | op-osd: avail devices for node node1: [{Name:nbd2 FullPath: Config:map[]}]
2018-11-02 12:35:11.521186 I | op-osd: Removing previous provision job for node node1 to start a new one
2018-11-02 12:35:11.559263 I | op-osd: batch job rook-ceph-osd-prepare-node1 still exists
2018-11-02 12:35:13.567977 I | op-osd: batch job rook-ceph-osd-prepare-node1 deleted
2018-11-02 12:35:13.596416 I | op-osd: osd provision job started for node node1
2018-11-02 12:35:13.736269 I | op-osd: avail devices for node node2: [{Name:nbd2 FullPath: Config:map[]}]
2018-11-02 12:35:13.746099 I | op-osd: Removing previous provision job for node node2 to start a new one
2018-11-02 12:35:13.776247 I | op-osd: batch job rook-ceph-osd-prepare-node2 still exists
2018-11-02 12:35:15.783867 I | op-osd: batch job rook-ceph-osd-prepare-node2 deleted
2018-11-02 12:35:15.800116 I | op-osd: osd provision job started for node node2
2018-11-02 12:35:15.800153 I | op-osd: start osds after provisioning is completed, if needed
2018-11-02 12:35:15.814330 I | op-osd: osd orchestration status for node node0 is starting
2018-11-02 12:35:15.814380 I | op-osd: osd orchestration status for node node1 is starting
2018-11-02 12:35:15.814407 I | op-osd: osd orchestration status for node node2 is starting
2018-11-02 12:35:15.814425 I | op-osd: 0/3 node(s) completed osd provisioning, resource version 1700997
2018-11-02 12:35:16.101410 I | op-osd: osd orchestration status for node node1 is computingDiff
2018-11-02 12:35:16.373259 I | op-osd: osd orchestration status for node node1 is orchestrating
2018-11-02 12:35:16.418521 I | op-osd: osd orchestration status for node node1 is completed
2018-11-02 12:35:16.418560 I | op-osd: starting 1 osd daemons on node node1
2018-11-02 12:35:16.443045 I | op-osd: deployment for osd 1 already exists. updating if needed
2018-11-02 12:35:16.452053 I | op-k8sutil: updating deployment rook-ceph-osd-id-1
2018-11-02 12:35:18.596535 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-id-1
2018-11-02 12:35:18.596571 I | op-osd: started deployment for osd 1 (dir=false, type=bluestore)
2018-11-02 12:35:18.624152 I | op-osd: osd orchestration status for node node2 is computingDiff
2018-11-02 12:35:18.624585 I | op-osd: osd orchestration status for node node2 is orchestrating
2018-11-02 12:35:18.624924 I | op-osd: osd orchestration status for node node2 is completed
2018-11-02 12:35:18.624948 I | op-osd: starting 1 osd daemons on node node2
2018-11-02 12:35:18.645996 I | op-osd: deployment for osd 2 already exists. updating if needed
2018-11-02 12:35:18.658175 I | op-k8sutil: updating deployment rook-ceph-osd-id-2
2018-11-02 12:35:18.807313 I | op-k8sutil: finished waiting for updated deployment rook-ceph-osd-id-2
2018-11-02 12:35:18.807350 I | op-osd: started deployment for osd 2 (dir=false, type=bluestore)
2018-11-02 12:36:18.830319 I | op-osd: waiting on orchestration status update from 1 remaining nodes