Skip to content

Instantly share code, notes, and snippets.

@myclau

myclau/Drain k8s node

Last active June 9, 2020 06:57
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save myclau/d63b44bb6f19b023a87c74f58c32367a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save myclau/d63b44bb6f19b023a87c74f58c32367a to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. myclau revised this gist Jun 9, 2020. 1 changed file with 0 additions and 1 deletion.
    1 change: 0 additions & 1 deletion Drain k8s node
    View file
    Original file line number Diff line number Diff line change
    @@ -2,7 +2,6 @@
    kubectl get no
    set NODENAME=xxxxxxxxxxxxxxx

    #nodename example: shkdevk8s05.devhk.shkp.com
    kubectl cordon $NODENAME


  2. myclau created this gist Jun 9, 2020.
    33 changes: 33 additions & 0 deletions Drain k8s node
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,33 @@
    #find node names
    kubectl get no
    set NODENAME=xxxxxxxxxxxxxxx

    #nodename example: shkdevk8s05.devhk.shkp.com
    kubectl cordon $NODENAME


    #check pod on node
    kubectl get pods --all-namespaces --field-selector spec.nodeName=$NODENAME -o wide

    #find mysql pod
    kubectl get pods --all-namespaces --field-selector spec.nodeName=$NODENAME -o wide | grep mysql

    #manual remove pod
    kubectl -n <namespacename> delete pod <podname>

    #remove stuck pods
    kubectl -n <namespacename> delete pod <podname> --grace-period 0 --force

    #drain node
    kubectl drain $NODENAME --ignore-daemonsets

    if it appear something like
    ```
    There are pending nodes to be drained:
    $NODENAME
    error: pods with local storage (use --delete-local-data to override): podname1-75ddd47ff5-hptq2, podname2-7d8cf5b448-j8bx4
    ```
    kubectl get pods --all-namespaces --field-selector spec.nodeName=$NODENAME -o wide | grep -E "podname1-75ddd47ff5-hptq2|podname2-7d8cf5b448-j8bx4" | awk '{print $2 " --namespace=" $1}' | xargs kubectl delete pod

    #check pod on node
    kubectl get pods --all-namespaces --field-selector spec.nodeName=$NODENAME -o wide