Skip to content

Instantly share code, notes, and snippets.

@tomotake-koike

tomotake-koike/ocp-sm-networkpolicy-tat.md

Last active June 2, 2020 10:40
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save tomotake-koike/deab2bf40bbcf90350217c4736e3d8b9 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save tomotake-koike/deab2bf40bbcf90350217c4736e3d8b9 to your computer and use it in GitHub Desktop.
Download ZIP
OCP ServiceMesh NetworkPolicy操作による挙動の違い
Raw
ocp-sm-networkpolicy-tat.md

NetworkPolicy操作による挙動の違い

NetworkPolicyで疎通を止める検証をしていたところ、遮蔽した場合にエラーレスポンスのTATに差分があった。
本件、OCP ServiceMesh Training中に遭遇。

正常時についてはTATは短い

$ time curl -v http://$ERDEMO_USER-incident-service.apps.$SUBDOMAIN_BASE/incidents > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 34.192.130.81:80...
* TCP_NODELAY set
* Connected to user11-incident-service.apps.cluster-fcca.fcca.example.opentlc.com (34.192.130.81) port 80 (#0)
> GET /incidents HTTP/1.1
> Host: user11-incident-service.apps.cluster-fcca.fcca.example.opentlc.com
> User-Agent: curl/7.65.3
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json;charset=UTF-8
< date: Tue, 02 Jun 2020 09:04:52 GMT
< x-envoy-upstream-service-time: 715
< server: istio-envoy
< x-envoy-decorator-operation: user11-incident-service.user11-er-demo.svc.cluster.local:8080/*
< transfer-encoding: chunked
< Set-Cookie: 6199eae1064d5224ac681ff6e207997b=90cd7f1ea4f736ce3d6fd81ebe980831; path=/; HttpOnly
< Cache-control: private
<
{ [5230 bytes data]
100 11619    0 11619    0     0   9332      0 --:--:--  0:00:01 --:--:--  9332
* Connection #0 to host user11-incident-service.apps.cluster-fcca.fcca.example.opentlc.com left intact
curl -v http://$ERDEMO_USER-incident-service.apps.$SUBDOMAIN_BASE/incidents >  0.01s user 0.01s system 1% cpu 1.260 total

TATが遅いケース

以下の場合には503 Service Unavailableが返るものの20秒近く掛かっている

  • DeploymentConfigをeditし、spec.template.labelsのmaistra.io/expose-route: "true"を外した場合（Podの自動落とし上げ有り）

  • 生成済みのPodからlabelmaistra.io/expose-routeを削除
    kubectl label -n $ERDEMO_NS $(kubectl get po -n $ERDEMO_NS -l app=${ERDEMO_USER}-incident-service -o name) maistra.io/expose-route-

  • NetworkPolicyの削除（allow-from-all-namespacesやallow-from-ingress-namespace）

      $ oc edit dc -n $ERDEMO_NS user11-incident-service
  deploymentconfig.apps.openshift.io/user11-incident-service edited
  $ kubectl get po -n $ERDEMO_NS -l app=${ERDEMO_USER}-incident-service
  NAME                              READY   STATUS    RESTARTS   AGE
  user11-incident-service-6-2smp6   2/2     Running   0          14m
  $ time curl -v http://$ERDEMO_USER-incident-service.apps.$SUBDOMAIN_BASE/incidents > /dev/null
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                  Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 34.192.130.81:80...
  * TCP_NODELAY set
  * Connected to user11-incident-service.apps.cluster-fcca.fcca.example.opentlc.com (34.192.130.81) port 80 (#0)
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0> GET /incidents HTTP/1.1
  > Host: user11-incident-service.apps.cluster-fcca.fcca.example.opentlc.com
  > User-Agent: curl/7.65.3
  > Accept: */*
  >
    0     0    0     0    0     0      0      0 --:--:--  0:00:20 --:--:--     0* Mark bundle as not supporting multiuse
  * HTTP 1.0, assume close after body
  < HTTP/1.0 503 Service Unavailable
  < Pragma: no-cache
  < Cache-Control: private, max-age=0, no-cache, no-store
  < Connection: close
  < Content-Type: text/html
  <
  { [2669 bytes data]
  100  3131    0  3131    0     0    152      0 --:--:--  0:00:20 --:--:--   759
  * Closing connection 0
  curl -v http://$ERDEMO_USER-incident-service.apps.$SUBDOMAIN_BASE/incidents >  0.01s user 0.01s system 0% cpu 20.552 total

この状態に陥ってもpodの落とし上げを手動で行うとTATも早く返るようになる

    $ kubectl delete -n $ERDEMO_NS $(kubectl get po -n $ERDEMO_NS  -l app=${ERDEMO_USER}-incident-service -o name)
    pod "user11-incident-service-6-2smp6" deleted
    $ kubectl get po -n $ERDEMO_NS -l app=${ERDEMO_USER}-incident-service
    NAME                              READY   STATUS    RESTARTS   AGE
    user11-incident-service-6-485ck   1/2     Running   0          42s
    $ time curl -v http://$ERDEMO_USER-incident-service.apps.$SUBDOMAIN_BASE/incidents > /dev/null
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 18.211.83.180:80...
    * TCP_NODELAY set
    * Connected to user11-incident-service.apps.cluster-fcca.fcca.example.opentlc.com (18.211.83.180) port 80 (#0)
    > GET /incidents HTTP/1.1
    > Host: user11-incident-service.apps.cluster-fcca.fcca.example.opentlc.com
    > User-Agent: curl/7.65.3
    > Accept: */*
    >
    * Mark bundle as not supporting multiuse
    * HTTP 1.0, assume close after body
    < HTTP/1.0 503 Service Unavailable
    < Pragma: no-cache
    < Cache-Control: private, max-age=0, no-cache, no-store
    < Connection: close
    < Content-Type: text/html
    < 
    { [2669 bytes data]
    100  3131    0  3131    0     0   6249      0 --:--:-- --:--:-- --:--:--  6249
    * Closing connection 0
    curl -v http://$ERDEMO_USER-incident-service.apps.$SUBDOMAIN_BASE/incidents >  0.01s user 0.00s system 2% cpu 0.512 total
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment