Skip to content

Instantly share code, notes, and snippets.

@nikhilsuvarna

nikhilsuvarna/app_instance_identity_intro_to_envoy.md

Last active August 22, 2023 14:43
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save nikhilsuvarna/bd0aa0ef01880270c13d145c61a4af22 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save nikhilsuvarna/bd0aa0ef01880270c13d145c61a4af22 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. nikhilsuvarna revised this gist Oct 29, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion app_instance_identity_intro_to_envoy.md
    View file
    Original file line number Diff line number Diff line change
    @@ -382,6 +382,6 @@ curl -H "Host: jdoe-spring-music.cfapps-14.haas-59.pez.pivotal.io" https://10.19
    - Sample apps : https://github.com/emalm/tls-example-apps
    - Sample apps : https://github.com/nebhale/mtls-sample
    - How login to an app container from diego cell (backdoor method): https://gist.github.com/nikhilsuvarna/aa2593af5e45721049e3f1c774e238c9
    - LTS PAS issue : https://github.com/pivotal/lts-pas-issues/issues/127



  2. nikhilsuvarna revised this gist Oct 29, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions app_instance_identity_intro_to_envoy.md
    View file
    Original file line number Diff line number Diff line change
    @@ -382,5 +382,6 @@ curl -H "Host: jdoe-spring-music.cfapps-14.haas-59.pez.pivotal.io" https://10.19
    - Sample apps : https://github.com/emalm/tls-example-apps
    - Sample apps : https://github.com/nebhale/mtls-sample
    - How login to an app container from diego cell (backdoor method): https://gist.github.com/nikhilsuvarna/aa2593af5e45721049e3f1c774e238c9
    - LTS PAS issue : https://github.com/pivotal/lts-pas-issues/issues/127


  3. nikhilsuvarna revised this gist Oct 28, 2018. 1 changed file with 18 additions and 14 deletions.
    32 changes: 18 additions & 14 deletions app_instance_identity_intro_to_envoy.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,5 @@


    # Application Instance Identity and Intro to Envoy in PCF

    *(the content below is heavily borrowed from Eric Malm's blog post on [application identity](https://content.pivotal.io/blog/new-in-pcf-2-1-app-container-identity-assurance-via-automatic-cert-rotation) and Aaron Hurley's CFSummit talk on upcoming changes to [routing tier in CF](https://www.youtube.com/watch?v=5f7Bm1-E3nw))*
    @@ -8,31 +10,33 @@
    * [PCF 2.0 - Secure Service-Instance Credential Delivery](#pcf-20---secure-service-instance-credential-delivery)
    * [PCF 2.1 - Improved Routing Security and Resilience with Envoy](#pcf-21---improved-routing-security-and-resilience-with-envoy)
    * [Before Envoy](#before-envoy)
    * [After Envoy :](#after-envoy-)
    * [After Envoy](#after-envoy-)
    * [Peek inside how the "data path" is wired](#peek-inside-how-the-data-path-is-wired)
    * [Envoy](#envoy)
    * [Envoy Components](#envoy-components)
    * [envoy configuartion yaml mapping :](#envoy-configuartion-yaml-mapping-)
    * [envoy configuartion yaml mapping](#envoy-configuartion-yaml-mapping-)
    * [PCF 2.3 - "Strict" Route Integrity](#pcf-23---strict-route-integrity)
    * [Resources :](#resources-)
    * [Resources ](#resources-)

    ## Timeline


    ## Timeline :

    #### PCF 1.12 - Enabled Application Instance Identity
    ### PCF 1.12 - Enabled Application Instance Identity
    Cloud Foundry issues a unique certificate for each running app instance. This mechanism encodes the identity of the application instance on the platform in several different ways. Further, the certificate is valid for only 24 hours. The platform regenerates it, and replaces it, in the app instance filesystem automatically, shortly before it expires. So if any other service trusts PCF’s certificate authority, it is then set up to authenticate the application instances running on it, and then to authorize them based on the application metadata. This pervasive availability of this strong security fundamental allows both the platform to become more secure by default and to make it easy for your applications to do the same.

    * PCF docs - https://docs.pivotal.io/pivotalcf/1-12/installing/highlights.html#app-identity
    * PCF docs - https://docs.pivotal.io/pivotalcf/1-12/security/networking/tls-info.html#container-creds
    * original OSS CF proposal : https://docs.google.com/document/d/1OWrqaNEQkl8VXd8r3W6GgDEXxd3sXw5C-20dAu76HOk/


    #### PCF 2.0 - Secure Service-Instance Credential Delivery
    ### PCF 2.0 - Secure Service-Instance Credential Delivery
    service brokers can deliver service-instance credentials to applications through the CredHub component, instead of passing them back to Cloud Controller in the service-binding response. This is an advantage, as it helps your applications comply with regulations or internal audits

    * PCF docs - https://docs.pivotal.io/pivotalcf/2-1/opsguide/secure-si-creds.html
    * PCF docs - https://docs.pivotal.io/spring-cloud-services/1-5/service-broker-and-instances.html#using-credhub-for-service-instance-credentials

    #### PCF 2.1 - Improved Routing Security and Resilience with Envoy
    ### PCF 2.1 - Improved Routing Security and Resilience with Envoy
    ensures that the routers always connect to the app instance they intend to, and that they encrypt the traffic with TLS all the way to the app container itself.

    References :
    @@ -105,7 +109,7 @@ With the introduction of Envoy, there is a change in how data path (traffic to t

    This routing process requires the router's registrations to be up to date, though. If the system fails to update them, the router can misroute a request to an app instance that no longer exists, or, even worse, to a completely different app instance. To defend against this possibility, the routers expect the cells to broadcast the route registrations for their apps frequently. The routers then intentionally discard registrations that haven't been updated in the last 120 seconds. Cloud Foundry rightly prioritizes security over availability.

    #### After Envoy :
    #### After Envoy


    ![](https://i.imgur.com/h9i7dIc.png)
    @@ -222,7 +226,7 @@ Reference : https://www.cncf.io/wp-content/uploads/2018/05/projectFAQ_envoy.pdf

    Cloud Foundry has adopted Envoy to provide proxy to implement better security as described above

    ### Envoy Components
    #### Envoy Components

    ![](https://i.imgur.com/vgkfRpl.png)

    @@ -237,7 +241,7 @@ Clusters - routes then point to a cluster, clusters tell envoy how to send traff
    Endpoints - hosts that are able to recieve the traffic. Endpoint discovery service


    ### envoy configuartion yaml mapping :
    #### envoy configuartion yaml mapping

    ![](https://i.imgur.com/MnSMsiI.jpg)

    @@ -320,9 +324,7 @@ How is the envoy configuration and binaries injected?

    This is done via bindmount when the container is created



    #### PCF 2.3 - "Strict" Route Integrity
    ### PCF 2.3 - "Strict" Route Integrity

    Strict route integrity is implemented in 2.3. The UI for ERT now has 3 radio buttons instead of the checkbox from 2.2/2.1
    ![](https://i.imgur.com/LJoPfB6.png)
    @@ -371,7 +373,9 @@ curl -H "Host: jdoe-spring-music.cfapps-14.haas-59.pez.pivotal.io" https://10.19



    #### Resources :


    #### Resources


    - Envoy configuration : https://github.com/cloudfoundry/diego-release/blob/master/docs/envoy-proxy-configuration.md
  4. nikhilsuvarna revised this gist Oct 28, 2018. 1 changed file with 15 additions and 0 deletions.
    15 changes: 15 additions & 0 deletions app_instance_identity_intro_to_envoy.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,6 +1,21 @@
    # Application Instance Identity and Intro to Envoy in PCF

    *(the content below is heavily borrowed from Eric Malm's blog post on [application identity](https://content.pivotal.io/blog/new-in-pcf-2-1-app-container-identity-assurance-via-automatic-cert-rotation) and Aaron Hurley's CFSummit talk on upcoming changes to [routing tier in CF](https://www.youtube.com/watch?v=5f7Bm1-E3nw))*

    * [Application Instance Identity and Intro to Envoy in PCF](#application-instance-identity-and-intro-to-envoy-in-pcf)
    * [Timeline :](#timeline-)
    * [PCF 1.12 - Enabled Application Instance Identity](#pcf-112---enabled-application-instance-identity)
    * [PCF 2.0 - Secure Service-Instance Credential Delivery](#pcf-20---secure-service-instance-credential-delivery)
    * [PCF 2.1 - Improved Routing Security and Resilience with Envoy](#pcf-21---improved-routing-security-and-resilience-with-envoy)
    * [Before Envoy](#before-envoy)
    * [After Envoy :](#after-envoy-)
    * [Peek inside how the "data path" is wired](#peek-inside-how-the-data-path-is-wired)
    * [Envoy](#envoy)
    * [Envoy Components](#envoy-components)
    * [envoy configuartion yaml mapping :](#envoy-configuartion-yaml-mapping-)
    * [PCF 2.3 - "Strict" Route Integrity](#pcf-23---strict-route-integrity)
    * [Resources :](#resources-)

    ## Timeline :

    #### PCF 1.12 - Enabled Application Instance Identity
  5. nikhilsuvarna revised this gist Oct 28, 2018. 1 changed file with 6 additions and 3 deletions.
    9 changes: 6 additions & 3 deletions app_instance_identity_intro_to_envoy.md
    View file
    Original file line number Diff line number Diff line change
    @@ -306,19 +306,20 @@ How is the envoy configuration and binaries injected?
    This is done via bindmount when the container is created


    #### PCF 2.3 - "Strict" Route Integrity Implementation

    #### PCF 2.3 - "Strict" Route Integrity

    Strict route integrity is implemented in 2.3. The UI for ERT now has 3 radio buttons instead of the checkbox from 2.2/2.1
    ![](https://i.imgur.com/LJoPfB6.png)

    All of the new properties that enable this feature are in the diego release spec [here](https://github.com/cloudfoundry/diego-release/blob/v2.14.0/jobs/rep/spec#L252:L281)
    All of the new properties that enable this feature are in the diego release spec [here](https://github.com/cloudfoundry/diego-release/blob/v2.14.0/jobs/rep/spec#L248:L281)


    Reference documentation around this feature is [here](https://docs.pivotal.io/pivotalcf/2-3/concepts/http-routing.html#tls-to-back-end) to prevent misrouting.

    Some caveats documented in the [PCF 2.3 Breaking Changes Section](https://docs.pivotal.io/pivotalcf/2-3/pcf-release-notes/breaking-changes.html#mutual-tls-id)

    Additional tips from @randomtask1155 :
    Additional tips from [Dan Lynch](https://github.com/randomtask1155) :

    * In PCF 2.3 when this feature is enabled, `cf curl /v2/apps/GUID/stas` will no longer return a port for the app. This is because cloud controller is looking for container port which is null and not the new envoy TLS port

    @@ -353,6 +354,8 @@ diego_cell/ebadf72b-8d0e-4e25-beb7-ef632a8aecd9:~# cfdot actual-lrp-groups | jq
    curl -H "Host: jdoe-spring-music.cfapps-14.haas-59.pez.pivotal.io" https://10.193.79.34:61000 -vvv --cert /tmp/cert --key /tmp/key -k
    ```



    #### Resources :


  6. nikhilsuvarna revised this gist Oct 28, 2018. 1 changed file with 47 additions and 0 deletions.
    47 changes: 47 additions & 0 deletions app_instance_identity_intro_to_envoy.md
    View file
    Original file line number Diff line number Diff line change
    @@ -306,6 +306,53 @@ How is the envoy configuration and binaries injected?
    This is done via bindmount when the container is created


    #### PCF 2.3 - "Strict" Route Integrity Implementation

    Strict route integrity is implemented in 2.3. The UI for ERT now has 3 radio buttons instead of the checkbox from 2.2/2.1
    ![](https://i.imgur.com/LJoPfB6.png)

    All of the new properties that enable this feature are in the diego release spec [here](https://github.com/cloudfoundry/diego-release/blob/v2.14.0/jobs/rep/spec#L252:L281)


    Reference documentation around this feature is [here](https://docs.pivotal.io/pivotalcf/2-3/concepts/http-routing.html#tls-to-back-end) to prevent misrouting.

    Some caveats documented in the [PCF 2.3 Breaking Changes Section](https://docs.pivotal.io/pivotalcf/2-3/pcf-release-notes/breaking-changes.html#mutual-tls-id)

    Additional tips from @randomtask1155 :

    * In PCF 2.3 when this feature is enabled, `cf curl /v2/apps/GUID/stas` will no longer return a port for the app. This is because cloud controller is looking for container port which is null and not the new envoy TLS port

    * Alternatively, use cfdot instead :

    ```
    # diego_cell/ebadf72b-8d0e-4e25-beb7-ef632a8aecd9:~# cfdot actual-lrp-groups | jq 'select(.instance.process_guid | contains("93d05447-6319-4a54-8e62-6228daef1768"))' | jq '[.instance.address, .instance.ports[0].host_tls_proxy_port] | "https://\(.[0]):\(.[1])"'
    "https://10.193.79.34:61000"
    ```

    * Curling an app directly from the cell will not work as the envoy tls proxy will only trust requests coming from the gorouter via mutual TLS.

    * Here is a procedure to curl an app container directly when "strict route integrity" is enabled.

    * Extract the router cert and key from `/var/vcap/jobs/gorouter/config/gorouter.yml` from these paramaters and write them to a file
    * backends.cert_chain
    * backends.private_key


    * get the port of the app using a combination of cf api and cfdot. Make sure to use the correct app GUID. Also you might want to select an instance port other than 0

    ```
    diego_cell/ebadf72b-8d0e-4e25-beb7-ef632a8aecd9:~# cfdot actual-lrp-groups | jq 'select(.instance.process_guid | contains("93d05447-6319-4a54-8e62-6228daef1768"))' | jq '[.instance.address, .instance.ports[0].host_tls_proxy_port] | "https://\(.[0]):\(.[1])"'
    "https://10.193.79.34:61000"
    ```

    * using the router cert and key query the app, run the following `curl` command from the gorouter.

    ```
    curl -H "Host: jdoe-spring-music.cfapps-14.haas-59.pez.pivotal.io" https://10.193.79.34:61000 -vvv --cert /tmp/cert --key /tmp/key -k
    ```

    #### Resources :


  7. nikhilsuvarna renamed this gist Sep 17, 2018. 1 changed file with 0 additions and 0 deletions.
    0 app_instance_identity_intro_to_envoy → app_instance_identity_intro_to_envoy.md
    View file
    File renamed without changes.
  8. nikhilsuvarna created this gist Sep 17, 2018.
    317 changes: 317 additions & 0 deletions app_instance_identity_intro_to_envoy
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,317 @@
    # Application Instance Identity and Intro to Envoy in PCF

    *(the content below is heavily borrowed from Eric Malm's blog post on [application identity](https://content.pivotal.io/blog/new-in-pcf-2-1-app-container-identity-assurance-via-automatic-cert-rotation) and Aaron Hurley's CFSummit talk on upcoming changes to [routing tier in CF](https://www.youtube.com/watch?v=5f7Bm1-E3nw))*
    ## Timeline :

    #### PCF 1.12 - Enabled Application Instance Identity
    Cloud Foundry issues a unique certificate for each running app instance. This mechanism encodes the identity of the application instance on the platform in several different ways. Further, the certificate is valid for only 24 hours. The platform regenerates it, and replaces it, in the app instance filesystem automatically, shortly before it expires. So if any other service trusts PCF’s certificate authority, it is then set up to authenticate the application instances running on it, and then to authorize them based on the application metadata. This pervasive availability of this strong security fundamental allows both the platform to become more secure by default and to make it easy for your applications to do the same.

    * PCF docs - https://docs.pivotal.io/pivotalcf/1-12/installing/highlights.html#app-identity
    * PCF docs - https://docs.pivotal.io/pivotalcf/1-12/security/networking/tls-info.html#container-creds
    * original OSS CF proposal : https://docs.google.com/document/d/1OWrqaNEQkl8VXd8r3W6GgDEXxd3sXw5C-20dAu76HOk/


    #### PCF 2.0 - Secure Service-Instance Credential Delivery
    service brokers can deliver service-instance credentials to applications through the CredHub component, instead of passing them back to Cloud Controller in the service-binding response. This is an advantage, as it helps your applications comply with regulations or internal audits

    * PCF docs - https://docs.pivotal.io/pivotalcf/2-1/opsguide/secure-si-creds.html
    * PCF docs - https://docs.pivotal.io/spring-cloud-services/1-5/service-broker-and-instances.html#using-credhub-for-service-instance-credentials

    #### PCF 2.1 - Improved Routing Security and Resilience with Envoy
    ensures that the routers always connect to the app instance they intend to, and that they encrypt the traffic with TLS all the way to the app container itself.

    References :
    https://docs.pivotal.io/pivotalcf/2-2/devguide/deploy-apps/instance-identity.html

    PAS UI setting :

    ![](https://i.imgur.com/Uq3vush.png)

    Corresponding property in `cf` yaml :

    ```
    - name: rep
    release: diego
    consumes: {}
    provides: {}
    properties:
    containers:
    proxy:
    enabled: true <---
    additional_memory_allocation_mb: 32
    trusted_ca_certificates:
    - "((/cf/diego-instance-identity-root-ca.certificate))"
    - |
    -----BEGIN CERTIFICATE-----
    MIIDUTCCAjmgAwIBAgIVAIGoikVSbjpQwLYyjgjpo9OB0FTGMA0GCSqGSIb3DQEB
    CwUAMB8xCzAJBgNVBAYTAlVTMRAwDgYDVQQKDAdQaXZvdGFsMB4XDTE4MDgwNjE4
    NDc0N1oXDTIyMDgwNzE4NDc0N1owHzELMAkGA1UEBhMCVVMxEDAOBgNVBAoMB1Bp
    dm90YWwwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwA
    ```

    `rep` config on diego cell

    ```
    compute/33877ada-710a-41fb-a215-f035d762ac4a:/var/vcap/jobs/rep/config# cat rep.json | jq .|grep proxy
    "enable_container_proxy": true,
    "proxy_memory_allocation_mb": 32,
    "container_proxy_path": "/var/vcap/packages/proxy",
    "container_proxy_config_path": "/var/vcap/data/rep/proxy_config",
    ```

    ```
    compute/33877ada-710a-41fb-a215-f035d762ac4a:/var/vcap/jobs/rep/config# ls -l /var/vcap/data/rep/proxy_config/b1d4075d-c7e2-4ea9-572c-db9c/
    total 16
    -rw-r--r-- 1 vcap vcap 606 Aug 24 14:25 envoy.yaml
    -rw-r--r-- 1 vcap vcap 11493 Aug 28 12:25 listeners.yaml

    ```
    From OSS docs how to enable instance identity : https://docs.cloudfoundry.org/adminguide/instance-identity.html



    With the introduction of Envoy, there is a change in how data path (traffic to the app) is wired inside of Cloud Foundry.

    #### Before Envoy

    ![](https://i.imgur.com/JDiFdCB.png)

    1. The app process listens on port 8080 inside its container.

    2. The Diego cell forwards traffic from port 61080 on the host to container port 8080.

    3. The Diego cell registers its 10.0.0.5 IP and the 61080 host port with the router as an backend for the example.com domain.

    4. The router receives an HTTP request for example.com

    5. The router connects to the 10.0.0.5:61080 address and forwards the request.

    6. The Diego cell forwards the request packets to the app in its container, which handles the request.

    This routing process requires the router's registrations to be up to date, though. If the system fails to update them, the router can misroute a request to an app instance that no longer exists, or, even worse, to a completely different app instance. To defend against this possibility, the routers expect the cells to broadcast the route registrations for their apps frequently. The routers then intentionally discard registrations that haven't been updated in the last 120 seconds. Cloud Foundry rightly prioritizes security over availability.

    #### After Envoy :


    ![](https://i.imgur.com/h9i7dIc.png)

    1. The app process listens on port 8080 inside its container.

    2. Envoy listens on port 8443 inside the container, terminates TLS with the instance credentials that contain the instance ID a7c, and forwards that traffic to port 8080.

    3. The Diego cell forwards traffic from port 61443 on the host to container port 8443.

    4. The Diego cell registers its 10.0.0.5 IP and the 61443 host port with the router as a TLS backend for the pivotal.io domain, along with the instance ID a7c.

    5. The router receives an HTTP request for example.com.

    6. The router connects via TLS to the 10.0.0.5:61443 address, verifies the a7c instance ID, and only then forwards the request.

    7. The Diego cell forwards the request payload to Envoy, which in turn forwards it to the app itself for processing.

    Now if the router connects to the wrong app instance because of a route registration that is out of date, its TLS handshake fails, and it backs out and tries a different instance. As a result, the routers also no longer need to drop out-of-date TLS registrations so aggressively. The routers can maintain app availability during extended failures of the route-registration system.

    What about when the instance certificates expire? In that case, because the Diego cell already knows it has issued new credentials, it also uses Envoy's dynamic configuration capabilities to update the credentials there as well. On subsequent connections, Envoy then uses the new set of credentials for TLS termination without skipping a beat.

    #### Peek inside how the "data path" is wired

    Using the instructions [here](https://gist.github.com/nikhilsuvarna/aa2593af5e45721049e3f1c774e238c9), connect to the app-instance/container

    1. Check for listening ports :

    ```
    oot@7cdd109d-2feb-43eb-6c00-720d:/# netstat -anp | grep LISTEN
    tcp 0 0 0.0.0.0:61001 0.0.0.0:* LISTEN 86/envoy
    tcp 0 0 0.0.0.0:61002 0.0.0.0:* LISTEN 86/envoy
    tcp 0 0 127.0.0.1:61004 0.0.0.0:* LISTEN 86/envoy
    tcp 0 0 0.0.0.0:2222 0.0.0.0:* LISTEN -
    tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN -
    ```

    2. the app listens or port 8080, the corresponding port configured on the envoy side is 61001. port 2222 is used for ssh and the corresponding port on the envoy isde is 61002

    3. port 61004 is the port for the envoy api
    4. envoy is configured using the following configuration file `/etc/cf-assets/envoy_config/envoy.yaml`:

    ```
    admin:
    access_log_path: /dev/null
    address:
    socket_address:
    address: 127.0.0.1
    port_value: 61004
    static_resources:
    clusters:
    - name: 0-service-cluster
    connect_timeout: 0.25s
    type: STATIC
    lb_policy: ROUND_ROBIN
    hosts:
    - socket_address:
    address: 10.255.247.3
    port_value: 8080 <---
    - name: 1-service-cluster
    connect_timeout: 0.25s
    type: STATIC
    lb_policy: ROUND_ROBIN
    hosts:
    - socket_address:
    address: 10.255.247.3
    port_value: 2222 <---
    dynamic_resources:
    lds_config:
    path: /etc/cf-assets/envoy_config/listeners.yaml
    ```

    5. `echo $CF_INSTANCE_PORTS` to list the corresponding envoy ports

    6. Check for `envoy` processes inside of the app container

    ```
    root@7cdd109d-2feb-43eb-6c00-720d:/# ps -ef
    UID PID PPID C STIME TTY TIME CMD
    root 1 0 0 23:25 ? 00:00:00 /tmp/garden-init
    vcap 13 0 0 23:25 ? 00:00:00 web
    vcap 16 0 0 23:25 ? 00:00:00 /tmp/lifecycle/diego-sshd --allowedKeyExchanges= --address=0.0.0.0:2222 --allowUnauthenticatedClients=fa
    root 55 0 0 23:25 ? 00:00:00 sh -c trap 'kill -9 0' TERM; /etc/cf-assets/envoy/envoy -c /etc/cf-assets/envoy_config/envoy.yaml --serv
    root 86 55 0 23:25 ? 00:00:01 /etc/cf-assets/envoy/envoy -c /etc/cf-assets/envoy_config/envoy.yaml --service-cluster proxy-cluster --s
    root 93 0 0 23:25 ? 00:00:00 /etc/cf-assets/healthcheck/healthcheck -port=8080 -timeout=1000ms -liveness-interval=30s
    root 144 0 1 23:50 pts/0 00:00:00 /bin/bash
    ```


    7. `iptables` configuration for DNAT to the container IP on the diego cell :

    ```
    iptables -t nat -nL | grep 61001
    DNAT tcp -- 0.0.0.0/0 10.193.68.33 tcp dpt:61079 to:10.255.247.28:61001 <---
    DNAT tcp -- 0.0.0.0/0 10.193.68.33 tcp dpt:61065 to:10.255.247.25:61001
    DNAT tcp -- 0.0.0.0/0 10.193.68.33 tcp dpt:61012 to:10.255.247.8:61001
    ```
    Note the IP address of the app container to find the corresponding DNAT rule for incoming traffic to the app via the diego cell


    ### Envoy
    What is Envoy?

    Created at Lyft, Envoy is a high performance open source service mesh proxy that makes the network transparent to apps.

    - written in C++
    - minimal cpu and mem footprint
    - api driven config (dynamic configuration via Discovery Services xDS APIs)
    - L4 (TCP) proxy
    - bidirectional transparent proxy
    - sidecar and ingress models (CF uses the ingress model) (see [reference](https://www.envoyproxy.io/docs/envoy/latest/intro/deployment_types/deployment_types) here for details)

    Reference : https://www.cncf.io/wp-content/uploads/2018/05/projectFAQ_envoy.pdf

    Cloud Foundry has adopted Envoy to provide proxy to implement better security as described above

    ### Envoy Components

    ![](https://i.imgur.com/vgkfRpl.png)



    Listeners - which port it is listening to, which protocol should I be using (LDS)

    Routes - Listeners are mapped to routes - routes tell envoy where traffic to be sent, like a matcher like a host header, route discovery

    Clusters - routes then point to a cluster, clusters tell envoy how to send traffic, tells whether to use TLS or LB strategy, cluster discovery service

    Endpoints - hosts that are able to recieve the traffic. Endpoint discovery service


    ### envoy configuartion yaml mapping :

    ![](https://i.imgur.com/MnSMsiI.jpg)


    `listeners.yaml` file

    ```
    root@b1d4075d-c7e2-4ea9-572c-db9c:/etc/cf-assets/envoy_config# cat listeners.yaml
    version_info: "0"
    resources:
    - '@type': type.googleapis.com/envoy.api.v2.Listener
    name: listener-8080
    address:
    socket_address:
    address: 0.0.0.0
    port_value: 61001
    filter_chains:
    - filters:
    - name: envoy.tcp_proxy
    config:
    stat_prefix: 0-stats
    cluster: 0-service-cluster
    tls_context:
    common_tls_context:
    tls_certificates:
    - certificate_chain:
    inline_string: |
    -----BEGIN CERTIFICATE-----

    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----

    -----END CERTIFICATE-----
    private_key:
    inline_string: |
    -----BEGIN RSA PRIVATE KEY-----

    -----END RSA PRIVATE KEY-----
    tls_params:
    cipher_suites: '[ECDHE-RSA-AES256-GCM-SHA384|ECDHE-RSA-AES128-GCM-SHA256]'
    - '@type': type.googleapis.com/envoy.api.v2.Listener
    name: listener-2222
    address:
    socket_address:
    address: 0.0.0.0
    port_value: 61002
    filter_chains:
    - filters:
    - name: envoy.tcp_proxy
    config:
    stat_prefix: 1-stats
    cluster: 1-service-cluster
    tls_context:
    common_tls_context:
    tls_certificates:
    - certificate_chain:
    inline_string: |
    -----BEGIN CERTIFICATE-----

    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----

    -----END CERTIFICATE-----
    private_key:
    inline_string: |
    -----BEGIN RSA PRIVATE KEY-----

    -----END RSA PRIVATE KEY-----
    tls_params:
    cipher_suites: '[ECDHE-RSA-AES256-GCM-SHA384|ECDHE-RSA-AES128-GCM-SHA256]'
    ```


    routes are what will match on your header info - it will point to a cluster. cluster will have routing config and group of end points - upstream IP/hosts that will handle the traffic

    ![](https://i.imgur.com/EhaKsem.png)


    How is the envoy configuration and binaries injected?

    This is done via bindmount when the container is created


    #### Resources :


    - Envoy configuration : https://github.com/cloudfoundry/diego-release/blob/master/docs/envoy-proxy-configuration.md
    - Sample apps : https://github.com/emalm/tls-example-apps
    - Sample apps : https://github.com/nebhale/mtls-sample
    - How login to an app container from diego cell (backdoor method): https://gist.github.com/nikhilsuvarna/aa2593af5e45721049e3f1c774e238c9