Skip to content

Instantly share code, notes, and snippets.

@MithunArunan

MithunArunan/Engineering.md

Last active September 12, 2022 20:03
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save MithunArunan/8e7a3df05862cbf6647ad3bde8ce884e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save MithunArunan/8e7a3df05862cbf6647ad3bde8ce884e to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. MithunArunan revised this gist Jun 6, 2019. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion networking.md
    View file
    Original file line number Diff line number Diff line change
    @@ -60,4 +60,8 @@ Frame: The smallest unit of communication in HTTP/2, each containing a frame hea

    [gRPC - AwesomeList](https://github.com/grpc-ecosystem/awesome-grpc)

    [gRPC - microservices - example](https://github.com/harlow/go-micro-services)
    [gRPC - microservices - example](https://github.com/harlow/go-micro-services)

    [gRPC in production](https://news.ycombinator.com/item?id=14822294)

    [gRPC design and implementation](https://platformlab.stanford.edu/Seminar%20Talks/gRPC.pdf)
  2. MithunArunan revised this gist Jun 3, 2019. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion testing.md
    View file
    Original file line number Diff line number Diff line change
    @@ -17,6 +17,9 @@

    - [Vegeta](https://github.com/tsenart/vegeta)


    ## References

    [Ask HN: Which load testing tool do you use? Any recommendations?](https://news.ycombinator.com/item?id=15733910)
    [Ask HN: Which load testing tool do you use? Any recommendations?](https://news.ycombinator.com/item?id=15733910)

    [Vegeta: HTTP load testing tool written in Go](https://news.ycombinator.com/item?id=6229874)
  3. MithunArunan revised this gist May 31, 2019. 1 changed file with 12 additions and 9 deletions.
    21 changes: 12 additions & 9 deletions testing.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,19 +1,22 @@
    ## Unit Testing
    [Go - Best practices for unit tests](https://medium.com/@sebdah/go-best-practices-testing-3448165a0e18)

    ## API Testing
    Postman
    ## API Testing tools
    - Postman

    ## Integration Testing
    ## Integration Testing tools
    - Postman

    Postman
    ## Stress Testing tools

    ## Stress Testing
    - [Lucust.io](https://locust.io/)

    [Distributed load testing using kubernetes](https://cloud.google.com/solutions/distributed-load-testing-using-kubernetes)
    - [Gatling](https://gatling.io/)

    [Lucust.io](https://locust.io/)
    - [Tsung](https://github.com/processone/tsung)

    [Gatling](https://gatling.io/)
    - [Vegeta](https://github.com/tsenart/vegeta)

    [Tsung](https://github.com/processone/tsung)
    ## References

    [Ask HN: Which load testing tool do you use? Any recommendations?](https://news.ycombinator.com/item?id=15733910)
  4. MithunArunan revised this gist May 3, 2019. 1 changed file with 0 additions and 7 deletions.
    7 changes: 0 additions & 7 deletions kong-api-gateway.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +0,0 @@
    ![Kong - kubernetes](https://raw.githubusercontent.com/Kong/kubernetes-ingress-controller/master/docs/images/deployment.png)

    ## References

    https://github.com/Kong/kubernetes-ingress-controller

    https://medium.com/@jeffzzq/using-kong-with-kubernetes-c39b74f1843
  5. MithunArunan renamed this gist May 3, 2019. 1 changed file with 0 additions and 0 deletions.
    0 Best-Practices-Engineering.md → Engineering.md
    View file
    File renamed without changes.
  6. MithunArunan revised this gist May 3, 2019. 3 changed files with 1 addition and 12 deletions.
    1 change: 1 addition & 0 deletions Best-Practices-Engineering.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1 @@
    .
    2 changes: 0 additions & 2 deletions Best-Practices-For-Product.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,2 +0,0 @@
    [ISO - 25010 standard for product quality](http://iso25000.com/index.php/en/iso-25000-standards/iso-25010)
    [Quality Attributes in Software Architecture](https://hackernoon.com/quality-attributes-in-software-architecture-3844ea482732)
    10 changes: 0 additions & 10 deletions Case-Studies.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,10 +0,0 @@
    ## Case Studies

    ### B2B2C
    - Freshworks
    - Zoho

    ### B2C
    - Zerodha
    - GoJek
    - Razorpay
  7. MithunArunan revised this gist Oct 1, 2018. 1 changed file with 39 additions and 0 deletions.
    39 changes: 39 additions & 0 deletions RDMS-vs-ES.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,39 @@
    ### Questions
    1. How to handle errors in logging the data? Type error or so on? How will we be notified?
    RBDMS will throw error in sentry
    2. What are the data points?
    3. How to retain 1 year old data and use it?
    4. How to add new data?
    5. How to make breaking changes to the data stored for a better version of it?
    6. What is the overhead (ms) in using DB over ES.
    7. No updates to the call information?
Consider this scenario, you get a call for order refund status in e-commerce domain we raise a concerned ticket in the e-commerce platform and keep the call details as unresolved. We need to mark this as resolved by the client either by contacting the customer manually or in an automated way.
    8. Multi tenancy ? Separate database for each client?

    Possible categories of data
    - Write only data.
    - Significant/Critical data.
    - Not so significant data.
    - Structured data.
    - Unstructured data.

    Possible Architectural Solutions
    1. To store call logs in RDBMS - Straight forward approach.
    2. To push details to a message queue - Consume and store in RDBMS, to avoid overhead time of storing in DB.
    3. Push details to a message queue - Consume and store in NOSQL like Mongo, Cassandra.
    4. Push and forget all the structured data directly to ES.
    5. Use application logs and store in ES using beats or fluentd.
    6. Keep the data model generic and just use the stucts

    To build Activity dashboard and query the data.

    — RDBMS

    — Logging approach

    RESEARCH

    ES - Primary data store

    Good option for cases with only writes (no updates), many reads and wherein there is no need for transactions, integrity, constraints (datatype, PK, FK, NOT NULL, DEFAULT, UNIQUE), correctness and robustness

    Elasticsearch is commonly used in addition to another database.
  8. MithunArunan revised this gist Sep 14, 2018. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion Best-Practices-For-Product.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1 +1,2 @@
    [ISO - 25010 standard](http://iso25000.com/index.php/en/iso-25000-standards/iso-25010)
    [ISO - 25010 standard for product quality](http://iso25000.com/index.php/en/iso-25000-standards/iso-25010)
    [Quality Attributes in Software Architecture](https://hackernoon.com/quality-attributes-in-software-architecture-3844ea482732)
  9. MithunArunan revised this gist Sep 14, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion Best-Practices-For-Product.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1 +1 @@
    .
    [ISO - 25010 standard](http://iso25000.com/index.php/en/iso-25000-standards/iso-25010)
  10. MithunArunan revised this gist Aug 28, 2018. 1 changed file with 10 additions and 0 deletions.
    10 changes: 10 additions & 0 deletions Case-Studies.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,10 @@
    ## Case Studies

    ### B2B2C
    - Freshworks
    - Zoho

    ### B2C
    - Zerodha
    - GoJek
    - Razorpay
  11. MithunArunan revised this gist Jul 25, 2018. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion monitoring.md
    View file
    Original file line number Diff line number Diff line change
    @@ -14,7 +14,10 @@ For each microservice, track the folowing
    ### Open Source APM products

    * Elastic.co APM
    * Prometheus & Grafana
    * OpenCensus
    * Prometheus
    * Zipkin
    * Jaegar

    ## References

  12. MithunArunan renamed this gist Jul 19, 2018. 1 changed file with 0 additions and 0 deletions.
    0 Best-Practices-For-Prodcut.md → Best-Practices-For-Product.md
    View file
    File renamed without changes.
  13. MithunArunan revised this gist Jul 19, 2018. 3 changed files with 34 additions and 0 deletions.
    0 Architecture.md → Best-Practices-For-Prodcut.md
    View file
    File renamed without changes.
    6 changes: 6 additions & 0 deletions SSL.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,6 @@
    ## OpenSSL

    ## Reference
    [Digicert](https://www.digicert.com/ssl/)

    [OpenSSL](https://security.stackexchange.com/questions/59184/which-browsers-are-using-openssl)
    28 changes: 28 additions & 0 deletions frameworks.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,28 @@
    # Building internal core libraries
    Building Core libraries with Meta Programming

    ## WHY?

    * Re-Use across projects is much cleaner
    * Separation of responsibility - part of your code may be more suitable different developers or teams
    * You can benefit from improvements in the libraries that other teams make without having to locate the specific code
    * Cleaner design and code - thinking about structuring things into libraries should result in related groups of functionality in each library and you tend to separate generic code, into the libraries, from the application specifics, in the app.

    ## When to create a wrapper for external library
    * If you use only small subset of the external library
    * If there is any possibility of changing the external library in the future

    ## When not to create a wrapper for external library
    * If there is any no possibility of changing the external library

    ## How wrapping up libraries help?
    * Your code base becomes more flexible to changes
    * You can define the API of the wrapper independently of the API of the library
    * Unit testing is way simpler
    * You create a loosely coupled system

    ## Reference

    [Why develop internal libraries?](https://softwareengineering.stackexchange.com/questions/356290/why-develop-internal-libraries-for-internal-applications)

    [Why always use a wrapper](https://softwareengineering.stackexchange.com/questions/107338/using-third-party-libraries-always-use-a-wrapper)
  14. MithunArunan revised this gist Jul 14, 2018. 1 changed file with 0 additions and 89 deletions.
    89 changes: 0 additions & 89 deletions kops.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,89 +0,0 @@
    ## Setup Kops CLI and kubectl CLI

    ```bash
    curl -Lo kops https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-darwin-amd64
    chmod +x ./kops
    sudo mv ./kops /usr/local/bin/
    ```

    ## AWS - Kops

    ### Setup AWS CLI and kops IAM user/group

    ```bash
    aws iam create-group --group-name kops

    aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess --group-name kops
    aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonRoute53FullAccess --group-name kops
    aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess --group-name kops
    aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/IAMFullAccess --group-name kops
    aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess --group-name kops

    aws iam create-user --user-name kops

    aws iam add-user-to-group --user-name kops --group-name kops

    aws iam create-access-key --user-name kops

    export AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id)
    export AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key)
    ```

    ### Cluster state storage

    ```bash
    aws s3api create-bucket \
    --bucket product-example-com-state-store \
    --region us-west-2 \
    --create-bucket-configuration LocationConstraint=us-west-2
    ```

    ### Create cluster

    ```bash
    export NAME=product.k8s.local
    export KOPS_STATE_STORE=s3://product-example-com-state-store
    ```

    ```bash
    aws ec2 describe-availability-zones --region us-west-2
    kops create cluster \
    --zones us-west-2a \
    ${NAME}
    kops edit cluster ${NAME}
    kops update cluster ${NAME} --yes
    kops get nodes
    kops validate cluster

    kops delete cluster --name ${NAME}
    kops delete cluster --name ${NAME} --yes
    ```

    ### Run k8s dashboard

    ```bash
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
    kops get secrets kube --type secret -oplaintext
    ```

    ### Cluster spec & network topology

    ## AWS - Kops - Terraform


    ## OnPremise - Kops


    ## References

    https://kubernetes.io/docs/getting-started-guides/scratch/

    https://github.com/kubernetes/kops

    https://github.com/kubernetes/kops/blob/master/docs/aws.md

    https://kubernetes.io/docs/getting-started-guides/kops/

    https://kubernetes.io/docs/getting-started-guides/aws/

    https://kubernetes.io/docs/getting-started-guides/kubespray/
  15. MithunArunan revised this gist Jul 14, 2018. 1 changed file with 0 additions and 17 deletions.
    17 changes: 0 additions & 17 deletions logging.md
    View file
    Original file line number Diff line number Diff line change
    @@ -10,23 +10,6 @@ Visualization - Kibana

    ![EFK](https://blog.ptrk.io/content/images/2018/04/elk.png)

    * Collect stdout/stderr logs using fluentd in kubernetes cluster as DaemonSet.
    * Add kubernetes metadata to the logs
    * Logrotate and Backup all the raw logs to s3 with kubernetes metadata (if needed to use other than ES as a backend store)
    * Store all the logs in elastic search backend in a parsed format
    * Backup all the elastic search index periodically
    * Connect Kibana dashboard to ES backend and query the logs



    [fluent-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch)

    [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter)

    [EFK stack - kubernetes](https://blog.ptrk.io/how-to-deploy-an-efk-stack-to-kubernetes/)



    ## Application loggers

    * Environment specific log encoding - JSON (production), console(development)
  16. MithunArunan revised this gist Jul 14, 2018. 3 changed files with 0 additions and 114 deletions.
    1 change: 0 additions & 1 deletion CNCF.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1 +0,0 @@
    ![CNCF - Trail map](https://pbs.twimg.com/media/DfbNNaOWAAAlNbC.jpg:large)
    17 changes: 0 additions & 17 deletions docker.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,17 +0,0 @@
    # Docker - Standards
    Certain set of standards for creating images <TODO>
    /home/<app-name>
    /vol/data
    /vol/models - AI models

    ## Tagging images

    Before deploying any image let’s create another tag, preferably not with latest.
    <image-name>:master

    ##### Major release
    <image-name>:<version>

    #### Minor release
    <image-name>:<version>-<commit-id-7chars>
    [12 Factor App - Docker](https://github.com/docker/labs/tree/master/12factor)
    96 changes: 0 additions & 96 deletions kubernetes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,96 +0,0 @@
    # Kubernetes

    ## Access Kubernetes Cluster

    [Refer](https://docs.bitnami.com/kubernetes/how-to/configure-rbac-in-your-kubernetes-cluster/#use-case-1-create-user-with-limited-namespace-access)
    ```bash
    openssl genrsa -out mithun.key 2048
    openssl req -new -key mithun.key -out mithun.csr -subj "/CN=mithun/O=admin"
    openssl x509 -req -in mithun.csr -CA /etc/kubernetes/pki/ca.crt -CAkey CA_LOCATION/ca.key -CAcreateserial -out employee.crt -days 500
    ```

    ```bash
    kubectl config set-cluster <cluster_name> --server=https://<master-node-ip>:<master-node-port> --insecure-skip-tls-verify=true
    kubectl config get-clusters

    kubectl config set-credentials <cluster_name> --client-certificate= --client-key= --cluster=<cluster_name>
    kubectl config set-credentials <cluster_name> --username=<username> --password=<password> --cluster=<cluster_name>

    kubectl config set-context <cluster_name> --user=<cluster_name> --cluster=<cluster_name>
    kubectl config use-context <cluster_name>
    kubectl config view
    kubectl get pods

    ```




    ## Kubernetes Cluster configurations
    Grouping all the kubernetes and docker configurations in one place
    k8s-configs
    dockerfiles - base docker images


    #### Services
    Vault (for storing secrets)
    Vault-ui
    Kube-ops-view
    All other microservices

    #### K8S configs

    ###### Deployment.yaml
    Create a label ‘app’ for grouping pods

    ###### Service.yaml
    Use ClusterIP for exposing the services internally, let’s create ingress when we would like to expose them to public.

    ClusterIP - Exposes the service on a cluster-internal IP. Choosing this value makes the service only reachable from within the cluster. This is the default ServiceType
    LoadBalancer - Exposes the service externally using a cloud provider’s load balancer
    NodePort - Exposes the service on each Node’s IP at a static port (the NodePort)

    ###### Ingress.yaml

    ###### Pvc.yaml - Persistent Volume Claim


    #### k8s commands
    kubectl apply -f k8s-spec-directory/ → kubectl apply -f juno/

    telepresence --swap-deployment voice-worker --docker-run -it -v $PWD:/home/voice-worker gcr.io/vernacular-tools/voice-services/voice-worker:1

    ###### Setting up Vault in local

    docker pull vault
    docker pull consul
    docker pull djenriquez/vault-ui

    ###### Vault binary download

    docker run --cap-add=IPC_LOCK -p 8200:8200 -e 'VAULT_DEV_ROOT_TOKEN_ID=roottoken' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' -d --name=vault vault
    docker run -d -p 8201:8201 -e PORT=8201 -e VAULT_URL_DEFAULT=http://192.168.12.155:8200 -e VAULT_AUTH_DEFAULT=GITHUB --name vault-ui djenriquez/vault-ui



    ## Next Steps

    * Telepresence

    * [Minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/)

    * Dockers for development

    * Helm


    ## References
    [Kubernetes - Design principles](http://blog.kubernetes.io/2018/03/principles-of-container-app-design.html)

    [Kubernetes configuration examples](https://github.com/kubernetes/examples)

    [GKE - letsencrypt](https://github.com/ahmetb/gke-letsencrypt)

    [Kubernetes - Vault integration](https://blog.openshift.com/vault-integration-using-kubernetes-authentication-method/)

    [Kubernetes - NFS on GCP](https://cloud.google.com/solutions/filers-on-compute-engine)
  17. MithunArunan revised this gist Jul 10, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions testing.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,5 @@
    ## Unit Testing
    [Go - Best practices for unit tests](https://medium.com/@sebdah/go-best-practices-testing-3448165a0e18)

    ## API Testing
    Postman
  18. MithunArunan revised this gist Jul 3, 2018. 1 changed file with 25 additions and 2 deletions.
    27 changes: 25 additions & 2 deletions networking.md
    View file
    Original file line number Diff line number Diff line change
    @@ -22,19 +22,42 @@ Optimized encoding mechanism between the socket interface and the higher HTTP AP

    ![Binary framing layer archi](https://developers.google.com/web/fundamentals/performance/http2/images/binary_framing_layer01.svg)

    ### Streams, messages and frames
    #### Streams, messages and frames

    Stream: A bidirectional flow of bytes within an established connection, which may carry one or more messages.

    Message: A complete sequence of frames that map to a logical request or response message.

    Frame: The smallest unit of communication in HTTP/2, each containing a frame header, which at a minimum identifies the stream to which the frame belongs.

    #### Request/Response multiplexing

    #### Stream prioritization

    #### One connection per origin

    #### Flow control

    #### Server push

    ### Benefits of gRPC

    * Canonical

    * Performance

    * Backward compatibility

    * Polyglot

    ## References

    [High Performance Browser Networking](https://hpbn.co/) by Ilya Grigorik

    [HTTP/2](https://developers.google.com/web/fundamentals/performance/http2/)

    [gRPC - Principles](https://grpc.io/blog/principles)
    [gRPC - Principles](https://grpc.io/blog/principles)

    [gRPC - AwesomeList](https://github.com/grpc-ecosystem/awesome-grpc)

    [gRPC - microservices - example](https://github.com/harlow/go-micro-services)
  19. MithunArunan revised this gist Jul 3, 2018. 1 changed file with 34 additions and 3 deletions.
    37 changes: 34 additions & 3 deletions networking.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,9 +1,40 @@
    # HTTP/2 - gRPC
    ## SPDY - HTTP/2.0 - gRPC

    SPDY was an experimental protocol, developed at Google and announced in mid 2009, whose primary goal was to try to reduce the load latency of web pages by addressing some of the well-known performance limitations of HTTP/1.1.

    HTTP/2 reduces latency by enabling full request and response multiplexing, minimize protocol overhead via efficient compression of HTTP header fields, support for request prioritization, allows multiple concurrent exchanges on the same connection and server push.

    [RFC 7540](https://tools.ietf.org/html/rfc7540) (HTTP/2) and [RFC 7541](https://tools.ietf.org/html/rfc7541) (HPACK)

    ### Drawbacks of HTTP/1.x

    HTTP/0.9 was a one-line protocol to bootstrap the World Wide Web.

    HTTP/1.0 documented the popular extensions to HTTP/0.9 in an informational standard.

    HTTP/1.1 introduced an official IETF standard.

    HTTP/1.x clients need to use multiple connections to achieve concurrency and reduce latency; HTTP/1.x does not compress request and response headers, causing unnecessary network traffic; HTTP/1.x does not allow effective resource prioritization, resulting in poor use of the underlying TCP connection; and so on.

    ### Binary framing layer

    Optimized encoding mechanism between the socket interface and the higher HTTP API exposed to our applications: the HTTP semantics, such as verbs, methods, and headers, are unaffected, but the way they are encoded while in transit is different. Instead of new line delimited plaintext.

    ![Binary framing layer archi](https://developers.google.com/web/fundamentals/performance/http2/images/binary_framing_layer01.svg)

    ### Streams, messages and frames

    Stream: A bidirectional flow of bytes within an established connection, which may carry one or more messages.

    Message: A complete sequence of frames that map to a logical request or response message.

    Frame: The smallest unit of communication in HTTP/2, each containing a frame header, which at a minimum identifies the stream to which the frame belongs.

    Reduces latency by enabling full request and response multiplexing, minimize protocol overhead via efficient compression of HTTP header fields, and add support for request prioritization and server push

    ## References

    [High Performance Browser Networking](https://hpbn.co/) by Ilya Grigorik

    [HTTP/2](https://developers.google.com/web/fundamentals/performance/http2/)
    [HTTP/2](https://developers.google.com/web/fundamentals/performance/http2/)

    [gRPC - Principles](https://grpc.io/blog/principles)
  20. MithunArunan revised this gist Jul 3, 2018. 1 changed file with 9 additions and 0 deletions.
    9 changes: 9 additions & 0 deletions networking.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,9 @@
    # HTTP/2 - gRPC

    Reduces latency by enabling full request and response multiplexing, minimize protocol overhead via efficient compression of HTTP header fields, and add support for request prioritization and server push

    ## References

    [High Performance Browser Networking](https://hpbn.co/) by Ilya Grigorik

    [HTTP/2](https://developers.google.com/web/fundamentals/performance/http2/)
  21. MithunArunan revised this gist Jun 26, 2018. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions kubernetes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -92,3 +92,5 @@ docker run -d -p 8201:8201 -e PORT=8201 -e VAULT_URL_DEFAULT=http://192.168.12.1
    [GKE - letsencrypt](https://github.com/ahmetb/gke-letsencrypt)

    [Kubernetes - Vault integration](https://blog.openshift.com/vault-integration-using-kubernetes-authentication-method/)

    [Kubernetes - NFS on GCP](https://cloud.google.com/solutions/filers-on-compute-engine)
  22. MithunArunan revised this gist Jun 22, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion pub_sub.md
    View file
    Original file line number Diff line number Diff line change
    @@ -48,7 +48,7 @@ How external clients connect to rabbitmq?

    How does node discovery happen?

    (Messages stored in disk?)[https://stackoverflow.com/questions/38444425/how-does-rabbitmq-actually-store-the-message-physically] - */var/lib/rabbitmq/mnesia/rabbit@hostname/queues* - (File locations)[https://www.rabbitmq.com/relocate.html]
    [Messages stored in disk?](https://stackoverflow.com/questions/38444425/how-does-rabbitmq-actually-store-the-message-physically) - */var/lib/rabbitmq/mnesia/rabbit@hostname/queues* - [File locations](https://www.rabbitmq.com/relocate.html)

    ```bash
    rabbitmq-server
  23. MithunArunan revised this gist Jun 22, 2018. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions pub_sub.md
    View file
    Original file line number Diff line number Diff line change
    @@ -48,6 +48,8 @@ How external clients connect to rabbitmq?

    How does node discovery happen?

    (Messages stored in disk?)[https://stackoverflow.com/questions/38444425/how-does-rabbitmq-actually-store-the-message-physically] - */var/lib/rabbitmq/mnesia/rabbit@hostname/queues* - (File locations)[https://www.rabbitmq.com/relocate.html]

    ```bash
    rabbitmq-server
    rabbitmqctl status
  24. MithunArunan revised this gist Jun 22, 2018. 1 changed file with 27 additions and 3 deletions.
    30 changes: 27 additions & 3 deletions pub_sub.md
    View file
    Original file line number Diff line number Diff line change
    @@ -23,13 +23,37 @@ Message Queues - RabbitMQ, Kafka
    Push API
    Pull API



    #### Other supported protocols
    MQTT
    STOMP

    ## Reliability guide
    #### RabbitMQ Clustering - Reliability guide

    `Clustering`

    `Federation`

    `Shovel`

    Nodes are equal peers, No Master/Slave setup. Data is sharded between the nodes and can be viewed by the client from any node.

    All data/state for the cluster are replicated, not the queues. Each queue has a master node.
    - Mirrored Queues
    - Non Mirrored Quueues

    Node discovery happens with ErlangCookie located at */var/lib/rabbitmq/.erlang.cookie* using anyone of the standard peer discovery plugins *rabbit_peer_discovery_k8s*

    Disk vs RAM Nodes - One disk node should be always present
    How external clients connect to rabbitmq?

    How does node discovery happen?

    ```bash
    rabbitmq-server
    rabbitmqctl status
    rabbitmq-plugins list
    rabbitmqadmin
    ```

    ## Reference

  25. MithunArunan revised this gist Jun 14, 2018. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion monitoring.md
    View file
    Original file line number Diff line number Diff line change
    @@ -22,4 +22,8 @@ For each microservice, track the folowing

    https://www.elastic.co/solutions/apm

    https://medium.com/@timfpark/simple-kubernetes-cluster-monitoring-with-prometheus-and-grafana-dd27edb1641
    https://medium.com/@timfpark/simple-kubernetes-cluster-monitoring-with-prometheus-and-grafana-dd27edb1641

    https://github.com/kubernetes/heapster

    https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus
  26. MithunArunan revised this gist Jun 13, 2018. 1 changed file with 3 additions and 1 deletion.
    4 changes: 3 additions & 1 deletion pub_sub.md
    View file
    Original file line number Diff line number Diff line change
    @@ -43,4 +43,6 @@ STOMP

    [RabbitMQ vs Kafka](https://content.pivotal.io/rabbitmq/understanding-when-to-use-rabbitmq-or-apache-kafka)

    [Amqp - Docs](https://godoc.org/github.com/streadway/amqp)
    [Amqp - Docs](https://godoc.org/github.com/streadway/amqp)

    [RabbitMQ HA - Kubernetes](https://github.com/kubernetes/charts/tree/master/stable/rabbitmq-ha)
  27. MithunArunan revised this gist Jun 12, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions CNCF.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1 @@
    ![CNCF - Trail map](https://pbs.twimg.com/media/DfbNNaOWAAAlNbC.jpg:large)
  28. MithunArunan revised this gist Jun 11, 2018. 1 changed file with 5 additions and 0 deletions.
    5 changes: 5 additions & 0 deletions testing.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,3 +1,8 @@
    ## Unit Testing

    ## API Testing
    Postman

    ## Integration Testing

    Postman
  29. MithunArunan revised this gist May 29, 2018. 1 changed file with 9 additions and 0 deletions.
    9 changes: 9 additions & 0 deletions kubernetes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -2,6 +2,13 @@

    ## Access Kubernetes Cluster

    [Refer](https://docs.bitnami.com/kubernetes/how-to/configure-rbac-in-your-kubernetes-cluster/#use-case-1-create-user-with-limited-namespace-access)
    ```bash
    openssl genrsa -out mithun.key 2048
    openssl req -new -key mithun.key -out mithun.csr -subj "/CN=mithun/O=admin"
    openssl x509 -req -in mithun.csr -CA /etc/kubernetes/pki/ca.crt -CAkey CA_LOCATION/ca.key -CAcreateserial -out employee.crt -days 500
    ```

    ```bash
    kubectl config set-cluster <cluster_name> --server=https://<master-node-ip>:<master-node-port> --insecure-skip-tls-verify=true
    kubectl config get-clusters
    @@ -17,6 +24,8 @@ kubectl get pods
    ```




    ## Kubernetes Cluster configurations
    Grouping all the kubernetes and docker configurations in one place
    k8s-configs
  30. MithunArunan revised this gist May 28, 2018. 2 changed files with 31 additions and 13 deletions.
    16 changes: 16 additions & 0 deletions docker.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1 +1,17 @@
    # Docker - Standards
    Certain set of standards for creating images <TODO>
    /home/<app-name>
    /vol/data
    /vol/models - AI models

    ## Tagging images

    Before deploying any image let’s create another tag, preferably not with latest.
    <image-name>:master

    ##### Major release
    <image-name>:<version>

    #### Minor release
    <image-name>:<version>-<commit-id-7chars>
    [12 Factor App - Docker](https://github.com/docker/labs/tree/master/12factor)
    28 changes: 15 additions & 13 deletions kubernetes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,21 +1,21 @@
    # Kubernetes

    ## Dockerizing Applications
    Certain set of standards for creating images <TODO>
    /home/<app-name>
    /vol/data
    /vol/models - AI models
    ## Access Kubernetes Cluster

    ## Tagging images
    ```bash
    kubectl config set-cluster <cluster_name> --server=https://<master-node-ip>:<master-node-port> --insecure-skip-tls-verify=true
    kubectl config get-clusters

    Before deploying any image let’s create another tag, preferably not with latest.
    <image-name>:master
    kubectl config set-credentials <cluster_name> --client-certificate= --client-key= --cluster=<cluster_name>
    kubectl config set-credentials <cluster_name> --username=<username> --password=<password> --cluster=<cluster_name>

    ##### Major release
    <image-name>:<version>
    kubectl config set-context <cluster_name> --user=<cluster_name> --cluster=<cluster_name>
    kubectl config use-context <cluster_name>
    kubectl config view
    kubectl get pods

    ```

    #### Minor release
    <image-name>:<version>-<commit-id-7chars>

    ## Kubernetes Cluster configurations
    Grouping all the kubernetes and docker configurations in one place
    @@ -29,7 +29,7 @@ Before deploying any image let’s create another tag, preferably not with lates
    Kube-ops-view
    All other microservices

    #### k8s configs
    #### K8S configs

    ###### Deployment.yaml
    Create a label ‘app’ for grouping pods
    @@ -62,6 +62,8 @@ docker pull djenriquez/vault-ui
    docker run --cap-add=IPC_LOCK -p 8200:8200 -e 'VAULT_DEV_ROOT_TOKEN_ID=roottoken' -e 'VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200' -d --name=vault vault
    docker run -d -p 8201:8201 -e PORT=8201 -e VAULT_URL_DEFAULT=http://192.168.12.155:8200 -e VAULT_AUTH_DEFAULT=GITHUB --name vault-ui djenriquez/vault-ui



    ## Next Steps

    * Telepresence