Skip to content

Instantly share code, notes, and snippets.

@muriloaj

muriloaj/awc-ecs-access-to-aws-efs.md

Forked from duluca/awc-ecs-access-to-aws-efs.md
Created May 9, 2018 22:24
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save muriloaj/4636a0ea6a7bb415db628b22e31866ab to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save muriloaj/4636a0ea6a7bb415db628b22e31866ab to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @duluca duluca revised this gist Jan 22, 2018. 1 changed file with 7 additions and 7 deletions.
    14 changes: 7 additions & 7 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -84,13 +84,13 @@ This is a big deal.
    4. Service name: app-name
    5. No of tasks: 2, min healthy: 100, max healthy: 200 for highly available blue/green deployment setup
    6. Configure ELB
    1. Application Load Balancer
    2. ecsServiceRole
    3. Select app-name-prod-elb from before
    4. Select app-name:0:3000 container from before
    5. Add to ELB
    6. Target Group Name: app-name from before
    7. Save
    * Application Load Balancer
    * ecsServiceRole
    * Select app-name-prod-elb from before
    * Select app-name:0:3000 container from before
    * Add to ELB
    * Target Group Name: app-name from before
    * Save
    7. Create Service
    8. View Service
    9. Verify information
  2. @duluca duluca revised this gist Jan 22, 2018. 1 changed file with 7 additions and 7 deletions.
    14 changes: 7 additions & 7 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -84,13 +84,13 @@ This is a big deal.
    4. Service name: app-name
    5. No of tasks: 2, min healthy: 100, max healthy: 200 for highly available blue/green deployment setup
    6. Configure ELB
    1. Application Load Balancer
    2. ecsServiceRole
    3. Select app-name-prod-elb from before
    4. Select app-name:0:3000 container from before
    5. Add to ELB
    6. Target Group Name: app-name from before
    7. Save
    1. Application Load Balancer
    2. ecsServiceRole
    3. Select app-name-prod-elb from before
    4. Select app-name:0:3000 container from before
    5. Add to ELB
    6. Target Group Name: app-name from before
    7. Save
    7. Create Service
    8. View Service
    9. Verify information
  3. @duluca duluca revised this gist Jan 22, 2018. 1 changed file with 7 additions and 7 deletions.
    14 changes: 7 additions & 7 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -84,13 +84,13 @@ This is a big deal.
    4. Service name: app-name
    5. No of tasks: 2, min healthy: 100, max healthy: 200 for highly available blue/green deployment setup
    6. Configure ELB
    6.1. Application Load Balancer
    6.2. ecsServiceRole
    6.3. Select app-name-prod-elb from before
    6.4. Select app-name:0:3000 container from before
    6.5. Add to ELB
    6.6. Target Group Name: app-name from before
    6.7. Save
    1. Application Load Balancer
    2. ecsServiceRole
    3. Select app-name-prod-elb from before
    4. Select app-name:0:3000 container from before
    5. Add to ELB
    6. Target Group Name: app-name from before
    7. Save
    7. Create Service
    8. View Service
    9. Verify information
  4. @duluca duluca revised this gist Sep 6, 2017. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -74,7 +74,8 @@ This is a big deal.
    13. Next
    14. New target group, name: app-name
    15. Health-checks: Keep default "/" if serving a website on HTTP, but if deploying an API and/or redirecting all HTTP calls to HTTPS, ensure your app defines a custom route that is not redirected to HTTPS. On HTTP server GET "/healthCheck" return simple 200 message saying "I'm healthy" -- verify that this does not redirect to HTTPS, otherwise lot's of pain and suffering will occur. Health checks on AWS will fail.
    16. Next:Review, then Create
    16. DO NOT REGISTER ANY TARGETS: ECS will do this for you, if you do so yourself, you will provision a semi-broken infrastructure
    17. Next:Review, then Create

    #### Create Service
    1. Go to Amazon ECS
  5. @duluca duluca revised this gist Aug 17, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion awc-ecs-access-to-aws-efs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -151,4 +151,4 @@ Test what you have done.

    Go ahead and save some data.

    Then scale your instance size down to 0 and scale it back up again and see if the data is still accessible.
    Then scale your EC2 instance size down to 0 (the sledgehammer) and scale it back up again and see if the data is still accessible.
  6. @duluca duluca revised this gist Aug 17, 2017. 2 changed files with 83 additions and 53 deletions.
    127 changes: 82 additions & 45 deletions awc-ecs-access-to-aws-efs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,9 +4,37 @@ If you would like to persist data from your ECS containers, i.e. hosting databas
    > Don't know how to create your own AWS ECS Cluster? Go [here](https://gist.github.com/duluca/ebcf98923f733a1fdb6682f111b1a832#file-step-by-step-how-to-for-aws-ecs-md)!
    ## New Cluster
    - Sadly the EC2 provisioning process doesn't allow you to configure EFS during the initial config. After your create your cluster, follow the guide below.

    ## Existing ECS Cluster
    Sadly the EC2 provisioning process doesn't allow you to configure EFS during the initial config. After your create your cluster, follow the guide below.

    ## New Task Definition for Web App
    If you're using an Alpine-based Node server like [duluca/minimal-node-web-server]() follow this guide:
    0. Go to Amazon ECS
    1. Task Definitions -> Create new Task Definition
    2. Name: app-name-task, role: none, network: bridge
    3. Add container, name: app-name from before, image: URI from before, but append ":latest"
    4. Soft limit, 256 MB for Node.js
    5. Port mappings, Container port: 3000
    6. Log configuration: awslogs; app-name-logs, region, app-name-prod

    ## New Task Definition for Database
    If you're hosting a lightweight database like [mongo](https://hub.docker.com/_/mongo/) or [excellalabs/mongo](https://hub.docker.com/r/excellalabs/mongo/):
    0. Go to Amazon ECS
    1. Task Definitions -> Create new Task Definition
    2. Name: mongodb-task, role: none, network: bridge
    3. Add container, name: mongodb-prod, image: mongo or excellalabs/mongo, append a version number like ":3.4.7"
    4. Soft limit, 1024 MB
    5. Port mappings, Container port: 27017
    6. Log configuration: awslogs; mongodb-prod-logs, region, mongodb-prod
    7. Add Env Variables, see [excellalabs/mongo repo](https://github.com/excellalabs/mongo-docker) for details
    MONGODB_ADMIN_PASS
    MONGODB_APPLICATION_DATABASE
    MONGODB_APPLICATION_PASS
    MONGODB_APPLICATION_USER
    > It is not a security best practice to store such secrets in an encrypted form. If you'd like to do the right way, here's your homework: https://aws.amazon.com/blogs/compute/managing-secrets-for-amazon-ecs-applications-using-parameter-store-and-iam-roles-for-tasks/
    8. Then create a new service based on this task definition.
    8.1. Make sure that under Deployment Options Minimum healthy percent is 0 and Maximum percent 100. You don't ever want to seperate Mongo instances mounted to the same data source.

    ## Existing ECS Cluster with Existing Task Definition for Container

    ### Create a new KMS encryption key
    If you would like to encrypt your file system at-rest, then you must have a KMS key.
    @@ -33,53 +61,62 @@ If you would like to encrypt your file system at-rest, then you must have a KMS
    9. Back on the EFS main page, expand the EFS definition, if not already expanded
    10. Copy the **DNS name**

    ### Create a new launch configuration
    1. ECS -> Cluster
    2. Switch to ECS Instances tab
    3. Actions -> View Cluster Resources
    4. Click on the 'Launch configuration' that is linked
    5. Select the correct Launch configuration on the table and hit 'Copy launch configuration'
    6. Switch to 'Configure Details' tab
    7. Expand **Advanced Details**
    8. Paste the following script in to the **User data** field:
    ```Bash
    #!/bin/bash
    # Install nfs-utils
    cloud-init-per once yum_update yum update -y
    cloud-init-per once install_nfs_utils yum install -y nfs-utils

    # Create /efs folder
    cloud-init-per once mkdir_efs mkdir /efs

    EFS_URI=

    # Mount /efs
    cloud-init-per once mount_efs echo -e '$EFS_URI:/ /efs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0' >> /etc/fstab
    mount -a

    # Set any ECS agent configuration options
    echo "ECS_CLUSTER=default" >> /etc/ecs/ecs.config
    ### Update Your Cloud Formation Template
    1. CloudFormation
    2. Select EC2ContainerService-cluster-name
    3. View/edit design template
    4. Modify the YML to add `EfsUri` amongst the input parameters
    ```yml
    EfsUri:
    Type: String
    Description: >
    EFS volume DNS URI you would like to mount your EC2 instances to. Directory -> /mnt/efs
    Default: ''
    ```
    9. Define EFS_URI using the DNS name copied from the previous part
    10. If you are not using the default cluster, be sure to replace the ECS_CLUSTER=default line
    11. Skip to review
    12. Create launch configuration
    13. Proceed without a key pair
    14. Note down the name your new configuration

    ### Update Existing Auto Scaling Group
    1. ECS -> Cluster
    2. Switch to ECS Instances tab
    3. Actions -> View Cluster Resources
    4. Click on the 'Auto Scaling Group' that is linked
    5. Select the correct Launch configuration on the table and hit Actions -> Edit
    6. Update the Launch Configuration to the new one you just created
    7. Save
    5. Find `EcsInstanceLc` update its `UserData` property to look like:
    ```yml
    UserData: !If
    - SetEndpointToECSAgent
    - Fn::Base64: !Sub |
    #!/bin/bash
    # Install nfs-utils
    cloud-init-per once yum_update yum update -y
    cloud-init-per once install_nfs_utils yum install -y nfs-utils
    # Create /efs folder
    cloud-init-per once mkdir_efs mkdir /efs
    # Mount /efs
    cloud-init-per once mount_efs echo -e '${EfsUri}:/ /efs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0' >> /etc/fstab
    mount -a
    echo ECS_CLUSTER=${EcsClusterName} >> /etc/ecs/ecs.config
    echo ECS_BACKEND_HOST=${EcsEndpoint} >> /etc/ecs/ecs.config
    - Fn::Base64: !Sub |
    #!/bin/bash
    # Install nfs-utils
    cloud-init-per once yum_update yum update -y
    cloud-init-per once install_nfs_utils yum install -y nfs-utils
    # Create /efs folder
    cloud-init-per once mkdir_efs mkdir /efs
    # Mount /efs
    cloud-init-per once mount_efs echo -e '${EfsUri}:/ /efs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0' >> /etc/fstab
    mount -a
    echo ECS_CLUSTER=${EcsClusterName} >> /etc/ecs/ecs.config
    ```
    5. Validate the template
    6. Save the template to S3 and copy the URL
    7. Select your CloudFormation stack again -> Update stack
    8. Paste in the S3 url -> Next
    9. Now you'll see an `EfsUri` parameter, define it using the DNS name copied from the previous part
    10. On the review screen make sure it is only updating the Auto Scaling Group (ASG) and the Launch Configuration (LC)
    11. Let it update the stack

    ### And Now, The Fun Part -- Updating Your ECS Instances
    1. ECS -> Cluster
    2. Switch to ECS Instances tab
    There are two paths forward here, one is the hammer, which will **bring down** your applications:
    There are two paths forward here, one is the sledgehammer, which will **bring down** your applications:
    3. Scale ECS instances to 0 **Note** This is the part where your applications come down
    4. After all instances have been brougt down, scale back up to 2 (or more)
    Or perform a rolling update, which will **keep alive** your application:
    9 changes: 1 addition & 8 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -49,17 +49,10 @@ This is a big deal.
    “imageRepo”: “000000000000.dkr.ecr.us-east-1.amazonaws.com/app-name"
    5. Create

    ### Create Task Role
    1. IAM -> Roles
    2. Create new role
    3. Select Amanzon EC2
    4. Select 'AmazonEC2ContainerServiceFullAccess' & 'AmazonEC2ContainerServiceRole' policy -> Next
    5. prod-ecs-taskRole

    #### Create Task Definition
    0. Go to Amazon ECS
    1. Task Definitions -> Create new Task Definition
    2. Name: app-name-task, role: 'prod-ecs-taskRole', network: bridge
    2. Name: app-name-task, role: none, network: bridge
    3. Add container, name: app-name from before, image: URI from before, but append ":latest"
    4. Soft limit, 256 MB for Node.js
    5. Port mappings, Container port: 3000
  7. @duluca duluca revised this gist Aug 17, 2017. 1 changed file with 6 additions and 1 deletion.
    7 changes: 6 additions & 1 deletion awc-ecs-access-to-aws-efs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -79,8 +79,13 @@ echo "ECS_CLUSTER=default" >> /etc/ecs/ecs.config
    ### And Now, The Fun Part -- Updating Your ECS Instances
    1. ECS -> Cluster
    2. Switch to ECS Instances tab
    3. Scale ECS instances to 0 **Note** This will bring down your applications
    There are two paths forward here, one is the hammer, which will **bring down** your applications:
    3. Scale ECS instances to 0 **Note** This is the part where your applications come down
    4. After all instances have been brougt down, scale back up to 2 (or more)
    Or perform a rolling update, which will **keep alive** your application:
    3. Click on the EC2 instance and on the EC2 dashboard, select Actions -> State -> Terminate
    4. Wait while the instance is terminated and reprovisioned
    5. Rinse and repeat for the next instance

    ### Update Task Definition to Mount to the EFS Volume
    1. ECS -> Task definitions
  8. @duluca duluca revised this gist Aug 16, 2017. 2 changed files with 5 additions and 5 deletions.
    4 changes: 2 additions & 2 deletions awc-ecs-access-to-aws-efs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -18,7 +18,7 @@ If you would like to encrypt your file system at-rest, then you must have a KMS
    7. Carefuly select 'Key Administrators'
    8. Uncheck 'Allow key administrators to delete this key.' to prevent accidental deletions
    9. Key Usage Permissions
    10.Select the 'Task Role' that was created when configuring your AWS ECS Cluster. If not see the **Create Task Role** section in the guide linked above. You'll need to update existing task definitions, and update your service with the new task definition for the changes to take affect.
    10. Select the 'Task Role' that was created when configuring your AWS ECS Cluster. If not see the **Create Task Role** section in the guide linked above. You'll need to update existing task definitions, and update your service with the new task definition for the changes to take affect.
    11. Finish

    ### Create a new EFS
    @@ -93,7 +93,7 @@ echo "ECS_CLUSTER=default" >> /etc/ecs/ecs.config
    8. Select mount point 'efs'
    9. Provide the internal container path. i.e. for MongoDB default is '/data/db'
    10. Update
    11.Create
    11. Create

    ### Update ECS Service with the new Task Definition
    1. ECS -> Clusters
    6 changes: 3 additions & 3 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -24,9 +24,9 @@ Some of the instructions make references to `package.json` for `npm script for A
    6. Create new prod-vpc
    7. Create new prod-security-group
    8. Allow port 80 and 443 for HTTP and HTTPS inbound
    9.Allow port range 32768-61000 so that ECS can dynamically scale instances and run healh checks
    10.Container instance IAM role: select 'prod-ecs-instanceRole' that you just created, if not 'ecsIntanceRole'
    11.Create
    9. Allow port range 32768-61000 so that ECS can dynamically scale instances and run healh checks
    10. Container instance IAM role: select 'prod-ecs-instanceRole' that you just created, if not 'ecsIntanceRole'
    11. Create

    #### Verify Security Group Config
    This is a big deal.
  9. @duluca duluca revised this gist Aug 15, 2017. 1 changed file with 22 additions and 14 deletions.
    36 changes: 22 additions & 14 deletions awc-ecs-access-to-aws-efs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -67,7 +67,7 @@ echo "ECS_CLUSTER=default" >> /etc/ecs/ecs.config
    13. Proceed without a key pair
    14. Note down the name your new configuration

    ## Update Existing Auto Scaling Group
    ### Update Existing Auto Scaling Group
    1. ECS -> Cluster
    2. Switch to ECS Instances tab
    3. Actions -> View Cluster Resources
    @@ -76,29 +76,37 @@ echo "ECS_CLUSTER=default" >> /etc/ecs/ecs.config
    6. Update the Launch Configuration to the new one you just created
    7. Save

    ## And Now, The Fun Part -- Updating Your ECS Instances
    ### And Now, The Fun Part -- Updating Your ECS Instances
    1. ECS -> Cluster
    2. Switch to ECS Instances tab
    3. Scale ECS instances to 0 **Note** This will bring down your applications
    4. After all instances have been brougt down, scale back up to 2 (or more)

    ## Update Task Definition to Mount to the EFS Volume
    ### Update Task Definition to Mount to the EFS Volume
    1. ECS -> Task definitions
    2. Create new revision
    3. Add volume
    4. Name: 'efs', Source Path: '/mnt/efs/your-dir'
    5. Add
    6. Click on container name, under Storage and Logs
    7. Select mount point 'efs'
    8. Provide the internal container path. i.e. for MongoDB default is '/data/db'
    9. Update
    10.Create

    ## Update ECS Service with the new Task Definition
    3. If you already have not added it, make sure the Role here matches the one for the KMS key
    4. Add volume
    5. Name: 'efs', Source Path: '/mnt/efs/your-dir'
    6. Add
    7. Click on container name, under Storage and Logs
    8. Select mount point 'efs'
    9. Provide the internal container path. i.e. for MongoDB default is '/data/db'
    10. Update
    11.Create

    ### Update ECS Service with the new Task Definition
    1. ECS -> Clusters
    2. Click on Service name
    3. Update
    4. Type in the new task definition name
    5. Update service

    Your service should re-provision the existing containers and **voila, you're done!**
    Your service should re-provision the existing containers and **voila, you're done!**

    ## Last, But Not Least -- Test
    Test what you have done.

    Go ahead and save some data.

    Then scale your instance size down to 0 and scale it back up again and see if the data is still accessible.
  10. @duluca duluca revised this gist Aug 15, 2017. 2 changed files with 100 additions and 10 deletions.
    90 changes: 82 additions & 8 deletions awc-ecs-access-to-aws-efs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,9 +4,44 @@ If you would like to persist data from your ECS containers, i.e. hosting databas
    > Don't know how to create your own AWS ECS Cluster? Go [here](https://gist.github.com/duluca/ebcf98923f733a1fdb6682f111b1a832#file-step-by-step-how-to-for-aws-ecs-md)!
    ## New Cluster
    - Follow the directions in the referenced guide to start setting up your cluster.
    - When configuring the EC2 Container Instance, choose **Advanced Details**
    - Paste the following script in to the **User data** field:
    - Sadly the EC2 provisioning process doesn't allow you to configure EFS during the initial config. After your create your cluster, follow the guide below.

    ## Existing ECS Cluster

    ### Create a new KMS encryption key
    If you would like to encrypt your file system at-rest, then you must have a KMS key.
    > If not, you may skip but it is **strongly** recommended that you encrypt your data - no matter how unimportant you think your data is at the moment.
    3. Headover to IAM -> Encryption Keys
    4. Create key
    5. Provide Alias and a description
    6. Tag with 'Environment': 'production'
    7. Carefuly select 'Key Administrators'
    8. Uncheck 'Allow key administrators to delete this key.' to prevent accidental deletions
    9. Key Usage Permissions
    10.Select the 'Task Role' that was created when configuring your AWS ECS Cluster. If not see the **Create Task Role** section in the guide linked above. You'll need to update existing task definitions, and update your service with the new task definition for the changes to take affect.
    11. Finish

    ### Create a new EFS
    1. Launch EFS
    2. Create file system
    3. Select the VPC that your ECS cluster resides in
    4. Select the AZs that your container instances reside in
    5. Next
    6. Add a name
    7. Enable encryption (You WANT this -- see above)
    8. Create File System
    9. Back on the EFS main page, expand the EFS definition, if not already expanded
    10. Copy the **DNS name**

    ### Create a new launch configuration
    1. ECS -> Cluster
    2. Switch to ECS Instances tab
    3. Actions -> View Cluster Resources
    4. Click on the 'Launch configuration' that is linked
    5. Select the correct Launch configuration on the table and hit 'Copy launch configuration'
    6. Switch to 'Configure Details' tab
    7. Expand **Advanced Details**
    8. Paste the following script in to the **User data** field:
    ```Bash
    #!/bin/bash
    # Install nfs-utils
    @@ -16,15 +51,54 @@ cloud-init-per once install_nfs_utils yum install -y nfs-utils
    # Create /efs folder
    cloud-init-per once mkdir_efs mkdir /efs

    EFS_URI=

    # Mount /efs
    cloud-init-per once mount_efs echo -e 'fs-abcd1234.efs.us-east-1.amazonaws.com:/ /efs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0' >> /etc/fstab
    cloud-init-per once mount_efs echo -e '$EFS_URI:/ /efs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0' >> /etc/fstab
    mount -a

    # Set any ECS agent configuration options
    echo "ECS_CLUSTER=default" >> /etc/ecs/ecs.config
    ```
    - If you are not using the default cluster, be sure to replace the ECS_CLUSTER=default line
    - If you are not using the default cluster, be sure to replace the ECS_CLUSTER=default line
    9. Define EFS_URI using the DNS name copied from the previous part
    10. If you are not using the default cluster, be sure to replace the ECS_CLUSTER=default line
    11. Skip to review
    12. Create launch configuration
    13. Proceed without a key pair
    14. Note down the name your new configuration

    ## Existing ECS Cluster
    If you have an existing ECS Cluster some ex
    ## Update Existing Auto Scaling Group
    1. ECS -> Cluster
    2. Switch to ECS Instances tab
    3. Actions -> View Cluster Resources
    4. Click on the 'Auto Scaling Group' that is linked
    5. Select the correct Launch configuration on the table and hit Actions -> Edit
    6. Update the Launch Configuration to the new one you just created
    7. Save

    ## And Now, The Fun Part -- Updating Your ECS Instances
    1. ECS -> Cluster
    2. Switch to ECS Instances tab
    3. Scale ECS instances to 0 **Note** This will bring down your applications
    4. After all instances have been brougt down, scale back up to 2 (or more)

    ## Update Task Definition to Mount to the EFS Volume
    1. ECS -> Task definitions
    2. Create new revision
    3. Add volume
    4. Name: 'efs', Source Path: '/mnt/efs/your-dir'
    5. Add
    6. Click on container name, under Storage and Logs
    7. Select mount point 'efs'
    8. Provide the internal container path. i.e. for MongoDB default is '/data/db'
    9. Update
    10.Create

    ## Update ECS Service with the new Task Definition
    1. ECS -> Clusters
    2. Click on Service name
    3. Update
    4. Type in the new task definition name
    5. Update service

    Your service should re-provision the existing containers and **voila, you're done!**
    20 changes: 18 additions & 2 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,6 +6,15 @@ This is a multi-step configuration -- easy mistakes are likely. Be patient! The
    Some of the instructions make references to `package.json` for `npm script for AWS ECS` users. You may safely ignore these steps.

    ### Creating Amazon ECS Infrastructure

    #### Create a new IAM role
    > If you plan on having multiple clusters (which is likely to happen at some point) then you should define its own IAM role to prevent any future unintended or malicious access AWS resources.
    1. IAM -> Roles
    2. Create new role
    3. Select Amanzon EC2
    4. Select AmazonEC2ContainerServiceforEC2Role policy -> Next
    5. prod-ecs-instanceRole

    #### Create Cluster
    1. Go to Amazon ECS
    2. Clusters -> Create Cluster
    @@ -16,7 +25,7 @@ Some of the instructions make references to `package.json` for `npm script for A
    7. Create new prod-security-group
    8. Allow port 80 and 443 for HTTP and HTTPS inbound
    9.Allow port range 32768-61000 so that ECS can dynamically scale instances and run healh checks
    10.Container instance IAM role: ecsIntanceRole
    10.Container instance IAM role: select 'prod-ecs-instanceRole' that you just created, if not 'ecsIntanceRole'
    11.Create

    #### Verify Security Group Config
    @@ -40,10 +49,17 @@ This is a big deal.
    “imageRepo”: “000000000000.dkr.ecr.us-east-1.amazonaws.com/app-name"
    5. Create

    ### Create Task Role
    1. IAM -> Roles
    2. Create new role
    3. Select Amanzon EC2
    4. Select 'AmazonEC2ContainerServiceFullAccess' & 'AmazonEC2ContainerServiceRole' policy -> Next
    5. prod-ecs-taskRole

    #### Create Task Definition
    0. Go to Amazon ECS
    1. Task Definitions -> Create new Task Definition
    2. Name: app-name-task, role: none, network: bridge
    2. Name: app-name-task, role: 'prod-ecs-taskRole', network: bridge
    3. Add container, name: app-name from before, image: URI from before, but append ":latest"
    4. Soft limit, 256 MB for Node.js
    5. Port mappings, Container port: 3000
  11. @duluca duluca revised this gist Aug 15, 2017. 1 changed file with 9 additions and 8 deletions.
    17 changes: 9 additions & 8 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -12,14 +12,12 @@ Some of the instructions make references to `package.json` for `npm script for A
    3. Name: prod-ecs-cluster
    4. On-Demand Instance
    5. 2 m4.large instances across two AZs for highly available config
    6. Advanced details:
    - If you'd like to persist data in your containers via Docker volume mounting, then configure EFS. See [this guide]() first and come back.
    7. Create new prod-vpc
    8. Create new prod-security-group
    9. Allow port 80 and 443 for HTTP and HTTPS inbound
    10.Allow port range 32768-61000 so that ECS can dynamically scale instances and run healh checks
    11.Container instance IAM role: ecsIntanceRole
    12.Create
    6. Create new prod-vpc
    7. Create new prod-security-group
    8. Allow port 80 and 443 for HTTP and HTTPS inbound
    9.Allow port range 32768-61000 so that ECS can dynamically scale instances and run healh checks
    10.Container instance IAM role: ecsIntanceRole
    11.Create

    #### Verify Security Group Config
    This is a big deal.
    @@ -118,6 +116,9 @@ Now you need to deploy an application on your newly-minted cloud infrastructure.
    Go to the ELB DNS address and see if your app works.
    If you used Route 53 to connect your domain with your ELB or through your own DNS provider, then go to the URL and see if things work.

    ### I Would Like to Persist Data
    If you'd like to persist data in your containers via Docker volume mounting, then configure EFS. See [this guide](https://gist.github.com/duluca/ebcf98923f733a1fdb6682f111b1a832#file-awc-ecs-access-to-aws-efs-md).

    ## Troubleshooting
    1. ELB DNS works, but URL doesn't? Your DNS configuration is wrong.
    2. ELB DNS doesn't work. Then check the health of your ECS Service, see step 3 below.
  12. @duluca duluca revised this gist Aug 15, 2017. 2 changed files with 9 additions and 7 deletions.
    2 changes: 1 addition & 1 deletion awc-ecs-access-to-aws-efs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    ## Configuring AWS ECS to have access to AWS EFS
    If you would like to persist data from your ECS containers, i.e. hosting databases like MySQL or MongoDB with Docker, you need to ensure that you can mount the data directory of the database in the container to volume that's not going to dissappear when your container or worse yet, the EC2 instance that hosts your containers, is restarted or scaled up or down for any reason.

    > Don't know how to create your own AWS ECS Cluster? Go [here](https://gist.github.com/duluca/ebcf98923f733a1fdb6682f111b1a832#file-step-by-step-how-to-for-aws-ecs-md
    > Don't know how to create your own AWS ECS Cluster? Go [here](https://gist.github.com/duluca/ebcf98923f733a1fdb6682f111b1a832#file-step-by-step-how-to-for-aws-ecs-md)!
    ## New Cluster
    - Follow the directions in the referenced guide to start setting up your cluster.
    14 changes: 8 additions & 6 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -12,12 +12,14 @@ Some of the instructions make references to `package.json` for `npm script for A
    3. Name: prod-ecs-cluster
    4. On-Demand Instance
    5. 2 m4.large instances across two AZs for highly available config
    6. Create new prod-vpc
    7. Create new prod-security-group
    8. Allow port 80 and 443 for HTTP and HTTPS inbound
    9. Allow port range 32768-61000 so that ECS can dynamically scale instances and run healh checks
    10. Container instance IAM role: ecsIntanceRole
    11.Create
    6. Advanced details:
    - If you'd like to persist data in your containers via Docker volume mounting, then configure EFS. See [this guide]() first and come back.
    7. Create new prod-vpc
    8. Create new prod-security-group
    9. Allow port 80 and 443 for HTTP and HTTPS inbound
    10.Allow port range 32768-61000 so that ECS can dynamically scale instances and run healh checks
    11.Container instance IAM role: ecsIntanceRole
    12.Create

    #### Verify Security Group Config
    This is a big deal.
  13. @duluca duluca revised this gist Aug 15, 2017. 1 changed file with 30 additions and 0 deletions.
    30 changes: 30 additions & 0 deletions awc-ecs-access-to-aws-efs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,30 @@
    ## Configuring AWS ECS to have access to AWS EFS
    If you would like to persist data from your ECS containers, i.e. hosting databases like MySQL or MongoDB with Docker, you need to ensure that you can mount the data directory of the database in the container to volume that's not going to dissappear when your container or worse yet, the EC2 instance that hosts your containers, is restarted or scaled up or down for any reason.

    > Don't know how to create your own AWS ECS Cluster? Go [here](https://gist.github.com/duluca/ebcf98923f733a1fdb6682f111b1a832#file-step-by-step-how-to-for-aws-ecs-md
    ## New Cluster
    - Follow the directions in the referenced guide to start setting up your cluster.
    - When configuring the EC2 Container Instance, choose **Advanced Details**
    - Paste the following script in to the **User data** field:
    ```Bash
    #!/bin/bash
    # Install nfs-utils
    cloud-init-per once yum_update yum update -y
    cloud-init-per once install_nfs_utils yum install -y nfs-utils

    # Create /efs folder
    cloud-init-per once mkdir_efs mkdir /efs

    # Mount /efs
    cloud-init-per once mount_efs echo -e 'fs-abcd1234.efs.us-east-1.amazonaws.com:/ /efs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0' >> /etc/fstab
    mount -a

    # Set any ECS agent configuration options
    echo "ECS_CLUSTER=default" >> /etc/ecs/ecs.config
    ```
    - If you are not using the default cluster, be sure to replace the ECS_CLUSTER=default line
    - If you are not using the default cluster, be sure to replace the ECS_CLUSTER=default line

    ## Existing ECS Cluster
    If you have an existing ECS Cluster some ex
  14. @duluca duluca revised this gist Aug 12, 2017. 1 changed file with 0 additions and 1 deletion.
    1 change: 0 additions & 1 deletion step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -94,7 +94,6 @@ This is a big deal.
    "awsEcsCluster": "prod-ecs-cluster",
    "awsService": "app-name"
    ```

    #### Setup Logs
    1. cloudwatch -> logs
    2. Create Log group
  15. @duluca duluca revised this gist Aug 12, 2017. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -23,6 +23,7 @@ Some of the instructions make references to `package.json` for `npm script for A
    This is a big deal.
    1. Go EC2 -> Network & Security -> Security Groups
    2. Verify there ports are open:

    | Type | Protocol | Port Range | Source |
    | --- | --- | --- | --- |
    | HTTP (80) | TCP (6) |80 | 0.0.0.0/0 |
  16. @duluca duluca revised this gist Aug 12, 2017. 1 changed file with 6 additions and 11 deletions.
    17 changes: 6 additions & 11 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -23,18 +23,13 @@ Some of the instructions make references to `package.json` for `npm script for A
    This is a big deal.
    1. Go EC2 -> Network & Security -> Security Groups
    2. Verify there ports are open:
    | | | |
    | Type | Protocol | Port Range | Source |
    | --- | --- | --- | --- |
    |HTTP (80)|TCP (6)|80|0.0.0.0/0|
    |HTTP (80)|TCP (6)|80|::/0|
    |Custom TCP Rule|TCP (6)|32768-61000|0.0.0.0/0|
    |HTTPS (443)|TCP (6)|443|0.0.0.0/0|
    |HTTPS (443)|TCP (6)|443|::/0|

    | Command | Description |
    | --- | --- |
    | git status | List all new or modified files |
    | git diff | Show file differences that haven't been staged |
    | HTTP (80) | TCP (6) |80 | 0.0.0.0/0 |
    | HTTP (80) | TCP (6) |80 | ::/0 |
    | Custom TCP Rule | TCP (6) | 32768-61000 | 0.0.0.0/0 |
    | HTTPS (443) | TCP (6) | 443| 0.0.0.0/0 |
    | HTTPS (443) | TCP (6) | 443| ::/0 |

    #### Create Container Repository
    1. Go to Amazon ECS
  17. @duluca duluca revised this gist Aug 12, 2017. 1 changed file with 5 additions and 0 deletions.
    5 changes: 5 additions & 0 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -31,6 +31,11 @@ This is a big deal.
    |HTTPS (443)|TCP (6)|443|0.0.0.0/0|
    |HTTPS (443)|TCP (6)|443|::/0|

    | Command | Description |
    | --- | --- |
    | git status | List all new or modified files |
    | git diff | Show file differences that haven't been staged |

    #### Create Container Repository
    1. Go to Amazon ECS
    2. Repositories -> Create Repository
  18. @duluca duluca revised this gist Aug 12, 2017. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -23,7 +23,8 @@ Some of the instructions make references to `package.json` for `npm script for A
    This is a big deal.
    1. Go EC2 -> Network & Security -> Security Groups
    2. Verify there ports are open:
    |---|---|---|---|
    | | | |
    | --- | --- | --- | --- |
    |HTTP (80)|TCP (6)|80|0.0.0.0/0|
    |HTTP (80)|TCP (6)|80|::/0|
    |Custom TCP Rule|TCP (6)|32768-61000|0.0.0.0/0|
  19. @duluca duluca revised this gist Aug 12, 2017. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -23,6 +23,7 @@ Some of the instructions make references to `package.json` for `npm script for A
    This is a big deal.
    1. Go EC2 -> Network & Security -> Security Groups
    2. Verify there ports are open:
    |---|---|---|---|
    |HTTP (80)|TCP (6)|80|0.0.0.0/0|
    |HTTP (80)|TCP (6)|80|::/0|
    |Custom TCP Rule|TCP (6)|32768-61000|0.0.0.0/0|
  20. @duluca duluca revised this gist Aug 12, 2017. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    ## Setting up Your Own AWS ECS Cluster
    This is a multi-step configuration -- easy mistakes are likely. Be patient! The pay-off will be worth it. Rudimentary knowledge and awareness of the AWS landscape is not necessarily required, but will make it easier to set things up.

    > Enable fantastic Blue-Green deployments with _[npm scripts for AWS ECS]_(https://gist.github.com/duluca/2b67eb6c2c85f3d75be8c183ab15266e#file-npm-scripts-for-aws-ecs-md).
    > Enable fantastic Blue-Green deployments with [_npm scripts for AWS ECS_](https://gist.github.com/duluca/2b67eb6c2c85f3d75be8c183ab15266e#file-npm-scripts-for-aws-ecs-md).
    Some of the instructions make references to `package.json` for `npm script for AWS ECS` users. You may safely ignore these steps.

  21. @duluca duluca revised this gist Aug 12, 2017. 1 changed file with 43 additions and 33 deletions.
    76 changes: 43 additions & 33 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,5 @@
    ## Setting up Your Own AWS ECS Cluster
    This is a multi-step configuration -- easy mistakes are likely. Be patient! The pay-off will be worth it.
    This is a multi-step configuration -- easy mistakes are likely. Be patient! The pay-off will be worth it. Rudimentary knowledge and awareness of the AWS landscape is not necessarily required, but will make it easier to set things up.

    > Enable fantastic Blue-Green deployments with _[npm scripts for AWS ECS]_(https://gist.github.com/duluca/2b67eb6c2c85f3d75be8c183ab15266e#file-npm-scripts-for-aws-ecs-md).
    @@ -15,8 +15,19 @@ Some of the instructions make references to `package.json` for `npm script for A
    6. Create new prod-vpc
    7. Create new prod-security-group
    8. Allow port 80 and 443 for HTTP and HTTPS inbound
    9. Container instance IAM role: ecsIntanceRole
    10.Create
    9. Allow port range 32768-61000 so that ECS can dynamically scale instances and run healh checks
    10. Container instance IAM role: ecsIntanceRole
    11.Create

    #### Verify Security Group Config
    This is a big deal.
    1. Go EC2 -> Network & Security -> Security Groups
    2. Verify there ports are open:
    |HTTP (80)|TCP (6)|80|0.0.0.0/0|
    |HTTP (80)|TCP (6)|80|::/0|
    |Custom TCP Rule|TCP (6)|32768-61000|0.0.0.0/0|
    |HTTPS (443)|TCP (6)|443|0.0.0.0/0|
    |HTTPS (443)|TCP (6)|443|::/0|

    #### Create Container Repository
    1. Go to Amazon ECS
    @@ -74,41 +85,40 @@ Some of the instructions make references to `package.json` for `npm script for A
    11. Publish and release image with npm run aws:publish
    12. On the Service Events tabs keep an eye on health check errors


    - Update package.json
    #### Update package.json
    ```json
    "awsRegion": "us-east-1",
              "awsEcsCluster": "prod-ecs-cluster",
              "awsService": "app-name"

    cloudwatch -> logs
    1. Create Log group
    2. app-name-logs
    "awsEcsCluster": "prod-ecs-cluster",
    "awsService": "app-name"
    ```

    #### Setup Logs
    1. cloudwatch -> logs
    2. Create Log group
    3. app-name-logs

    Route 53
    #### Route 53 DNS Update
    > If you don't use Route 53, don't panic. Just create an A record to the ELB's DNS address and you're done.
    1. hosted zone
    2. select domain
    3. create record set
    4. alias 'yes'
    5. Select ELB App load balancer from the list
    6. create



    Open up port for
    HTTP (80)
    TCP (6)
    80
    0.0.0.0/0
    Custom TCP Rule
    TCP (6)
    32768-61000
    0.0.0.0/0
    HTTPS (443)
    TCP (6)
    443
    0.0.0.0/0
    HTTPS (443)
    TCP (6)
    443
    ::/0
    6. Create

    ## Phew!!
    ### Now what?
    Now you need to deploy an application on your newly-minted cloud infrastructure. Enable fantastic Blue-Green deployments with _[npm scripts for AWS ECS]_(https://gist.github.com/duluca/2b67eb6c2c85f3d75be8c183ab15266e#file-npm-scripts-for-aws-ecs-md).

    ### Then what?
    Go to the ELB DNS address and see if your app works.
    If you used Route 53 to connect your domain with your ELB or through your own DNS provider, then go to the URL and see if things work.

    ## Troubleshooting
    1. ELB DNS works, but URL doesn't? Your DNS configuration is wrong.
    2. ELB DNS doesn't work. Then check the health of your ECS Service, see step 3 below.
    3. Go to ECS -> Your Cluster -> click on Your Service and switch to the events tab:
    If you don't see `service your-app has reached a steady state.` then your container is having trouble starting or AWS is failing to perform a health check.
    4. To see what's wrong with your container, go to the Cloudwatch Logs you setup earlier and you'll be able to see the console logs of your application.
    5. Service is healthy, logs look fine. Things still don't work? Then re-check security group port rules and target group port rules and any AWS IAM security role you may have setup or may be overriding some default behavior that hasn't been covered.
    6. Call someone who knows better :)
  22. @duluca duluca created this gist Aug 12, 2017.
    114 changes: 114 additions & 0 deletions step-by-step-how-to-for-aws-ecs.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,114 @@
    ## Setting up Your Own AWS ECS Cluster
    This is a multi-step configuration -- easy mistakes are likely. Be patient! The pay-off will be worth it.

    > Enable fantastic Blue-Green deployments with _[npm scripts for AWS ECS]_(https://gist.github.com/duluca/2b67eb6c2c85f3d75be8c183ab15266e#file-npm-scripts-for-aws-ecs-md).
    Some of the instructions make references to `package.json` for `npm script for AWS ECS` users. You may safely ignore these steps.

    ### Creating Amazon ECS Infrastructure
    #### Create Cluster
    1. Go to Amazon ECS
    2. Clusters -> Create Cluster
    3. Name: prod-ecs-cluster
    4. On-Demand Instance
    5. 2 m4.large instances across two AZs for highly available config
    6. Create new prod-vpc
    7. Create new prod-security-group
    8. Allow port 80 and 443 for HTTP and HTTPS inbound
    9. Container instance IAM role: ecsIntanceRole
    10.Create

    #### Create Container Repository
    1. Go to Amazon ECS
    2. Repositories -> Create Repository
    3. Enter your app-name
    4. Copy repository URI, add to package.json
    “imageRepo”: “000000000000.dkr.ecr.us-east-1.amazonaws.com/app-name"
    5. Create

    #### Create Task Definition
    0. Go to Amazon ECS
    1. Task Definitions -> Create new Task Definition
    2. Name: app-name-task, role: none, network: bridge
    3. Add container, name: app-name from before, image: URI from before, but append ":latest"
    4. Soft limit, 256 MB for Node.js
    5. Port mappings, Container port: 3000
    6. Log configuration: awslogs; app-name-logs, region, app-name-prod

    #### Create ELB
    1. Go to Amazon EC2
    2. Load Balancers -> Create Load Balancer
    3. Application Load Balancer
    4. Name: app-name-prod-elb
    5. Add listener: HTTPS, 443
    6. AZs, select prod-vpc, select all
    7. Tags -> Domain, app-name.yourdomain.com
    8. Next
    9. Choose or create SSL cert (star is recommended: add *.yourdomain.com and yourdomain.com separately on the cert)
    10. Select default ELB security policy
    11. Next
    12. Create prod-cluster specific security group only allowing port 80 and 443 inbound
    13. Next
    14. New target group, name: app-name
    15. Health-checks: Keep default "/" if serving a website on HTTP, but if deploying an API and/or redirecting all HTTP calls to HTTPS, ensure your app defines a custom route that is not redirected to HTTPS. On HTTP server GET "/healthCheck" return simple 200 message saying "I'm healthy" -- verify that this does not redirect to HTTPS, otherwise lot's of pain and suffering will occur. Health checks on AWS will fail.
    16. Next:Review, then Create

    #### Create Service
    1. Go to Amazon ECS
    2. Clusters -> Select "prod-ecs-cluster"
    3. Task Definition: app-name-task from before
    4. Service name: app-name
    5. No of tasks: 2, min healthy: 100, max healthy: 200 for highly available blue/green deployment setup
    6. Configure ELB
    6.1. Application Load Balancer
    6.2. ecsServiceRole
    6.3. Select app-name-prod-elb from before
    6.4. Select app-name:0:3000 container from before
    6.5. Add to ELB
    6.6. Target Group Name: app-name from before
    6.7. Save
    7. Create Service
    8. View Service
    9. Verify information
    10. Build image with npm run image:build
    11. Publish and release image with npm run aws:publish
    12. On the Service Events tabs keep an eye on health check errors


    - Update package.json
    "awsRegion": "us-east-1",
              "awsEcsCluster": "prod-ecs-cluster",
              "awsService": "app-name"

    cloudwatch -> logs
    1. Create Log group
    2. app-name-logs


    Route 53
    1. hosted zone
    2. select domain
    3. create record set
    4. alias 'yes'
    5. Select ELB App load balancer from the list
    6. create



    Open up port for
    HTTP (80)
    TCP (6)
    80
    0.0.0.0/0
    Custom TCP Rule
    TCP (6)
    32768-61000
    0.0.0.0/0
    HTTPS (443)
    TCP (6)
    443
    0.0.0.0/0
    HTTPS (443)
    TCP (6)
    443
    ::/0