Skip to content

Instantly share code, notes, and snippets.

@konradko

konradko/sqs_notes.md

Forked from bramswenson/sqs_notes.md
Created September 7, 2016 14:09
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save konradko/59a6ee74988de5c42bc24f8258fe3cee to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save konradko/59a6ee74988de5c42bc24f8258fe3cee to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -51,6 +51,7 @@

    ## Interesting Things

    * [Scaling SQS](http://aws.amazon.com/blogs/aws/scaling-with-amazon-sqs/)
    * [Benchmarking SQS](http://www.warski.org/blog/2014/06/benchmarking-sqs/) (nice article)
    * [SQS from the shell](https://github.com/mapbox/sqscmd) (could be useful for scripting/testing)
    * [Neato way to backup a queue from the folks at Medium](https://github.com/Medium/draccus) (looks very useful as a daemon too)
    @@ -62,5 +63,6 @@
    ## Alternatives

    * [CloudAMQP](https://www.cloudamqp.com/) (As an alternative I like this since it could be self-hosted at any time)
    * [Dotcloud suggested their hosted RabbitMQ customers to migrate to CloudAMQP](http://blog.dotcloud.com/sunsetting-the-rabbitmq-service)
    * [IronMQ](http://www.iron.io/mq)
    * [Beanstalk vs IronMQ vs SQS](http://www.sitepoint.com/message-queues-comparing-beanstalkd-ironmq-amazon-sqs/)
  2. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -41,6 +41,8 @@
    * Python - [boto.sqs](http://boto.readthedocs.org/en/latest/ref/sqs.html) [boto.sqs tutorial](http://boto.readthedocs.org/en/latest/sqs_tut.html)
    * Python - [aws-sdk](http://aws.amazon.com/sdk-for-python/)
    * Python - [mock sqs for testing](https://github.com/pcsforeducation/sqs-mock-python)
    * Python - [async SQS on tornado](https://github.com/bittorrent/tornado-sqs)
    * Python - [twisted worker daemon for SQS](https://github.com/mtak/txsqs)
    * Ruby - [aws-sdk](http://aws.amazon.com/sdk-for-ruby/)
    * Ruby - [sidekiq-sqs](https://github.com/jmoses/sidekiq-sqs)
    * [Super Simple Node and Ruby Example](http://128bitstudios.com/2012/06/03/sqs-amazons-simple-queue-service-with-node-js-and-ruby/)
    @@ -49,6 +51,7 @@

    ## Interesting Things

    * [Benchmarking SQS](http://www.warski.org/blog/2014/06/benchmarking-sqs/) (nice article)
    * [SQS from the shell](https://github.com/mapbox/sqscmd) (could be useful for scripting/testing)
    * [Neato way to backup a queue from the folks at Medium](https://github.com/Medium/draccus) (looks very useful as a daemon too)
    * [Queue abstraction on top of SQS and Rabbit](https://github.com/wishpond-dev/simple-messaging) (this is a great idea)
  3. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 3 additions and 0 deletions.
    3 changes: 3 additions & 0 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -36,8 +36,11 @@

    ## Programming SQS

    * Python - [SimpleQ](https://github.com/rdegges/simpleq) (+1 on this one for python code)
    * Python - [Kombu supports SQS](http://kombu.readthedocs.org/en/latest/index.html)
    * Python - [boto.sqs](http://boto.readthedocs.org/en/latest/ref/sqs.html) [boto.sqs tutorial](http://boto.readthedocs.org/en/latest/sqs_tut.html)
    * Python - [aws-sdk](http://aws.amazon.com/sdk-for-python/)
    * Python - [mock sqs for testing](https://github.com/pcsforeducation/sqs-mock-python)
    * Ruby - [aws-sdk](http://aws.amazon.com/sdk-for-ruby/)
    * Ruby - [sidekiq-sqs](https://github.com/jmoses/sidekiq-sqs)
    * [Super Simple Node and Ruby Example](http://128bitstudios.com/2012/06/03/sqs-amazons-simple-queue-service-with-node-js-and-ruby/)
  4. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 9 additions and 0 deletions.
    9 changes: 9 additions & 0 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -44,6 +44,15 @@
    * [Making the most of SQS](http://mauricio.github.io/2014/09/01/make-the-most-of-sqs.html)
    * Python - [Celery SQS transport](http://celery.readthedocs.org/en/latest/getting-started/brokers/sqs.html) (I dislike celery generally but include for completeness)

    ## Interesting Things

    * [SQS from the shell](https://github.com/mapbox/sqscmd) (could be useful for scripting/testing)
    * [Neato way to backup a queue from the folks at Medium](https://github.com/Medium/draccus) (looks very useful as a daemon too)
    * [Queue abstraction on top of SQS and Rabbit](https://github.com/wishpond-dev/simple-messaging) (this is a great idea)
    * [Sidekiq alternative based on SQS](https://github.com/phstc/shoryuken)
    * [Another abstraction, this time SQS or Redis](https://github.com/Sently/job-broker)
    * [AMQP <--> SQS Facade](https://github.com/markbirbeck/amqp-sqs)

    ## Alternatives

    * [CloudAMQP](https://www.cloudamqp.com/) (As an alternative I like this since it could be self-hosted at any time)
  5. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -46,6 +46,6 @@

    ## Alternatives

    * [CloudAMQP](https://www.cloudamqp.com/) (As an alternative I like this since it could be self-hosted eventually, more options are better I think)
    * [CloudAMQP](https://www.cloudamqp.com/) (As an alternative I like this since it could be self-hosted at any time)
    * [IronMQ](http://www.iron.io/mq)
    * [Beanstalk vs IronMQ vs SQS](http://www.sitepoint.com/message-queues-comparing-beanstalkd-ironmq-amazon-sqs/)
  6. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -46,5 +46,6 @@

    ## Alternatives

    * [CloudAMQP](https://www.cloudamqp.com/) (As an alternative I like this since it could be self-hosted eventually, more options are better I think)
    * [IronMQ](http://www.iron.io/mq)
    * [Beanstalk vs IronMQ vs SQS](http://www.sitepoint.com/message-queues-comparing-beanstalkd-ironmq-amazon-sqs/)
  7. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 8 additions and 1 deletion.
    9 changes: 8 additions & 1 deletion sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -39,5 +39,12 @@
    * Python - [boto.sqs](http://boto.readthedocs.org/en/latest/ref/sqs.html) [boto.sqs tutorial](http://boto.readthedocs.org/en/latest/sqs_tut.html)
    * Python - [aws-sdk](http://aws.amazon.com/sdk-for-python/)
    * Ruby - [aws-sdk](http://aws.amazon.com/sdk-for-ruby/)
    * Ruby - [sidekiq-sqs](https://github.com/jmoses/sidekiq-sqs)
    * [Super Simple Node and Ruby Example](http://128bitstudios.com/2012/06/03/sqs-amazons-simple-queue-service-with-node-js-and-ruby/)
    * [Making the most of SQS](http://mauricio.github.io/2014/09/01/make-the-most-of-sqs.html)
    * [Making the most of SQS](http://mauricio.github.io/2014/09/01/make-the-most-of-sqs.html)
    * Python - [Celery SQS transport](http://celery.readthedocs.org/en/latest/getting-started/brokers/sqs.html) (I dislike celery generally but include for completeness)

    ## Alternatives

    * [IronMQ](http://www.iron.io/mq)
    * [Beanstalk vs IronMQ vs SQS](http://www.sitepoint.com/message-queues-comparing-beanstalkd-ironmq-amazon-sqs/)
  8. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 11 additions and 2 deletions.
    13 changes: 11 additions & 2 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,8 +1,9 @@
    # Amazon SQS Notes


    ## Queue Structure
    ## Queue Structure

    * [Queues are distributed](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/SQSConcepts.html)
    * Queues are limited to 120000 in-flight messages (use more small queues instead of one large queue)
    * Queues may have any number of clients sending/receiving messages at once
    * [Queues can be created with a built in delay on message visibility](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-delay-queues.html)
    @@ -31,4 +32,12 @@
    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * [Using SNS to "subscribe" to queue may be the best approach for message processors](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * Processors are best single threaded single minded connecting to at most to 1 send and 1 receive queue
    * Message processors should be idempotent since each message may be delivered/processed more than once
    * Message processors should be idempotent since each message may be delivered/processed more than once

    ## Programming SQS

    * Python - [boto.sqs](http://boto.readthedocs.org/en/latest/ref/sqs.html) [boto.sqs tutorial](http://boto.readthedocs.org/en/latest/sqs_tut.html)
    * Python - [aws-sdk](http://aws.amazon.com/sdk-for-python/)
    * Ruby - [aws-sdk](http://aws.amazon.com/sdk-for-ruby/)
    * [Super Simple Node and Ruby Example](http://128bitstudios.com/2012/06/03/sqs-amazons-simple-queue-service-with-node-js-and-ruby/)
    * [Making the most of SQS](http://mauricio.github.io/2014/09/01/make-the-most-of-sqs.html)
  9. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 2 additions and 0 deletions.
    2 changes: 2 additions & 0 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -5,6 +5,8 @@

    * Queues are limited to 120000 in-flight messages (use more small queues instead of one large queue)
    * Queues may have any number of clients sending/receiving messages at once
    * [Queues can be created with a built in delay on message visibility](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-delay-queues.html)
    * [Queues can be created just for dumping messages whose processing has failed](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/SQSDeadLetterQueue.html)

    ## Message Lifecycle

  10. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 9 additions and 7 deletions.
    16 changes: 9 additions & 7 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,21 +4,16 @@
    ## Queue Structure

    * Queues are limited to 120000 in-flight messages (use more small queues instead of one large queue)
    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * [Using SNS to "subscribe" to queue may be the best approach fpr message processors](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * Any number of clients can send and receive messages to and from queues
    * Processors are best single threaded single minded connecting to at most to 1 send and 1 receive queue
    * Queues may have any number of clients sending/receiving messages at once

    ## Message Lifecycle

    * Messages are "best-effort ordered" given their [distributed nature](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html)
    * Message processors should be idempotent since each message may be delivered/processed more than once
    * Use the [Visibility timeout](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html) on queues and messages to ensure that a message is only available to a `ReceiveMessage` once or until the timeout expires
    * Adjust the visibility timeout in-flight to extend or cancel the timeout
    * Messages should be manually deleted from the queue after processing
    * [Messages have unique IDs, but the receipt handle is also required in order to delete them](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ImportantIdentifiers.html)


    ## Message Structure

    * Messages have a `body` for data
    @@ -27,4 +22,11 @@
    * Messages may define [timers](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-timers.html)
    * [Messages can be batched 10 at a time](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-batch-api.html)
    * 256KB limit total message size or batch message size
    * [Even with the 256KB limits, buffering and batching client side may reduce costs](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/buffering.html)
    * [Even with the 256KB limits, buffering and batching client side may reduce costs](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/buffering.html)

    ## Message Processors

    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * [Using SNS to "subscribe" to queue may be the best approach for message processors](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * Processors are best single threaded single minded connecting to at most to 1 send and 1 receive queue
    * Message processors should be idempotent since each message may be delivered/processed more than once
  11. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -5,9 +5,9 @@

    * Queues are limited to 120000 in-flight messages (use more small queues instead of one large queue)
    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * [Using SNS to "subscribe" to queue may be the best approach fpr message processors](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * Any number of clients can send and receive messages to and from queues
    * Processors are best single threaded single minded connecting to at most to 1 send and 1 receive queue
    * [Using SNS to "subscribe" to queue may be the best approach fpr message processors](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * Any number of clients can send and receive messages to and from queues
    * Processors are best single threaded single minded connecting to at most to 1 send and 1 receive queue

    ## Message Lifecycle

  12. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 9 additions and 5 deletions.
    14 changes: 9 additions & 5 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,19 +1,23 @@
    # Amazon SQS Notes


    ## Queue Structure

    * Queues are limited to 120000 in-flight messages (use more small queues instead of one large queue)
    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * [Using SNS to "subscribe" to queue may be the best approach fpr message processors](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * Any number of clients can send and receive messages to and from queues
    * Processors are best single threaded single minded connecting to at most to 1 send and 1 receive queue

    ## Message Lifecycle

    * Messages are "best-effort ordered" given their [distributed nature](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html)
    * Message processors should be idempotent since each message may be delivered/processed more than once
    * Queues are limited to 120000 in-flight messages (use more small queues instead of one large queue)
    * Use the [Visibility timeout](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html) on queues and messages to ensure that a message is only available to a `ReceiveMessage` once or until the timeout expires
    * Adjust the visibility timeout in-flight to extend or cancel the timeout
    * Messages should be manually deleted from the queue after processing
    * [Messages have unique IDs, but the receipt handle is also required in order to delete them](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ImportantIdentifiers.html)
    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * [Using SNS to "subscribe" to queue may be the best approach fpr message processors](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * Any numbuer of clients can send and receive messages to and from queues
    * Processors are best single threaded single minded connecting to at most to 1 send and 1 receive queue


    ## Message Structure

  13. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -9,7 +9,7 @@
    * Use the [Visibility timeout](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html) on queues and messages to ensure that a message is only available to a `ReceiveMessage` once or until the timeout expires
    * Adjust the visibility timeout in-flight to extend or cancel the timeout
    * Messages should be manually deleted from the queue after processing
    * [Messages have unique IDs, but the receipt handle known must be used for deleting them](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ImportantIdentifiers.html)
    * [Messages have unique IDs, but the receipt handle is also required in order to delete them](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ImportantIdentifiers.html)
    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * [Using SNS to "subscribe" to queue may be the best approach fpr message processors](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * Any numbuer of clients can send and receive messages to and from queues
  14. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -4,7 +4,7 @@
    ## Message Lifecycle

    * Messages are "best-effort ordered" given their [distributed nature](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html)
    * Message processors should be idempotent since they may be delivered/processed more than once
    * Message processors should be idempotent since each message may be delivered/processed more than once
    * Queues are limited to 120000 in-flight messages (use more small queues instead of one large queue)
    * Use the [Visibility timeout](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html) on queues and messages to ensure that a message is only available to a `ReceiveMessage` once or until the timeout expires
    * Adjust the visibility timeout in-flight to extend or cancel the timeout
  15. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -11,7 +11,7 @@
    * Messages should be manually deleted from the queue after processing
    * [Messages have unique IDs, but the receipt handle known must be used for deleting them](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ImportantIdentifiers.html)
    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * [Using SNS to "subscribe" to queue may be the best approach](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * [Using SNS to "subscribe" to queue may be the best approach fpr message processors](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * Any numbuer of clients can send and receive messages to and from queues
    * Processors are best single threaded single minded connecting to at most to 1 send and 1 receive queue

  16. @bramswenson bramswenson revised this gist Sep 3, 2014. 1 changed file with 20 additions and 2 deletions.
    22 changes: 20 additions & 2 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,8 +1,26 @@
    # Amazon SQS Notes


    ## Message Lifecycle

    * Messages are "best-effort ordered" given their [distributed nature](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html)
    * Message processors should be idempotent since they may be delivered/processed more than once
    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * Queues are limited to 120000 in-flight messages (use more small queues instead of one large queue)
    * Use the [Visibility timeout](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html) on queues and messages to ensure that a message is only available to a `ReceiveMessage` once or until the timeout expires
    * Adjust the visibility timeout in-flight to extend or cancel the timeout
    * Messages should be manually deleted from the queue after processing
    * [Messages have unique IDs, but the receipt handle known must be used for deleting them](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ImportantIdentifiers.html)
    * [Messages have unique IDs, but the receipt handle known must be used for deleting them](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ImportantIdentifiers.html)
    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * [Using SNS to "subscribe" to queue may be the best approach](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqssubscribe.html)
    * Any numbuer of clients can send and receive messages to and from queues
    * Processors are best single threaded single minded connecting to at most to 1 send and 1 receive queue

    ## Message Structure

    * Messages have a `body` for data
    * Message queues define their own line protocol (JSON is a simple human readbale choice)
    * Messages may have typed and validated attribues (best used for metadata, not actual data)
    * Messages may define [timers](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-message-timers.html)
    * [Messages can be batched 10 at a time](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-batch-api.html)
    * 256KB limit total message size or batch message size
    * [Even with the 256KB limits, buffering and batching client side may reduce costs](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/buffering.html)
  17. @bramswenson bramswenson created this gist Sep 3, 2014.
    8 changes: 8 additions & 0 deletions sqs_notes.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,8 @@
    # Amazon SQS Notes

    * Messages are "best-effort ordered" given their [distributed nature](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/DistributedQueues.html)
    * Message processors should be idempotent since they may be delivered/processed more than once
    * [Long Polling](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html) can reduce the number of `ReceiveMessage` calls
    * Queues are limited to 120000 in-flight messages (use more small queues instead of one large queue)
    * Messages should be manually deleted from the queue after processing
    * [Messages have unique IDs, but the receipt handle known must be used for deleting them](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/ImportantIdentifiers.html)