Skip to content

Instantly share code, notes, and snippets.

@a-hisame

a-hisame/gzip_s3_and_json.py

Created May 17, 2016 07:27
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save a-hisame/f90815f4fae695ad3f16cb48a81ec06e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save a-hisame/f90815f4fae695ad3f16cb48a81ec06e to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. a-hisame created this gist May 17, 2016.
    34 changes: 34 additions & 0 deletions gzip_s3_and_json.py
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,34 @@
    #!/usr/bin/env python
    # -*- coding: utf-8 -*-
    import json
    import gzip
    import StringIO

    import boto3

    def upload(bucket, key, obj):
    s3 = boto3.client('s3')
    inmem = StringIO.StringIO()
    with gzip.GzipFile(fileobj=inmem, mode='wb') as fh:
    fh.write(json.dumps(obj))
    inmem.seek(0)
    s3.put_object(Bucket=bucket, Body=inmem, Key=key)


    def download(bucket, key, else_value=None):
    s3 = boto3.client('s3')
    response = s3.get_object(Bucket=bucket, Key=key)
    content = response['Body'].read()
    with gzip.GzipFile(fileobj=StringIO.StringIO(content)) as fh:
    try:
    return json.loads(fh.read())
    except Exception as e:
    return else_value


    if __name__ == '__main__':
    BUCKET_NAME = ''
    KEY_NAME = ''
    upload(BUCKET_NAME, KEY_NAME, { u'あ': u'いうえお' })
    obj = download(BUCKET_NAME, KEY_NAME)
    print obj.get(u'あ')