vucong2409
/ ec2-required-lib.txt
Created
April 24, 2024 10:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| boto3==1.34.90 | |
| botocore==1.34.90 | |
| click==8.1.7 | |
| jmespath==1.0.1 | |
| python-dateutil==2.9.0.post0 | |
| s3transfer==0.10.1 | |
| six==1.16.0 | |
| urllib3==2.2.1 |
vucong2409
/ inject-ssm-secrets.py
Last active
May 2, 2024 03:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/python3 | |
| import click | |
| import boto3 | |
| from click import ClickException | |
| KEY_VALUE_LINE_PATTERN = "{}={}\n" | |
| @click.command() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import requests | |
| import json | |
| batchSizeParam = 100 | |
| # endIndexParam = 47142 | |
| endIndexParam = 47142 | |
| index_number = 0 | |
| apiBasePathParam = "{API_PATH}&endIndex=" | |
| batchSizeParamStr = "&batchsize=" |
vucong2409
/ spark_word_count_2.py
Last active
April 2, 2022 20:27
Spark WordCount2 with pattern file
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from operator import add | |
| from pyspark import SparkConf, SparkContext | |
| APP_NAME = "WordCount2" | |
| SPARK_MASTER_ADDRESS = "" | |
| HADOOP_ADDRESS = "" | |
| need_to_delete_char = [] | |
| with open("pattern.txt", "r", encoding='utf-8') as pattern_file: | |
| need_to_delete_char = pattern_file.read() \ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import scrapy | |
| URL_TEMPLATE = 'https://reviewedu.net/school/page/' | |
| URL_FIRST_PAGE = 'https://reviewedu.net/school/' | |
| INFO_BOX_XPATH = '//div[contains(@class,"box-text-products")]' | |
| UNI_NAME_XPATH = 'div[contains(@class,"title-wrapper")]/p/a/text()' | |
| RATING_XPATH = 'div[contains(@class,"rating-group")]/div[contains(@class,"rating-score__number")]/text()' | |
| NUMBER_OF_STUDENT_XPATH = 'div[contains(@class,"rating-score__text-6")]/text()' | |
| COURSE_LENGTH_XPATH = 'div[contains(@class,"hide-sm-less")]/div[contains(@class,"rating-score__text-5")]/text()' |