import boto3
client = boto3.client('glue')
response = client.create_crawler(
Name='SalesCSVCrawler',
Role='AWSGlueServiceRoleDefault',
DatabaseName='sales-cvs',
Description='Crawler for generated Sales schema',
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import dotenv from "dotenv"; | |
| const API_KEY = process.env.API_KEY; | |
| const API_ENDPOINT = process.env.API_ENDPOINT; // ?api-version=2023-07-01-preview is required as of August 2023 | |
| const validate = async (name: string, project: string, role: string, category: string, feedback: string, summary: string): Promise<{ status: string, recommendations: string }> => { | |
| var nominee = generateNomineeString(name, project, role, category, feedback, summary); | |
| // Using function call capability of OpenAI API to ensure response is properly formatted JSON | |
| // Before function calling I asked to reply via JSON, one of side effect was that in 'recomendations' |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ``` | |
| (email_ml) jtzero:~/Documents/ml$ PYSPARK_DRIVER_PYTHON=jupyter PYSPARK_DRIVER_PYTHON_OPTS='notebook' pyspark | |
| Traceback (most recent call last): | |
| File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/jupyter", line 8, in <module> | |
| sys.exit(main()) | |
| File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/jupyter_core/command.py", line 247, in main | |
| command = _jupyter_abspath(subcommand) | |
| File "/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/lib/python3.7/site-packages/jupyter_core/command.py", line 134, in _jupyter_abspath | |
| 'Jupyter command `{}` not found.'.format(jupyter_subcommand) | |
| Exception: Jupyter command `jupyter-/home/jtzero/.local/share/virtualenvs/ml-PBkX3P8r/bin/find_spark_home.py` not found. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| FROM jupyter/pyspark-notebook | |
| USER root | |
| # Add essential packages | |
| RUN apt-get update && apt-get install -y build-essential curl git gnupg2 nano apt-transport-https software-properties-common | |
| # Set locale | |
| RUN apt-get update && apt-get install -y locales \ | |
| && echo "en_US.UTF-8 UTF-8" > /etc/locale.gen \ | |
| && locale-gen | |
| # Add config to Jupyter notebook | |
| COPY jupyter/jupyter_notebook_config.py /home/jovyan/.jupyter/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| FROM alpine:3.10 | |
| ARG SPARK_VERSION=3.0.0-preview | |
| ARG HADOOP_VERSION_SHORT=3.2 | |
| ARG HADOOP_VERSION=3.2.0 | |
| ARG AWS_SDK_VERSION=1.11.375 | |
| RUN apk add --no-cache bash openjdk8-jre python3 | |
| # Download and extract Spark |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @media print { | |
| *, ::after, ::before { | |
| text-shadow: none !important; | |
| box-shadow: none !important | |
| } | |
| a, a:visited { | |
| text-decoration: underline | |
| } |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| success: function(data) { | |
| var blob=new Blob([data]); | |
| var link=document.createElement('a'); | |
| link.href=window.URL.createObjectURL(blob); | |
| link.download="<FILENAME_TO_SAVE_WITH_EXTENSION>"; | |
| link.click(); | |
| } |
NewerOlder