| .PHONY: psql up down venv check-deps update-deps install-deps isort black mypy flake8 bandit lint test migrate serve | |
| ifneq (,$(wildcard ./.env)) | |
| include .env | |
| export | |
| endif | |
| VENV=.venv | |
| PYTHON=$(VENV)/bin/python3 |
This guide assumes:
The following process is to create a Meltano application that contains the desired extractors and loaders, and then deploy it on GCP Kubernetes Engine.
Environment Requirements:
| """ | |
| AUTH_ROLE_ADMIN: the role of the bind user (should be Admin) | |
| AUTH_USER_REGISTRATION: boolean for automatically creating users on first log-in | |
| AUTH_USER_REGISTRATION_ROLE: the role which first-time users logging in will be assigned | |
| Possible Values: Admin, Viewer, User, Op, Public | |
| AUTH_LDAP_SERVER: the LDAP server URI | |
| AUTH_LDAP_SEARCH: update with the LDAP path under which you’d like the users to have access to Airflow (e.g. : 'dc=example,dc=com') |
| import com.uber.h3core.H3Core | |
| import scala.collection.JavaConversions._ | |
| import scala.collection.JavaConverters._ | |
| object H3 extends Serializable { | |
| val instance = H3Core.newInstance() | |
| } | |
| // Return the H3 | |
| val geoToH3 = udf{ (latitude: Double, longitude: Double, resolution: Int) => |
| #!/bin/bash | |
| curl -L -O https://downloads.redisinsight.redislabs.com/latest/redisinsight-linux64 | |
| chmod +x redisinsight-linux64 | |
| ./redisinsight-linux64 |
| from airflow.hooks.postgres_hook import PostgresHook | |
| from airflow.models import BaseOperator | |
| from airflow.utils.decorators import apply_defaults | |
| from datetime import datetime, timedelta | |
| from os import environ | |
| import csv | |
| class DataSourceToCsvOperator(BaseOperator): | |
| """ |
| http://www.oreilly.com/data/free/files/2014-data-science-salary-survey.pdf | |
| http://www.oreilly.com/data/free/files/2015-data-science-salary-survey.pdf | |
| http://www.oreilly.com/data/free/files/Data_Analytics_in_Sports.pdf | |
| http://www.oreilly.com/data/free/files/advancing-procurement-analytics.pdf | |
| http://www.oreilly.com/data/free/files/ai-and-medicine.pdf | |
| http://www.oreilly.com/data/free/files/analyzing-data-in-the-internet-of-things.pdf | |
| http://www.oreilly.com/data/free/files/analyzing-the-analyzers.pdf | |
| http://www.oreilly.com/data/free/files/architecting-data-lakes.pdf | |
| http://www.oreilly.com/data/free/files/being-a-data-skeptic.pdf | |
| http://www.oreilly.com/data/free/files/big-data-analytics-emerging-architecture.pdf |
| import os | |
| import re | |
| import yaml | |
| def parse_config(path=None, data=None, tag='!ENV'): | |
| """ | |
| Load a yaml configuration file and resolve any environment variables | |
| The environment variables must have !ENV before them and be in this format | |
| to be parsed: ${VAR_NAME}. |
