WW Henderson davesgonechina

How to deploy a Prefect agent to Google Kubernetes Engine

This post contains code and commands you can use to deploy Prefect agents to Google Cloud’s Google Kubernetes Engine. The agents stand ready to execute workflows triggered by Prefect projects. One agent can run tasks from multiple projects.

The example here demonstrates how to create a single agent with minimal customization. It is configured with a Dockerfile, which installs necessary dependencies, and a k8s.cfg file, which connects the system to a Prefect account.

Agents are deployed via the gcloud command-line utility and its kubectl extension. Proper permissions within a project on Google Cloud are required.

Getting started

Calibre Server on Linux

Introduction

Calibre is a powerful cross-platform, open source, ebook manager and editing platform. Its calibre-server component can be used to publish an e-book library on a local network. While you can launch calibre-server as a desktop application, it can also be run as a daemon on a headless Linux server.

This tutorial on setting up calibre-server using Ubuntu 14.04 is very good, but dated.

	javascript: Promise.all([import('https://unpkg.com/turndown@6.0.0?module'), import('https://unpkg.com/@tehshrike/readability@0.2.0'), ]).then(async ([{
	default: Turndown
	}, {
	default: Readability
	}]) => {

	/* Optional vault name */
	const vault = "Sapientes";

	/* Optional folder name such as "Clippings/" */

	javascript: Promise.all([import('https://unpkg.com/turndown@6.0.0?module'), import('https://unpkg.com/@tehshrike/readability@0.2.0'), ]).then(async ([{
	default: Turndown
	}, {
	default: Readability
	}]) => {

	/* Optional vault name */
	const vault = "";

	/* Optional folder name such as "Clippings/" */

	# update and upgrade installed packages
	apt update && apt upgrade

	# install cronie and termux-services
	pkg install cronie termux-services

	# enable the crond service
	sv-enable crond

	# install and enable vim

	# Combines several solutions found on the internet

	class ImplicitFTP_TLS(ftplib.FTP_TLS):
	"""FTP_TLS subclass to support implicit FTPS."""
	"""Constructor takes a boolean parameter ignore_PASV_host whether o ignore the hostname"""
	"""in the PASV response, and use the hostname from the session instead"""
	def __init__(self, args, *kwargs):
	self.ignore_PASV_host = kwargs.get('ignore_PASV_host') == True
	super().__init__(*args, {k: v for k, v in kwargs.items() if not k == 'ignore_PASV_host'})
	self._sock = None

	#!/bin/bash

	# Wylie Hobbs - 2018
	# Copy multiple tables at once with redshift unload-copy utility
	# Usage: ./redshift_unload_wrapper.sh
	# Requires a clone of amazon-redshift-utils and the packages: jq, awscli
	# Create redshift_unload_wrapper.sh, tables.txt and config.json in src/UnloadCopyUtility and run this script

	#tables.txt is newline delimited list of tables in SCHEMA.TABLE notation
	TABLES=$(cat tables.txt)

	# Found in https://github.com/dbt-labs/dbt-core/issues/7749

	1. Install jq
	2. dbt --quiet ls --output json --output-keys tags \| jq .tags \| jq -s 'add \| unique'


	def enable_download_in_headless_chrome(driver, download_dir):
	#add missing support for chrome "send_command" to selenium webdriver
	driver.command_executor._commands["send_command"] = ("POST", '/session/$sessionId/chromium/send_command')

	params = {'cmd': 'Page.setDownloadBehavior', 'params': {'behavior': 'allow', 'downloadPath': download_dir}}
	driver.execute("send_command", params)

	enable_download_in_headless_chrome(driver, os.getcwd())

	import pandas as pd
	import pandas.io.sql as sqlio
	import psycopg2

	conn = psycopg2.connect("host='{}' port={} dbname='{}' user={} password={}".format(host, port, dbname, username, pwd))
	sql = "select count(*) from table;"
	dat = sqlio.read_sql_query(sql, conn)
	conn = None