Michael Copeland jelloslinger

Dagster Installation in Docker

Dagster provides a mechanism to create pipeline DAGs which can be executed in a reproducible way. The DAGs can be visualized and run through a UI.

In order to run dagster, we first need to make sure we run python in a docker container.

The following code will create the python env in a docker container and will be removed right after we exit.

"g" is a threadlocal magic object that lets developer add / change / remove attributes during the request lifecycle. Learn more about this "g" here.

There is no OTB equivalent in FastAPI, but thanks to the new contextvars Python 3.7+ module, I made this simple demo.

Any comment or help to improve yhis recipe is welcome.

Python relative imports in AWS Lambda fail with attempted relative import with no known parent package

In AWS Lambda if I attempt an explicit relative import like this

.
├── lambda_file.py
└── example.py

	from typing import Optional, Generator
	from dataclasses import dataclass, field
	from enum import auto, Enum
	import asyncio
	from collections import defaultdict


	class State(Enum):
	OBSERVING = auto()
	OBSERVED = auto()

	export class FallbackOnErrorPlugin {
	constructor(fallbackURL) {
	this.fallbackURL = fallbackURL;
	}

	// This is called whenever there's a network response,
	// but we want special behavior for non-2xx statuses.
	fetchDidSucceed({response}) {
	if (response.ok) {
	// If it's a 2xx, it can be used as-is.

	from __future__ import annotations
	from typing import IO
	import asyncio
	import sys
	import contextlib
	import aiofiles.threadpool

	from chat_streams import split_lines, write, handle_writes

	async def handle_reads(reader: asyncio.StreamReader) -> None:

	version: '3.3'
	services:
	db:
	volumes:
	- app-db-data:/var/lib/postgresql/data/pgdata
	deploy:
	placement:
	constraints:
	- node.labels.${STACK_NAME}.app-db-data == true
	es01:

	import asyncio
	import random
	import time


	async def worker(name, queue):
	while True:
	# Get a "work item" out of the queue.
	sleep_for = await queue.get()

	<html>
	<head>
	<script src="qrc:///qtwebchannel/qwebchannel.js"></script>
	<style>
	::selection {
	background:transparent;
	}
	</style>
	</head>

	---
	AWSTemplateFormatVersion: '2010-09-09'
	Description: Public VPC and subnets

	Resources:

	#
	# Public VPC
	#
	PublicVpc: