Skip to content

Instantly share code, notes, and snippets.

@huang06
huang06 / split_dataframe_into_partitions.py
Created September 7, 2022 09:41
split pandas dataframe into parquet files
# https://stackoverflow.com/questions/50824943/how-to-efficiently-split-a-large-dataframe-into-many-parquet-files
# https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_to_dataset.html
import pandas as pd
import pyarrow.parquet as pq
import pyarrow as pa
df = pd.read_csv("CSV_LOCATION")
table = pa.Table.from_pandas(df)
pq.write_to_dataset(
@huang06
huang06 / ml-package-deps.md
Last active November 3, 2022 12:40
This gist lists the dependencies of the popular Machine Learning libraries.
package name package version python numpy keras protobuf scipy comment url
Tensorflow1 1.15.0 27,35,36,37 >=1.16.0,<2.0 - >= 3.6.1 - https://github.com/tensorflow/tensorflow/blob/v1.15.0/tensorflow/tools/pip_package/setup.py
1.15.2 35,36,37 >=1.16.0,<2.0 - >= 3.6.1 - https://github.com/tensorflow/tensorflow/blob/v1.15.2/tensorflow/tools/pip_package/setup.py
1.15.3 35,36,37 >=1.16.0,<2.0 - >= 3.6.1 - https://github.com/tensorflow/tensorflow/blob/v1.15.3/tensorflow/tools/pip_package/setup.py
1.15.4 35,36,37 >=1.16.0,<1.19.0 - >= 3.6.1 - https://github.com/tensorflow/tensorflow/blob/v1.15.4/tensorflow/tools/pip_package/setup.py
1.15.5 36,37 >=1.16.0,<1.19.0 - >= 3.6.1 - https://github.com/tensorflow/tensorflow/blob/v1.15.5/tensorflow/tools/pip_package/setup.py
Tensorflow2 2.6.0 36,37,38,39 ~= 1.19.2 ~= 2.6 >= 3.9.2 - https://github.com/tensorflow/tensorflow/blob/v2.6.0/tensorflow/tools/pip_package/setup.py
2.6.1 36,37
@huang06
huang06 / gitconfig
Last active August 4, 2022 14:04
My .gitconfig
[user]
email = my_email
name = my_username
[core]
ignorecase = false
editor = vim
[user]
useConfigOnly = false
[pull]
ff = only
@huang06
huang06 / bash_profile
Last active August 1, 2022 06:03
My bash settings on Apple Silicon
echo "This is coming from .bash_profile"
if [ -f /Users/tom/.bashrc ]; then
. /Users/tom/.bashrc
fi
@huang06
huang06 / vimrc
Last active January 28, 2024 05:05
My .vimrc
" =======================
" Plugin Configuration
" =======================
call plug#begin()
" The default plugin directory will be as follows:
" - Vim (Linux/macOS): '~/.vim/plugged'
" - Vim (Windows): '~/vimfiles/plugged'
" - Neovim (Linux/macOS/Windows): stdpath('data') . '/plugged'
" You can specify a custom plugin directory by passing it as the argument
" - e.g. `call plug#begin('~/.vim/plugged')`
@huang06
huang06 / ingress-helloworld-example.yaml
Created February 20, 2021 10:25
Ingress Helloworld Example
---
apiVersion: v1
kind: Namespace
metadata:
name: helloworld
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: helloworld
@huang06
huang06 / kubernetes-ingress-example.yaml
Last active February 20, 2021 10:08 — forked from alexellis/kubernetes-ingress-example.yaml
kubernetes-ingress-example.yaml
# Forward the IngressController
# kubectl port-forward ingress-nginx-controller 8080:80
# Access our service via Ingress:
# curl localhost:8080 -d Sysdig -H "Host: openfaas-figlet.local"
---
apiVersion: v1
kind: Namespace
metadata:
name: openfaas-figlet