Jason Tan jjasont

The latest and faster Spark write out can be done with S3A/magic committer

Ensure the usage of s3a prefix when reading and writing from/to S3 for better performance

spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
spark.hadoop.fs.s3a.committer.magic.enabled: "true"
spark.hadoop.fs.s3a.committer.name: "magic"
spark.sql.sources.commitProtocolClass: org.apache.spark.internal.io.cloud.PathOutputCommitProtocol

ISO Week-Year Calculation for AWS Quicksight (as of 30 July 2021)

This gists is an attempt to 'hack' or a workaround for making the starting week as Monday (ISO 8601), retrieving an ISO week number and the corresponding year of ISO week number As there's no such function per 30 July 2021 to perform any extraction of week number in Quicksight, the following are supposedly resolve such cases.

Start week is Monday (Ref #2)

Define a column with the following formula, for this usage it will be named week_start_monday

There's an edge case handling for when the date of interest is Sunday. When the day is Sunday, it should belong to the previous week instead of the next one (since the week runs from Monday-Sunday, instead of Sunday-Saturday)

Superset API Endpoint List

Superset is based on Flask App Builder (FAB). Hence, certain model can be manipulated via Flask REST API. Further information about the API can be found in the official Flask App Builder documentation here

This noted was aspired from the Issue #4708

API Login

Login to API required before usages of any the API endpoint at /api/v1/security/login.

Setup

$ pip install click requests

Usage

$ python query_export.py --redash-url "https://app.redash.io/" --api-key ""

	import mlflow

	# There are two ways to create parent/child runs in MLflow.

	# (1) The most common way is to use the fluent
	# mlflow.start_run API, passing nested=True:
	with mlflow.start_run():
	num_trials = 10
	mlflow.log_param("num_trials", num_trials)
	best_loss = 1e100

	# Detect Lock
	select a.txn_owner, a.txn_db, a.xid, a.pid, a.txn_start, a.lock_mode, a.relation as table_id,nvl(trim(c."name"),d.relname) as tablename, a.granted,b.pid as blocking_pid ,datediff(s,a.txn_start,getdate())/86400\|\|' days '\|\|datediff(s,a.txn_start,getdate())%86400/3600\|\|' hrs '\|\|datediff(s,a.txn_start,getdate())%3600/60\|\|' mins '\|\|datediff(s,a.txn_start,getdate())%60\|\|' secs' as txn_duration
	from svv_transactions a
	left join (select pid,relation,granted from pg_locks group by 1,2,3) b
	on a.relation=b.relation and a.granted='f' and b.granted='t'
	left join (select * from stv_tbl_perm where slice=0) c
	on a.relation=c.id
	left join pg_class d on a.relation=d.oid
	where a.relation is not null AND a.txn_owner='<db_username>';

	import re, fileinput

	print('SET FOREIGN_KEY_CHECKS=0;')

	def main():
	for line in fileinput.input():
	process = False
	for nope in ('BEGIN TRANSACTION','COMMIT',
	'sqlite_sequence','CREATE UNIQUE INDEX'):
	if nope in line: break

	# To dump a database into a custom-format archive file
	pg_dump -h <origin_postgres_host> \
	-p <origin_port> \
	-d <origin_postgres_db> \
	-U <username_origin_postgres_rds_db> \
	-W -Fc > <pg_dump_file>.psql

	# To restore a database into a custom-format archive file with no owner:
	pg_restore -h <target_postgres_host> \
	-U <username_target_postgres_db> \

	"""Direct Google Sheets Manipulation trial."""
	from predictorcls import Predictor
	from oauth2client.service_account import ServiceAccountCredentials
	import gspread
	# import time
	# import random

	scope = ['https://spreadsheets.google.com/feeds',
	'https://www.googleapis.com/auth/drive']