Ronaldo ronaldosaheki

How to install Hadoop on your local Kubernetes cluster

Okey this is not the easiest way of running Hadoop on your local computer and probably you should instead just install it locally.

However if you really insist doing this here's how:

Install kubectl, minikube and Docker if you don't already have it. I recommend using package-manager like Chocolatey. Minikube should install with VirtualBox as default driver which I recommend. When starting minikube we should increase its memory limit since our Hadoop node's pods need at least 2GB: minikube --memory 4096 --cpus 2 start (minikube's default is 1GB). NOTE: actually the Hadoop cluster by default uses about 10GB in memory limits and about 3GB running memory. From what I looked my k8s will overprovision to 300% of its capacity limits but use far less.
Install helm. Then run helm init.
Now you

Bash script which will:

	# Using Debian GNU/Linux 10 (buster)

	# Create local SSD file system and mount
	sudo mkfs.ext4 -F /dev/nvme0n1
	sudo mkdir -p /mnt/disks/nvme0n1
	sudo mount /dev/nvme0n1 /mnt/disks/nvme0n1
	sudo chmod a+w /mnt/disks/nvme0n1
	cd /mnt/disks/nvme0n1

	# Update

	PLAN_OPTIONS ?=
	APPLY_OPTIONS ?=
	EXCLUDE ?=
	INCLUDE ?=

	# For Terraform 0.12 (using `-no-color` to avoid dealing with terminal color)
	define PLAN_OPTIONS_EXCLUDE
	$(shell terraform show -no-color current.plan \| perl -nle 'if (/\s# (.*?)\s/) {print $$1}' \| grep -E -v '$(1)' \| sed -e 's/^/-target="/g' -e 's/$$/"/g' \| xargs)
	endef

	#!/usr/bin/env bash

	query_string=$(cat << EndOfMessage
	fields @timestamp, @logStream, headers.X-Amzn-Trace-Id, @transId, @message
	\| parse @message /(transactionId:[ ]?)(?<@transId>[a-zA-Z0-9]+)/
	\| filter @transId = a4c475516be5445a87fbb81bb7a4b365
	EndOfMessage
	) \
	&& \
	query_id=`aws logs start-query --log-group-name /aws/lambda/console_log \

	#!/usr/bin/env python
	from http import HTTPStatus
	from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
	import base64
	import json
	import logging
	import os
	import os.path
	import ssl

	#!/bin/bash

	if git rev-parse --verify HEAD &> /dev/null ; then
	against=HEAD
	else
	against=4b825dc642cb6eb9a060e54bf8d69288fbee4904
	fi

	files=$(
	git diff-index --cached --name-only $against \|

	#!/bin/bash

	FILE="client_key"
	OUTPUT_FILE="client_key-cert.pub"

	echo "Generating new key..."
	ssh-keygen -t rsa -f $FILE -N ''

	echo "Getting client key signed by Vault..."
	cat $FILE.pub \| vault write ssh-client-signer/sign/clientrole public_key=- \| grep "signed_key" \| cut -f 2 > $OUTPUT_FILE