zzjjzzgggg / ddp_notes.md

Created June 7, 2024 06:23 — forked from TengdaHan/ddp_notes.md

Multi-node-training on slurm with PyTorch

What's this?

A simple note for how to start multi-node-training on slurm scheduler with PyTorch.
Useful especially when scheduler is too busy that you cannot get multiple GPUs allocated, or you need more than 4 GPUs for a single job.
Requirement: Have to use PyTorch DistributedDataParallel(DDP) for this purpose.
Warning: might need to re-factor your own code.
Warning: might be secretly condemned by your colleagues because using too many GPUs.

zzjjzzgggg / swind

Last active February 11, 2018 15:51 — forked from andrewmcdonough/swind

A bash script to move the focused window in X to different parts of the screen. Most useful when you assign shortcut keys to the various options, e.g. assign '<Super>Up' to run 'swind up'

	#!/bin/bash

	# Written by Andrew McDonough
	# Prerequisites: xdotool must be installed and in your path (http://tinyurl.com/xdotool)
	# A simple bash script that uses xdotool to move the window that is currently in focus to different parts of the screen.
	# Particularly useful for reading web pages with flexible layouts on wide monitors.
	# Assign the various options to keyboard shortcuts e.g. '<Super>Left' assigned to 'swind left'
	# See http://tinyurl.com/ubuntukeys for help with assigning keyboard shortcuts.

	# sudo apt install xdotool

zzjjzzgggg / custom_iterator.cpp

Last active October 16, 2017 19:56 — forked from jeetsukumaran/custom_iterator.cpp

Sample C++/STL custom iterator

	// Sample custom iterator.
	// By perfectly.insane (http://www.dreamincode.net/forums/index.php?showuser=76558)
	// From: http://www.dreamincode.net/forums/index.php?showtopic=58468

	#include <iostream>
	#include <vector>
	#include <algorithm>
	#include <iterator>
	#include <cassert>

zzjjzzgggg / how-to-install-latest-gcc-on-ubuntu-lts.sh

Last active June 26, 2017 09:48 — forked from application2000/how-to-install-latest-gcc-on-ubuntu-lts.txt

How to install latest gcc on Ubuntu LTS 16.04, Linux Mint 18.1

	# These commands are based on a askubuntu answer http://askubuntu.com/a/581497
	# To install gcc-6 (gcc-6.1.1), I had to do more stuff as shown below.
	# USE THOSE COMMANDS AT YOUR OWN RISK. I SHALL NOT BE RESPONSIBLE FOR ANYTHING.
	# ABSOLUTELY NO WARRANTY.

	# If you are still reading let's carry on with the code.

	sudo apt-get update
	sudo apt-get install build-essential software-properties-common
	sudo add-apt-repository ppa:ubuntu-toolchain-r/test

zzjjzzgggg zzjjzzgggg

Multi-node-training on slurm with PyTorch

What's this?