Ignacio Franco ifranco14

Running KoboldAI in 8-bit mode

tl;dr use Linux, install bitsandbytes (either globally or in KAI's conda env, add load_in_8bit=True, device_map="auto" in model pipeline creation calls)

Many people are unable to load models due to their GPU's limited VRAM. These models contain billions of parameters (model weights and biases), each of which is a 32 (or 16) bit float. Thanks to the hard work of some researchers [1], it's possible to run these models using 8-bit numbers, which halves the required amount of VRAM compared to running in half-precision. E.g. if a model requires 16GB of VRAM, running with 8-bit inference only requires 8GB.

This guide was written for KoboldAI 1.19.1, and tested with Ubuntu 20.04. These instructions are based on work by Gmin in KoboldAI's Discord server, and Huggingface's efficient LM inference guide.

Requirements

How to calculate your hourly rate as a freelancer?

Many people struggle with this question. Some just try to make as much as a full-time employee makes (and ignore that they won't be able to bill as many days). Others follow tips on startup related websites that suggest to ask for 20% to 50% more than an salary would yield (and ignore the additional risk and expenses they have).

Below you will find some numbers to help you calculate how high your hourly or daily rate should be.

Your yearly income should be higher than an average salary

You take more risk than full time employees, phases without income are likely

	<?
	# MIT license, do whatever you want with it
	#
	# This is my invoice.php page which I use to make invoices that customers want,
	# with their address on it and which are easily printable. I love Stripe but
	# their invoices and receipts were too wild for my customers on Remote OK
	#

	require_once(__DIR__.'/../vendor/autoload.php');