timothelaborie

timothelaborie / fix_lmhead.py

Last active December 4, 2025 20:05

	import torch
	from torch.nn import Module
	from collections import OrderedDict
	from typing import Mapping, Any, List, NamedTuple

	from unsloth import tokenizer_utils
	def do_nothing(args, *kwargs):
	pass
	tokenizer_utils.fix_untrained_tokens = do_nothing

timothelaborie / gist:fe284d6ccb08b50bb0e9c34e28c622b6

Created June 26, 2024 18:34

adam_mini attempt for unsloth/Qwen2-7B-bnb-4bit

	import torch
	from torch.optim.optimizer import Optimizer
	import math
	import torch.distributed as dist
	from torch.optim.optimizer import _dispatch_sqrt

	device = 'cuda' if torch.cuda.is_available() else 'cpu'


	class Adam_mini(Optimizer):