Skip to content

Instantly share code, notes, and snippets.

@LiutongZhou

LiutongZhou/memory_efficient_training.md

Last active July 11, 2023 15:37
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save LiutongZhou/9db08ddd9fa6f52553449fa8b72e19d7 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save LiutongZhou/9db08ddd9fa6f52553449fa8b72e19d7 to your computer and use it in GitHub Desktop.
Download ZIP
Memory Efficient Training of LLMs
Raw
memory_efficient_training.md

Memory Efficient Training

Optimizers

Short Name Full Name URL
CAME Confidence-guided Adaptive Memory Efficient Optimization (ACL23 outstanding paper award) https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/CAME/came.py
LOMO LOw-Memory Optimization https://github.com/OpenLMLab/LOMO/blob/main/src/lomo.py

Low Rank Decomposition

LORA (Low Rank Adaption): https://github.com/microsoft/LoRA or https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora.py

Adapters

Quantization

QLORA: https://github.com/artidoro/qlora/blob/main/qlora.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment