Low-Rank Adaptation Of Large Language Models

This blog post provides a brief overview on the concept of low-rank adaptation of large language models (LLMs) and its implementation in PyTorch.

• Concept
• Implementation

Citations

• [1] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” arXiv preprint arXiv:2106.09685, submitted Jun. 17, 2021, revised Oct. 16, 2021. [Online]. Available: https://arxiv.org/abs/2106.09685

• [2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. “Attention is all you need”. In Advances in Neural Information Processing Systems, pp. 5998–6008, 2017.

References

• LoRA Fine-Tuning Tutorial - PyTorch

• Low-Rank Approximation - Wikipedia

• Rank (Linear Algebra) - Wikipedia

• Big Ideas in Applied Math: Low-Rank Matrices - Ethan Epperly

• Matrix Factorization Tutorial - PyProximal

• Matrix Decomposition Series: Low-Rank Matrix Factorization - Renda Zhang

• Fine-Tuning Using LoRA - GitHub

• LoRA for Sequence Classification with Roberta, Llama, and Mistral - Hugging Face Blog

• Implementing LoRA from Scratch for Fine-Tuning LLMs - Daily Dose of DS

• LoRA for Sequence Classification with Roberta, Llama, and Mistral - Mehdi Ir

• LoRA on MNIST - Sunil Kumar

• Efficient Fine-Tuning with LoRA - Databricks

• LoRA Fine-Tuning Tutorial - PyTorch

• LoRA: Low-Rank Adaptation of Large Language Models - Arxiv

• Practical Tips for Fine-Tuning LLMs - Sebastian Raschka

• What LoRA Alpha Actually Does (In Theory) - CivitAI

• Fine-Tuning with LoRA on 8-bit - Serp AI

• Eternal Question: What Rank R and Alpha to Use in LoRA - Reddit

• LLMs from Scratch - Rasbt

• Initialize perceptron weights with zero