Skip to main content

Ctrl+K

Omniverse

Notations

Machine Learning Notations

Influential Ideas and Papers

Generative Pre-trained Transformers
Low-Rank Adaptation Of Large Language Models
- Concept
- Implementation
Empirical Risk Minimization
- Concept: Empirical Risk Minimization
- Bayes Optimal Classifier
Is The Learning Problem Solvable?
- Concept: Learning Theory
Lloyd’s K-Means Clustering Algorithm
Naive Bayes
Mixture Models
- Concept
- Gaussian Mixture Models Implementation
Linear Regression
- Concept
- Implementation

Playbook

Training Dynamics And Tricks
Softmax Preserves Order, Is Translation Invariant But Not Invariant Under Scaling.
How to Inspect Function and Class Signatures in Python?

Probability Theory

Chapter 1. Mathematical Preliminaries
Chapter 2. Probability
Chapter 3. Discrete Random Variables
Chapter 4. Continuous Random Variables
Chapter 5. Joint Distributions
Chapter 6. Sample Statistics
Chapter 8. Estimation Theory
- Maximum Likelihood Estimation
  - Concept

Operations

Distributed Systems
Profiling
The Lifecycle of an AIOps System

Software Engineering

Config, State, Metadata Management
Design Patterns
Python
Concurrency, Parallelism and Asynchronous Programming

Computer Science

Type Theory, A Very Rudimentary Introduction

Data Structures and Algorithms

Complexity Analysis
- Master Theorem
List/Array
- Concept
- Questions
  - Two Sum
Hash Map
- Concept
- Questions
  - Two Sum
  - Group Anagrams
Two Pointers And Sliding Window
Stack
- Concept
- Questions
Queue
- Concept
- Double Ended Queue
  - Easy - Hot Potatoes
  - Palindrome Checker
Linear Search
- Concept
Binary Search
- Concept
- Koko Eating Bananas

Linear Algebra

Preliminaries
- Fields
- Systems of Linear Equations
Vectors

References, Resources and Roadmap

Bibliography
IEEE (Style) Citations

Repository
Open issue

.md

Training Dynamics And Tricks

Contents

Table of Contents

Training Dynamics And Tricks#

Table of Contents

Table of Contents #

How to Calculate the Number of FLOPs in Transformer Based Models?
Why Does Cosine Annealing With Warmup Stabilize Training?
How To Fine-Tune Decoder-Only Models For Sequence Classification Using Last Token Pooling?
How To Fine-Tune Decoder-Only Models For Sequence Classification With Cross-Attention?
How To Do Teacher-Student Knowledge Distillation?

previous

Implementation

next

How to Calculate the Number of FLOPs in Transformer Based Models?

Contents

Table of Contents

By Gao Hongnan

© Copyright 2024.

Connect with me!

Share this page!