Skip to main content

Ctrl+K

Omniverse.

Notations

Machine Learning Notations

Influential Ideas and Papers

Generative Pre-trained Transformers
Low-Rank Adaptation Of Large Language Models
- Concept
- Implementation
Empirical Risk Minimization
- Concept: Empirical Risk Minimization
- Bayes Optimal Classifier
Is The Learning Problem Solvable?
- Concept: Learning Theory
K-Means
Naive Bayes
Mixture Models
- Concept
- Gaussian Mixture Models Implementation

Playbook

Training Dynamics And Tricks
Softmax Preserves Order, Is Translation Invariant But Not Invariant Under Scaling.
How to Inspect Function and Class Signatures in Python?

Probability Theory

Chapter 1. Mathematical Preliminaries
Chapter 2. Probability
Chapter 3. Discrete Random Variables
Chapter 4. Continuous Random Variables
Chapter 5. Joint Distributions
Chapter 6. Sample Statistics
Chapter 8. Estimation Theory
- Maximum Likelihood Estimation
  - Concept

Operations

Distributed Systems
Profiling
The Lifecycle of an AIOps System

Software Engineering

Config, State, Metadata Management
Design Patterns
Python
- Decorator
- Pydantic Is All You Need - Jason Liu
Concurrency, Parallelism and Asynchronous Programming
- A Rudimentary Introduction to Generator and Yield in Python

Computer Science

Type Theory, A Very Rudimentary Introduction

Data Structures and Algorithms

Complexity Analysis
- Master Theorem
Stack
- Concept
Linear Search
- Concept
Binary Search
- Concept
- Koko Eating Bananas

Linear Algebra

Preliminaries
- Fields
- Systems of Linear Equations
Vectors

References, Resources and Roadmap

Bibliography
IEEE (Style) Citations

API Reference

API Reference

Repository
Open issue

.md

Training Dynamics And Tricks

Contents

Table of Contents

Training Dynamics And Tricks#

Table of Contents

Table of Contents #

How to Calculate the Number of FLOPs in Transformer Based Models?
Why Does Cosine Annealing With Warmup Stabilize Training?
How To Fine-Tune Decoder-Only Models For Sequence Classification Using Last Token Pooling?
How To Fine-Tune Decoder-Only Models For Sequence Classification With Cross-Attention?
How To Do Teacher-Student Knowledge Distillation?

previous

Gaussian Mixture Models Implementation

next

How to Calculate the Number of FLOPs in Transformer Based Models?

Contents

Table of Contents

By Gao Hongnan

© Copyright 2024.