Omniverse

Notations

• Machine Learning Notations

Influential Ideas and Papers

• Generative Pre-trained Transformers
  • Notations
  • The Concept of Generative Pre-trained Transformers (GPT)
  • The Implementation of Generative Pre-trained Transformers (GPT)
  • Training a Mini-GPT to Learn Two-Digit Addition
• Low-Rank Adaptation Of Large Language Models
  • Concept
  • Implementation
• Empirical Risk Minimization
  • Concept: Empirical Risk Minimization
  • Bayes Optimal Classifier
• Is The Learning Problem Solvable?
  • Concept: Learning Theory
• Lloyd’s K-Means Clustering Algorithm
  • Concept: K-Means Clustering
  • Implementation: K-Means (Lloyd)
  • Application: Image Compression and Segmentation
  • Conceptual Questions
• Naive Bayes
  • Concept
  • Naives Bayes Implementation
  • Naive Bayes Application: Penguins
  • Naive Bayes Application (MNIST)
• Mixture Models
  • Concept
  • Gaussian Mixture Models Implementation
• Linear Regression
  • Concept
  • Implementation

Playbook

• Training Dynamics And Tricks
  • How to Calculate the Number of FLOPs in Transformer Based Models?
  • Why Does Cosine Annealing With Warmup Stabilize Training?
  • How To Fine-Tune Decoder-Only Models For Sequence Classification Using Last Token Pooling?
  • How To Fine-Tune Decoder-Only Models For Sequence Classification With Cross-Attention?
  • How To Do Teacher-Student Knowledge Distillation?
• Softmax Preserves Order, Is Translation Invariant But Not Invariant Under Scaling.
• How to Inspect Function and Class Signatures in Python?

Probability Theory

• Chapter 1. Mathematical Preliminaries
  • Permutations and Combinations
  • Calculus
  • Contour Maps
  • Exercises
• Chapter 2. Probability
  • Probability Space
  • Probability Axioms
  • Conditional Probability
  • Independence
  • Baye’s Theorem and the Law of Total Probability
  • Summary
• Chapter 3. Discrete Random Variables
  • Random Variables
  • Discrete Random Variables
  • Probability Mass Function
  • Cumulative Distribution Function
  • Expectation
  • Moments and Variance
  • Discrete Uniform Distribution
    • Concept
    • Application
  • Bernoulli Distribution
    • Concept
    • Application
  • Independent and Identically Distributed (IID)
  • Binomial Distribution
    • Concept
    • Implementation
    • Real World Examples
  • Geometric Distribution
    • Concept
  • Poisson Distribution
    • Concept
    • Implementation
  • Important
  • Exercises
• Chapter 4. Continuous Random Variables
  • From Discrete to Continuous
  • Continuous Random Variables
  • Probability Density Function
  • Expectation
  • Moments and Variance
  • Cumulative Distribution Function
  • Mean, Median and Mode
  • Continuous Uniform Distribution
  • Exponential Distribution
  • Gaussian Distribution
  • Skewness and Kurtosis
  • Convolution and Sum of Random Variables
  • Functions of Random Variables
• Chapter 5. Joint Distributions
  • From Single Variable to Joint Distributions
  • Joint PMF and PDF
  • Joint Expectation and Correlation
  • Conditional PMF and PDF
  • Conditional Expectation and Variance
  • Sum of Random Variables
  • Random Vectors
  • Multivariate Gaussian Distribution
• Chapter 6. Sample Statistics
  • Moment Generating and Characteristic Functions
  • Probability Inequalities
  • Law of Large Numbers
• Chapter 8. Estimation Theory
  • Maximum Likelihood Estimation

Operations

• Distributed Systems
  • Notations
  • Basics Of Distributed Data Parallelism
  • How to Setup SLURM and ParallelCluster in AWS
  • Ablations
• Profiling
  • Synchronize CUDA To Time CUDA Operations
  • Profiling Code With Timeit
  • PyTorch’s Event And Profiler
  • Profile GPT Small Time And Memory
  • CUDA Memory Allocations
• The Lifecycle of an AIOps System
  • Stage 1. Problem Formulation
  • Stage 2. Project Scoping And Framing The Problem
  • Stage 3. Data Pipeline (Data Engineering and DataOps)
    • Stage 3.1. Data Source and Formats
    • Stage 3.2. Data Model and Storage
    • Stage 3.3. Extract, Transform, Load (ETL)
  • Stage 4. Data Extraction (MLOps), Data Analysis (Data Science), Data Preparation (Data Science)
  • Stage 5. Model Development and Training (MLOps)
    • Stage 5.1. Model Selection
    • Stage 5.2. Metric Selection
    • Stage 5.3. Experiment Tracking And Versioning
    • Stage 5.4. Model Testing
  • Stage 6. Model Evaluation (MLOps)
  • Stage 7. Model Validation, Registry and Pushing Model to Production (MLOps)
  • Stage 8. Model Serving (MLOps)
  • Stage 9. Model Monitoring (MLOps)
  • Stage 10. Continuous Integration, Deployment, Learning and Training (DevOps, DataOps, MLOps)

Software Engineering

• Config, State, Metadata Management
  • Configuration Management
  • Pydantic And Hydra
  • State And Metadata Management
• Design Patterns
  • Dependency Inversion Principle
  • Named Constructor
  • Strategy
  • Registry
  • Context Object Pattern (God Object)
  • Factory Method
  • Singleton
• Python
  • Init vs New
  • Global Interpreter Lock (GIL)
  • The Iterator Protocol
  • Decorator
  • Generators Over Lists For Memory Efficiency
  • Pydantic Is All You Need - Jason Liu
  • Do Not Use Mutable Default Arguments
  • Set Over List For Frequent Membership Tests
  • Late Binding Closures
  • Is vs Equality
• Concurrency, Parallelism and Asynchronous Programming
  • Overview Of Concurrency, Parallelism, and Asynchronous Execution
  • Thread Safety
  • A Rudimentary Introduction to Generator and Yield in Python

Computer Science

• Type Theory, A Very Rudimentary Introduction
  • Subtypes
  • Type Safety
  • Subsumption
  • Generics and Type Variables
  • Bound and Constraint in Generics and Type Variables
  • Invariance, Covariance and Contravariance
  • Function Overloading
  • Sentinel Types

Data Structures and Algorithms

• Complexity Analysis
  • Master Theorem
• List/Array
  • Concept
  • Questions
    • Two Sum
• Hash Map
  • Concept
  • Questions
    • Two Sum
    • Group Anagrams
• Two Pointers And Sliding Window
  • Two Pointers
  • Sliding Window
  • Questions
    • Two Pointers
      • Remove Duplicates from Sorted Array
      • Two Sum II - Input Array Is Sorted
    • Sliding Window
      • Find All Anagrams in a String
• Stack
  • Concept
  • Questions
    • Valid Parentheses
    • Min Stack
    • Implement Queue using Stacks
    • Reverse String
• Queue
  • Concept
  • Double Ended Queue
• Linear Search
  • Concept
• Binary Search
  • Concept
  • Koko Eating Bananas

Linear Algebra

• Preliminaries
  • Fields
  • Systems of Linear Equations
• Vectors
  • Vector and Its Definition
  • Vector and Its Operations
  • Vector Norm and Distance
  • A First Look at Vector Products

References, Resources and Roadmap

• Bibliography
• IEEE (Style) Citations