Machine Learning Notes

Information_Theory

Machine Learning Notes

Home
📚 Learning Plan
📊 Mathematical Foundations
📊 Mathematical Foundations
🔗 Engineering and Data Structures
🔗 Engineering and Data Structures
- Overview
- Data Structures
  Data Structures
  - Arrays
    Arrays
    
    Arrays Overview
    
    Dynamic Arrays
    
    Array Problems
  - Linked Lists
    Linked Lists
    
    Linked Lists Overview
    
    Implementation
    
    Problems
  - Stacks
    Stacks
    
    Stacks Overview
    
    Implementation
    
    Problems
  - Queues
    Queues
    
    Queues Overview
    
    Implementation
    
    Problems
  - Hash Tables
    Hash Tables
    
    Hash Tables Overview
    
    Hash Functions & Collisions
    
    Python Dictionaries
    
    Python Dictionary Operations
    
    Sets and Python Sets Overview
    
    Python Set Operations
    
    String Operations
    
    Hash Table Problems
  - Recursion
    Recursion
    
    Recursion Fundamentals
    
    Recursive Algorithms
    
    Recursion vs Iteration
    
    Common Recursive Patterns
- Algorithms
  Algorithms
  - Search Algorithms
    Search Algorithms
    
    Search Overview
    
    Binary Search
    
    Binary Search Variations
    
    Search Problems
  - Sorting Algorithms
    Sorting Algorithms
    
    Sorting Overview
    
    Sorting Problems
- Algorithmic Patterns
  Algorithmic Patterns
  - Two Pointers
    Two Pointers
    
    Two Pointers Overview
    
    Opposite Direction
    
    Same Direction
    
    Two Pointers Problems
  - Sliding Window
    Sliding Window
    
    Sliding Window Overview
    
    Fixed Size Window
    
    Variable Size Window
    
    Sliding Window Problems
- Problem Solving
  Problem Solving
  - Set & Dictionary Problems
    Set & Dictionary Problems
    
    Array Intersection
    
    Non-Repeating Elements
    
    Unique Elements
    
    Anagram Pairs
  - String Problems
    String Problems
    
    Unique Strings
- Resources
  Resources
📈 Probability & Statistics
📈 Probability & Statistics
- Probability & Markov
  Probability & Markov
🤖 ML Fundamentals
🤖 ML Fundamentals
- Overview
- Feature Engineering
  Feature Engineering
- Model Evaluation
  Model Evaluation
- Regularization
  Regularization
- Classical Algorithms
  Classical Algorithms
- Unsupervised Learning
  Unsupervised Learning
  - K-Nearest Neighbors
  - K-Means Clustering
💬 Language Models & NLP
💬 Language Models & NLP
- N-gram Language Modeling
- Information Theory Information Theory
  Table of contents
  - title: Introduction to Information Theory
    
    Introduction
    
    Knowledge Points
🧠 Neural Networks & Deep Learning
🧠 Neural Networks & Deep Learning
- Overview
- Perceptron Algorithm
⚡ Attention & Transformers
⚡ Attention & Transformers
- Overview
- Attention Fundamentals
  Attention Fundamentals
  - Overview
  - Mathematics
- Self-Attention
  Self-Attention
  - Overview
- Multi-Head Attention
  Multi-Head Attention
  - Overview
- Positional Encoding
  Positional Encoding
  - Overview
- Transformer Architecture
  Transformer Architecture
  - Overview
- Transformer Variants
  Transformer Variants
  - BERT & GPT
🚀 Projects & Integration
🚀 Projects & Integration
- Integration and Project

Information Theory

title: Introduction to Information Theory¶

Introduction¶

Information theory quantifies uncertainty and information content. Concepts such as entropy and cross-entropy are deeply entwined with language modeling and loss functions used to train neural networks.

Knowledge Points¶

Entropy
Cross-Entropy
Kullback-Leibler (KL) Divergence
Why information theory matters for language modeling
Manual entropy calculation exercise
Implementing cross-entropy loss