Introduction
General Concepts
1.
Maths
1.1.
Tensors
1.1.1.
Rank manipulations
1.1.2.
Broadcasting
1.1.3.
Einsum
1.2.
Gradients and Backpropagation
1.2.1.
Implementing Backpropagation
1.2.2.
Implementing Vectoring Calcs
1.2.3.
Pytorch Gradients Calcs
1.3.
Probability
1.4.
Sampling
1.5.
Neural Networks
1.5.1.
Kolmogorov-Arnorld Networks
2.
Data Preprocessing
2.1.
Characters and texts
2.1.1.
Codifica One Hot
2.1.2.
Embeddings
2.1.2.1.
Embeddings class implementation
2.2.
Images
2.3.
Splitting Dataset
2.3.1.
Data Batch
2.3.2.
Data Chunks
2.4.
Flatten input data
2.5.
Tokenization
2.5.1.
TikToken
2.5.2.
Sentencepiece
3.
Neural Networks
3.1.
Generalizzazioni del comportamento
3.2.
Activation function
3.2.1.
tanh() class implementation
3.3.
Output Loss Functions
3.3.1.
mean squared error
3.3.2.
negative log likelihood loss
3.4.
MakeMore, a classifier NN
3.4.1.
Linear Layer implementation
3.4.2.
NN Implementation
3.5.
Convolution
3.5.1.
MNIST CNN implementation
3.5.2.
WaveNet, a convolutional NN
3.6.
Generative AI (GPT)
3.6.1.
Self Attention
3.6.1.1.
Head Attention class implementation
3.6.2.
Feed-forward
3.6.3.
Transformer
3.6.3.1.
GPT Implementation
3.7.
Stable Diffusion
4.
Optimizations
4.1.
Normalizations
4.1.1.
Batch Normalization
4.1.1.1.
Implementing BatchNorm Class
4.1.2.
Images Normalization
4.2.
Optimizer
4.2.1.
SGD with momentum and RMSProp
4.2.2.
ADAM
4.2.3.
Pytorch Optimizers and Schedulers Characteristics
4.3.
Learning Rate
4.3.1.
Learning Rate Finder
4.4.
Tuning neurons size in layer
4.5.
Embedding vector scaling
4.6.
Fixing initial loss
4.7.
Tuning activation function
4.8.
Initialization
4.8.1.
Considerations
4.9.
Optimize deep neural networks
5.
Regularization
5.1.
Weight Decay
5.2.
Dropout
6.
Diagnostic tools
6.1.
Loss function measures
6.2.
Detect dead neurons
6.3.
Forward pass statistics
6.4.
Backward pass statistics
6.5.
Weights distribution statistics
6.6.
Update to data-ratio statistics
Practical Design
7.
PyTorch implementation
7.1.
Training loop
7.2.
Learner Framework
7.2.1.
Callbacks examples
7.2.2.
Hooks
7.3.
Custom Loss Functions
Light
Rust
Coal
Navy
Ayu (default)
AI appunti
Sentencepiece