Simple-Implementation-of-ML-Algorithms Using Only Numpy

Most commonly used algorithms including Regressions, Graph Models, Deep Learning, Ensemble Learning, Reinforcement Learning, Optimization Algorithms, Swarm Intelligence, Heuristic Search, and other basic ML algorithms (see below list for more details).
For an easy understanding, most of the codes implements only minimal version of the algorithm.
The implementation uses only numpy.
Most algorithms are within 100 lines of codes.

Decision boundary visualization for implemented classifiers in decision_boundary_visualization.py
Optimization process visualization in optimization_visualization.py:
digit images from the generative models trained with MNIST dataset:
- Deep Convolutional Generative Adversarial Network (DC GAN) in generative_adversarial_network.py:
- Variational Autoencoders (VAE) in variational_autoencoder.py:
- Average image by Restricted Boltzmann Machine (RBM) restricted_boltzman_machine.py:

Linear Regression linear_regression.py
- Matrix solver
- SGD/Adam solver
- L1 regularization Lasso
- L2 regularization Ridge
Logistic Regression logistic_regression.py
- Multi-class prediction
Factorization Machines factorization_machines.py
- Regularization
- Classification/regression

Decision Tree decision_tree.py
- Classification/regression
- Different metrics
- Feature importances
- Sample weights
Random Forest random_forest.py
Adaboost adaboost.py
Gradient Boosting Decision Tree (GBDT) gradient_boosting_decision_tree.py
- Shrinkage
- Line search of multiplier
XGBoost xgboost.py
- XGBoost Regression Tree
- Shrinkage

Gradient Descent and variations optimization_visualization.py / multilayer_perceptron.py
- Stochastic Gradient Descent
- Gradient Descent with Momentum
- Nesterov Momentum
- AdaGrad
- RMSProp
- Adam
Monte Carlo method for global optimum
- Simulated annealing optimization_visualization.py

Naive Bayes naive_bayes.py
- Multinomial model
- Document tokenizer
Beyesian Network beyesian_net.py
- Conditional probability MLE
- Beyesian inference
Hidden Markov Model hidden_markov_model.py
- Fitting by Baum-Welch
- Prediction by Viterbi
Markov Random Field markov_random_field.py
- Image Segmentation
- Simulated Annealing for optimal Energy Function

k-Nearest Neighbors (kNN) k_nearest_neighbors.py
Learning Vector Quantization
Support Vector Machine (SVM) support_vector_machine.py
- Soft boundary
- SMO algorithm
- Different heuristics for selecting pairs in SMO

Evolutionary Algorithm (EA) evolutionary_algorithm.py
- Training a NN model by Genetic Algorithm
- Selection by Fitness
- Crossover approaches
- Mutation rate
Ant Colony Optimization (ACO) ant_colony.py
- Traveling Salesman Problem

Monte Carlo tree search monte_carlo_tree_search.py
- Upper Confidence Bound 1 applied to trees (UCT)
Minimax minimax.py
- Alpha-Beta Pruning

Temporal difference learning temporal_difference.py
- Tabular Q Learning
Deep Q-Network (DQN) deep_q_network.py
- CNN Target & Policy Net
- Epsilon-Greedy

Clustering (WIP)
- k-Means / dbscan / spectrum / hierachical
Dimension Reduction (WIP)
- SOM
- Principal Component Analysis / Linear Discriminant Analysis / MDS / t-SNE

Feel free to use the code. Please contact me if you have any question: xiecng [at] gmail.com

xiecong/Simple-Implementation-of-ML-Algorithms