Overview#

A custom implementation of an Echo State Network (ESN), a type of recurrent neural network designed for temporal pattern recognition and time series forecasting. Built from scratch using Python, NumPy, and Matplotlib.

Key Achievements#

• Implemented recurrent neural network for time series prediction, achieving 0.142 MSE in 2Sine 1-Step model and 7.626 MSE in Lorenz 1-Step model, indicating significant generalization ability and data variation respectively
• Conducted systematic hyperparameter tuning using cross-validation and ridge regression to minimize prediction error
• Analyzed how error compounds overtime in multi-step predictions
• Demonstrated impact of model complexity on overfitting using Lorenz data

Implementation#

• Used batch training with input weights, recurrent weights, and sigmoid activation
• Applied ridge regression to determine optimized output weights
• Performed cross-validation to optimize hyperparameters
• Evaluated results using Mean-Square Error (MSE)

Conclusions#

• The Lorenz model demonstrates that increased model complexity leads to overfitting and a failure to generalize, as its data points had higher varience and its function was more complex compared to the 2Sine dataset. Additionally, the complexity of the Lorenz function causes a smaller regularization parameter, causing outliers to significantly affect weight optimizations.
• K-Step Ahead prediction produces less accurate results the more steps ahead you attempt to predict because error compounds overtime and predictions are fed back into the model.

Technologies#

Python, NumPy, Matplotlib, Ridge Regression, Neural Networks

Related Skills#

• Python
• Machine Learning
• NumPy
• Matplotlib