Tag Archives: autoencoder

Switch to Conjugate Gradient

Since I posted about autoencoder neural network on my blog (2 years ago), there are many people visit my Github for that code, Hooray !! Thank you very much everyone. And again I have a new update for that code. I switch to use Conjugate Gradient instead of generative back-propagation. Someone may think is that take 2 years to update it, NO but i’m too lazy.

While I posting autoencoder article, I realize that we need better than normal backpropagation. So I try to explore “What is the easy way to change and what algorithm to swiched to?”. Then I found that in scipy library, it contains optimization algorithm in scipy.optimize module. So you can change the optimization algorithm whatever you want that build within scipy.optimize module. This link is refer to Github page of old autoencoder but the Conjugate Gradient is on conjugate branch.

Why do I switch to Conjugate Gradient? After I study UFLDL lesson within advance optimization part. There is a phrase say that Conjugate Gradient is better than Gradient Descent (Classic Back-propagation). So I started to study about Conjugate Descent and other advance optimization, but I don’t understand them. Finally, I found out the workaround to improve my implementation by using scipy library. If anyone have any suggestion, please comment. Thanks.



Deep learning with Autoencoder

In past 10 years, machine learning is the most attractive subject, especially deep learning. Deep learning is the novel method to understand what ours brain think and percept. It begins in 1959, researchers found that cat’s brain can recognize picture by extract edges first, then lines, then surfaces, and objects. From this hypothesis, machine learning researchers adapt this idea to theirs algorithm and made its similar to real brain. At this time, there are several hardware that provide for deep learning algorithm, such as, Nervana Systems and Drive Px.

I am one of students that interested in this area, so I search for material to learn and practice about deep learning. And I found Machine Learning course from Coursera for beginner and deep learning website for expert. From a lot of articles about deep learning, I selected UFLDL tutorial to begin studying. First, I am not being expert in MATLAB that suggests in this tutorial. So I decide to use Theano python library which is my frequent programming language.

Autoencoder is the simple technique which I chose. Because it is easy to understand and can solve by simple neural network algorithm. Code of autoencoder that follows tutorial, neural network with regularization and sparsity penalty, is given.(https://github.com/chaiso-krit/autoencoder) Dataset that used in this code are 8×8 patch images and come from whiten images, provided by tutorial.

After running code for ~20 minutes, it will show what feature that autoencoder recognize, similar to following picture.

Autoencoder recognized feature