Deep Learning
Rohit Malshe, Chemical Engineer, Programmer, Author, Thinker, Engineer at Intel Corporation
Written Feb 10
I have borrowed a lot of slides from the famous talk by Brandon Rohrer, and enhanced them so as to elaborate further and add more knowledge on the top of it. You should totally check out that famous talk on YouTube.
What is deep learning?
We are all surrounded by data. An enormous amount of it. It is not only stored on big computational servers all over the world, on the computers in our offices, and cell phones in our pockets and constantly being generated, but also being transferred at a high volume in the air. This picture in the below only signifies how you can imagine about it.
Deep learning is good at identifying patterns. Imagine for example about the image in the below. When you look at it, you can see there are bright spots in the image, whereas a lot of spots are dark. If I was to feed this image to a deep learning image recognition kind of program it would be able to identify where the bright spots are.
This schematic shows the evolution of this science. It started all the way back in 1943, and continued progress happened on it over the years. In those decades, we were significantly limited by the computational power. It is only in the recent years, that we have massive computational capabilities available and we have some amazing programmers in the world who are developing this field.
To give you an analogy between brain and deep learning, think of an owl. If your brain is like an owl, then deep learning is just like a fighter jet. An owl is far more complex creature and it can do a lot of things, but a fighter jet is complex too, and it can do a significant number of things, but they are very narrow in the scope. Although they can be very large in magnitude.
Well then, why not think about the learning process first?
When someone first made the first plane fly, it took them an enormous amounts of efforts. Humans have continued to train the planes so much over the years that now we have some really amazing planes that can carry hundreds of passengers across the borders in very little time.
So in essence, we have learned, and we have made the robots learn.
The basic premises of deep learning start from the brain. It wants to emulate in some sense exactly what the human brain does. Human brain comprises of neural networks.
Neural networks looks somewhat like the picture shown in the below. They are highly dense, and interconnected.
Here is a schematic of a neuron, and a pictorial description of it. The units of a neuron are axon, soma and dandrites. There are more, but lets stick to these basic ones for now.
The soma collect information from the dendrites and passes it over to an axon.
The dendrites can connect to the axons of next layer neurons with a process called synapse. This shows how synapse happens. Its more like an electrical activation.
To think of this pictorially, one can draw a schematic like the one shown in the below.
There can be several neurons connected by the process of synapse, and they can form multiple layers.
One can form multiple nodes, and multiple layers. There is really no theoretical limits to how many layers can be formed.
A neural network is convoluted, when they are mixed up and sending many edges in the subsequent layers, and it is called deep, when there are at least ~ 3 or more layers of them.
It can be seriously deep. People have created up to ~ 128 layers.
Now here is how the deep learning process looks like.
Deep learning wants to take input, and pass it to subsequent layers, and so on until a desirable output is met. With iterative process, the weights of the branches is changed, so as to match the output with the desirable output.
For example, if you want to figure out when a restaurant is open in evening vs morning, you can create this network, and evolve the weights according the the formula: adjustment = error x gradient x delta.
For more complex examples ~ you can see that the desirable outcome is two swirls of blue and orange, but after a lot of iterations, the computer program has derived that the pattern looks somewhat similar to the input given to it.
This process is called training. Training is done iteratively until the weights no longer change, or until the error is below a reasonable value.
You can try to play with this toy model on this website: Tensorflow — Neural Network Playground
There are many other complex applications. One of them is face recognition and identification. The faces are converted into mathematical features, and data are passed to multiple layers and the deep learning algorithm is trained enough until a model is able to recognize the faces from images, or also identify whose faces it is able to detect. Pictorially, this can be seen in the image below.
When we feed the input as faces, it recognizes faces. When we feed the images of cars, it is able to recognize the cars.
There are more application of this science. One of them is in mitosis detection. Deep learning helps us understand this process.
It is in scanning the internet. Internet carries a lot of information that can be processed with deep learning engines and some really good conclusions can be derived.
I have come to know that people are applying deep learning in the fluid dynamics problems where conventional methods are not enough. (Don’t forget that Navier Stokes equations are very hard to deal with).
People classify music with this. Deep learning algorithms can simply take the whole musical album files and run them through the programs and over user data to predict which songs a person would like.
It can easily classify and segregate singers using this and can easily create music channels.
It can easily translate text and that too online. Look at this woman holding the picture in her hands, which carries text in English, but the cell phone camera + software + screen convert it real time into Spanish. Most likely all this was achieved with deep learning.
How powerful can this science be? Very powerful!
Look at this classic picture. This is a game called Go. I don’t know how to play this, but the claim of fame is that Google’s Deep Mind company trained a massive computer to play this game. Then it made the computer play against itself ~ 30 million times, and then they invited the world champion of the game. Their original thought was that they were at least 10 years away from playing against the world champion, but they actually achieved this feat in just one year.
I want to say that this is scary. This is exactly why Elon Musk is afraid of it. I have answered this sometime back, and I want to claim, that now I understand it much better why!
What company is Elon Musk referring as the one company he fears of their AI?
You can watch this whole video to learn how it all worked out. It was quite fascinating to me.
Now! The bigger question:
How do I learn deep learning?
There are some really good python libraries available. The two major backend platforms available are Tensorflow, and Theano.
Currently they both have good performance, but tensorflow is being considered to be better, because of ease of use, and performance on CPUs. I think thano works better on GPUs, so it has some advantages there.
Tensorflow website is in the link. TensorFlow
Theano is here. Welcome - Theano 0.8.2 documentation
To use them, one of the best frontend wrapper is called Keras. Keras Documentation.
Keras documentation is really amazing, and within a weekend, you can start your deep learning journey. You can follow Dan Van Boxel to learn this more hands on. The training sessions are not very long, and he shows lots of aspects systematically, and shows how you can figure out things by Googling them, and by reading documentation systematically. I think he is a very nice instructor. I certainly went through some of these instructive videos to see what he is doing and how he is applying it.
Here is my simplest deep learning code. Some of you who have deep knowledge of deep learning would probably suggest improvement, but I barely started on this weekend.
You can take a look at some examples on Kaggle, and learn Keras within no time. Here is how it will look like and feel in Python.
import keras
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import LabelEncoder
from keras.utils.np_utils import to_categorical
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.optimizers import SGD
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
from sklearn import metrics
df = pd.read_csv(path+'InputFile.csv')
df_copy = pd.get_dummies(df)
df_copy.head()
result_column = 'result_column'
df1 = df_copy
y = df1[result_column].values
df1 = df1.drop([result_column],axis=1)
X = df1.values
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.30)
print Xtrain.shape
print Xtest.shape
print ytrain.shape
print ytest.shape
Using Theano backend.
(10499L, 20L)
(4500L, 20L)
(10499L,)
(4500L,)
from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.utils import np_utils
input_dimenstions = Xtrain.shape[1]
print 'Building model'
print '---------------------------------------------------------------'
model = Sequential()
model.add(Dense(output_dim=64, input_dim=input_dimenstions))
model.add(Activation("relu"))
#model.add(keras.layers.core.Dropout(0.2))
model.add(Dense(output_dim=64))
model.add(Activation("relu"))
model.add(Dense(output_dim=2))
model.add(Activation("softmax"))
print 'Compiling model'
print '---------------------------------------------------------------'
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
ytrain = np_utils.to_categorical(ytrain)
print 'Fitting model'
print '---------------------------------------------------------------'
a = model.fit(Xtrain, ytrain, nb_epoch=1000, verbose = 0,shuffle=True)
print 'Calculating loss and metrics'
print '---------------------------------------------------------------'
ytest = np_utils.to_categorical(ytest)
score, acc = model.evaluate(Xtest, ytest, verbose=0)
print('Test score:', score)
print('Test accuracy:', acc)
print("Generating test predictions...")
testpreds = model.predict_classes(Xtest, verbose=0)
testpreds
Building model
---------------------------------------------------------------
Compiling model
---------------------------------------------------------------
Fitting model
---------------------------------------------------------------
Calculating loss and metrics
---------------------------------------------------------------
('Test score:', 0.17593039979454544)
('Test accuracy:', 0.96288888883590695)
Generating test predictions...
Stay blessed, and stay inspired!
