Deep learning for Hackers with MXNet (3): Instant neural art style transfer

This blog is originally posted in Chinese from my zhihu category.

So long from last blog, and thanks for coming back. Here I want to present a MXNet example of instant neural art style transfer, from which you can build your own Prisma app.

Do you know MXNet now can be installed via pip?

pip search mxnet
mxnet-cu75 (0.9.3a3)  - MXNet is an ultra-scalable deep learning framework. This version uses CUDA-7.5.
mxnet (0.9.3a3)       - MXNet is an ultra-scalable deep learning framework. This version uses openblas.
mxnet-cu80 (0.9.3a3)  - MXNet is an ultra-scalable deep learning framework. This version uses CUDA-8.0.

Let’s go

After installing MXNet, please do

git clone https://github.com/zhaw/neural_style

which includes three different implementations of fast neural art style transfer. Big thanks to the author Zhao Wei. In this blog, I am going to talk about Perceptual Losses by Justin Johnson et al described in this paper. After git clone, please go to neural_style/perceptual/ and execute the following script:

import make_image
maker = make_image.Maker('models/s4', (512, 512))
maker.generate('output.jpg', 'niba.jpg')

where output.jpg is the output and niba.jpg is picture of the cutest deep learning cat Niba. Within a blink, we can see the output like this:

niba-instant-transfer1.png

 

Beside this art style, multiple other pretrained neural art models are mentioned in README page under neural_style/perceptual/, please download them via the link mentioned in the page. These pretrained models should produce the art work and combine with ImageTrick:

montage output*.jpg -geometry +7+7+7 merge.jpg

 

niba-instant-transfer2.png

Please note: some machines may encounter the following error

terminate called after throwing an instance of 'dmlc::Error' what():  [21:25:23] src/engine/./threaded_engine.h:306: [21:25:23] src/operator/./convolution-inl.h:299: Check failed: (param_.workspace) >= (required_size)

The reason behind this is from the workspace size of the convolution layers, where the default workspace might be too small for some large images. Please edit symbol.py by adding workspace=4092 to each mx.symbol.Convolution function.

Hope you have some fun with your own Prisma app ­čÖé

Theory

Neural art transfer has been a hot topic in deep learning, and it starts from this paper A Neural Algorithm of Artistic Style. As we have discussed in the last blog, this idea leverages the power of convolutional network where the high level features can describe so called style of an image, if apply this high level feature to a new image, one can transfer the art style and generate new art work. In the original paper, gram matrix is used for this magic. To understand gram matrix magic, one can take look at my friend’s paper Demystifying Neural Style Transfer for further understanding. There are many blogs and papers trying to understand why neural art transfer works, and this paper is probably the only correct one.

Back to the original neural art transfer: the original version calculates the per-pixel loss from the content image to the style image, and introduces a very large gram matrix, meanwhile, it has to run a logistic regression for tuning the weight of each layer. This method needs much computing time due to couple of heavy load from per-pixel loss, gram matrix plus the LR. In the market, there are several faster implementation, where Perceptual Losses method is one of the fastest ones. Perceptual Losses introduces pretrained loss networks from ImageNet, and re-uses the content loss and style loss to calculate perceptual loss, however, it doesn’t update the loss network, which saves much computing time. It works like this: when give the input image (e.g. Niba) to the transform network, it calculates the loss from the pretrained loss network, and gets back to transform network to minimize the loss, so transform network can learn the loss network style from minimizing the loss.

Perceptual loss network needs a set of pretrained network where each network for a style. One can follow train.py under the same repo for creating new styles.

Appendix

Why I paused updating this blog for a long time and resume?

Because I was carefully thinking about teaching MXNet and deep learning in a different way, much different from many other blogs or medium posts where each tutorial starts with theory or math or whatever fundamental knowledge, needs at least 30 minutes reading time, professionals don’t like the repeated fundamental knowledge part since they already know it, but new readers can’t understand what to do.

I believe the only way readers can remember the knowledge is by JUST DO IT!. From last year, I opened my category on zhihu.com and started publishing two minutes demo of deep learning in Chinese. It turned out very welcomed: my 2000+ followers had much fun trying these demos, they really learned after doing it and reading the theory part. If miss some math knowledge, I showed them where to learn. So, I am thinking that, why don’t I translate it back to English, and share with more readers. I will keep posting more blogs like this, hope you like them.

And, as always, have you clicked the star and fork buttons on MXNet repo https://github.com/dmlc/mxnet ?

Setup Amazon AWS GPU instance with MXnet

Niba Meisje met de parel

(every blog should have a cat, right? This cat is “Niba”, and she is Pogo’s younger sister. The neural art style is “Girl with a Pearl Earring“. More about Neural Art with MXnet, please refer to my last blog.)

With popular requests, I wrote this blog for starting an Amazon AWS GPU instance and install MXnet for kaggle competitions, like Second Annual Data Science Bowl. Installing CUDA on AWS is kind of tricky: one needs to update kernels and solve some conflicts. I want to have special thanks to Caffe EC2 installation guide https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN)

Have you clicked “star” on MXnet’s github repo? If not yet, do it now:┬áhttps://github.com/dmlc/mxnet

Create AWS GPU instance

For creating an AWS instance, please follow http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html. AWS spot instance can be an inexpensive solution for competing on Kaggle, and one can request a g2.2xlarge (single GPU) or a g2.8xlarge (4x GPUs) instance from the AWS console. The price when I wrote this blog is about 0.08$/h for g2.2xlarge and 0.34$/h for g2.8xlarge. Once approved, one can start an official Ubuntu 14.04 instance and start installing MXnet. One can save this AMI image for future reference.

Install dependencies

Preparation

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install -y build-essential git libcurl4-openssl-dev libatlas-base-dev libopencv-dev python-numpy unzip

Reference: http://mxnt.ml/en/latest/build.html

Update Linux kernel on AWS

sudo apt-get install linux-image-extra-virtual

“Important: While installing the linux-image-extra-virtual, you may be prompted “What would you like to do about menu.lst?” I selected “keep the local version currently installed” ” as on Caffe reference.

Disable nouveau

nouveau conflicts with NVIDIA’s kernel module on AWS. One needs to edit /etc/modprobe.d/blacklist-nouveau.conf and disable nouveau.

sudo vi /etc/modprobe.d/blacklist-nouveau.conf

    blacklist nouveau
    blacklist lbm-nouveau
    options nouveau modeset=0
    alias nouveau off
    alias lbm-nouveau off

echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
sudo update-initramfs -u
sudo reboot

Wait before finish rebooting, usually 1 min, and login back to the instance to continue installation:

sudo apt-get install -y linux-source linux-headers-`uname -r`

Reference: https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN)

Install CUDA

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_7.5-18_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1404_7.5-18_amd64.deb
sudo apt-get update
sudo apt-get install -y cuda

Important: Please reboot the instance for loading the driver

sudo reboot

If everything is fine, nvidia-smi should look like this (4 GPU instance) for example:

Screen Shot 2016-01-15 at 11.58.02 PM

Optional: cuDNN

One can apply for the developer program here https://developer.nvidia.com/cudnn. When approved, download cuDNN for Linux (either v4 RC or v3 is fine), upload the cuDNN package from the local computer to the instance, and install cuDNN:

tar -zxf cudnn-7.0-linux-x64-v4.0-rc.tgz #or cudnn-7.0-linux-x64-v3.0-prod.tgz
cd cuda
sudo cp lib64/* /usr/local/cuda/lib64/
sudo cp include/cudnn.h /usr/local/cuda/include/

Reference: https://no2147483647.wordpress.com/2015/12/07/deep-learning-for-hackers-with-mxnet-1/

Install MXnet

git clone --recursive https://github.com/dmlc/mxnet
cd mxnet; cp make/config.mk .

#add CUDA options
echo "USE_CUDA=1" >>config.mk
echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk
#if you have cuDNN, uncomment the following line
#echo "USE_CUDNN=1" >>config.mk
echo "USE_BLAS=atlas" >> config.mk
echo "USE_DIST_KVSTORE = 1" >>config.mk
echo "USE_S3=1" >>config.mk
make -j8

Reference: http://mxnt.ml/en/latest/build.html

Add some link lib path

echo "export LD_LIBRARY_PATH=/home/ubuntu/mxnet/lib/:/usr/local/cuda-7.5/targets/x86_64-linux/lib/" >>> ~/.bashrc

Install python package

One can either use the system’s python or Miniconda/Anaconda python as mentioned in my previous blog. If use system’s python, do:

sudo apt-get install -y python-pip
cd python
python setup.py install --user

Test it

python example/image-classification/train_mnist.py --network lenet --gpus 0

One can also give --gpus 0,1,2,3 for using all 4 GPUs, if runs on a g2.8xlarge (4x GPUs) instance. Enjoy Kaggle competitions with MXnet!

Some trouble shooting

One may see some annoying message when starting training with GPU

libdc1394 error: Failed to initialize libdc1394

It is OpenCV problem on AWS. One can simply disable it by:

sudo ln /dev/null /dev/raw1394

Reference: http://stackoverflow.com/questions/12689304/ctypes-error-libdc1394-error-failed-to-initialize-libdc1394

Deep learning for hackers with MXnet (2): Neural art

Special thanks to Eric Xie for fixing the MXnet cuDNN problem. MXnet can fully utilize cuDNN for speeding up neural art.

Have you clicked “star” and “fork” on MXnet github repo? If not yet, do it now! https://github.com/dmlc/mxnet

Update: want an almost real time Neural Art? Let’s go to MXNet-GAN model: it is a┬áGenerative Adversarial Network pretrained model, please refer to┬áhttp://dmlc.ml/mxnet/2016/06/20/end-to-end-neural-style.html for details.

Neural art´╝Üpaint your cat like Van Gogh

Neural art is a deep learning algorithm which can learn the style from famous artwork and apply to a new image. For example, given a cat picture and a Van Gogh artwork, we can paint the cat in Van Gogh style, like this (Van Gogh Self-portrait in 1889 wikipedia):

pogo-vangogh-mxnet_600px

Neural art comes from this paper ÔÇťA Neural Algorithm of Artistic StyleÔÇŁ by Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge http://arxiv.org/abs/1508.06576. The basic idea is leveraging the power of Convolution Network (CNN) which can learn high level abstract features from the artwork and simulating the art style for generating a new image. These generative network is a popular research topic in deep learning. For example, you might know Google’s Inception which generates a goat-shape cloud image by given a goat image and a cloud image. Facebook also has their similar Deep Generative Image Models┬á,┬áinspired by this paper Generative Adversarial Networks where one of the authors, Bing Xu, is also the main author of the great MXnet.

Neural art algorithm has many implementations and applications, for example these two Lua/Torch7 ones LINK1 LINK2. The paper’s gitxiv also includes many interesting applications, for example, generating a neural art gif animation. All of these implemtations uses the VGG model of image classification as mentioned in the paper. With popular demand on github, MXnet has just published its fast and memory efficient implementation. Let’s have fun with MXnet!

Have you clicked the “star” and “fork” on MXnet github repo? If not yet, do it now! https://github.com/dmlc/mxnet

MXnet’s Neural art example

The neural art example is under mxnet/example/neural-style/. Since this example needs much computing work, GPU is highly recommended, please refer to my previous blog “Deep learning for hackers with MXnet (1) GPU installation and MNIST” for detailed installation guide. For sure, one can use CPU anyway since mxnet supports seamless CPU/GPU switch, just a reminder, it may take about 40-50 minutes for generating an image with CPU.

Optional: MXnet can speed up with cuDNN! cuDNN v3 and v4 both work with mxnet, where v4 is 2-3 seconds faster than v3 on my GTX 960 4GB. Please go to https://developer.nvidia.com/cudnn and apply for nVidia Developer program. If approved, one can install CuDNN with CUDA as simple as this (Reference: Install Caffe on EC2 from scratch (Ubuntu, CUDA 7, cuDNN))) :

tar -zxf cudnn-7.0-linux-x64-v3.0-prod.tgz
cd cuda
sudo cp lib64/* /usr/local/cuda/lib64/
sudo cp include/cudnn.h /usr/local/cuda/include/

And please turn on USE_CUDNN = 0 in make/config.mk and re-compile MXnet for CuDNN support, if not previously compiled. Please also update the python installation if necessary.

For readers who don’t have an GPU-ready MXnet, the market has these free or paid services and apps for trying Neural Art. Since neural art needs a lot of computing, all these paid or free services need to upload the images to servers, and wait for a long time for finishing processing, usually from hours (if lucky) to weeks:

  1. Deepart https://deepart.io/ Free submission and the average waiting time is a week ­čśŽ if one wants a faster processing, one can consider donation and speed up to…. 24 hours wait.
  2. Pikazo App: http://www.pikazoapp.com/ It is a paid app (2.99$) as deepart.io, and you still need to wait in line for a long time.
  3. AI Painter: https://www.instapainting.com/ai-painter It is a service from instapainting, free of charge, long waiting time too.
  4. DeepForger: https://twitter.com/DeepForger┬á(Thanks to┬áalexjc┬áfrom reddit) a twitter account that people can submit images and get Neural Art results within hours. “It’s a new algorithm based on StyleNet that does context-sensitive style, and the implementation scales to HD.”

For my dear readers who are lucky to have a GPU-ready MXnet, let do it with MXnet! I am going to use an image from my sister’s cat “pogo” and show every single detail of generating an art image, from end to end.

Steps and parameters

MXnet needs a VGG model.┬áWe need to download it for┬áthe first time running using download.sh. MXnet version of this VGG model takes about several MB where the Lua version of the same model costs about 1 GB. After having the model ready, let’s put the style image and the content image into the input folder. For example, I give mxnet the cat image as content, and Van Gogh’s painting as style:

python run.py --content-image input/pogo.jpg --style-image input/vangogh.jpg

After 1-2 minutes, we can see the output in the output folder like this:

pogo-gogh-512.jpg

Let’s try painting the cat “Pogo” in a modern art style. By replacing Van Gogh with ‘Blue Horse’ Modern Equine Art Contemporary Horse Daily Oil Painting by Texas Artist Laurie Pace (https://www.pinterest.com/pin/407223991276827181/) , pogo is painted like this:

python run.py --content-image input/pogo.jpg --style-image input/blue_horse.jpg

pogo-horse-512

Isn’t it cool?

In the python script run.py, there are some fine tune parameters for better results, and each of them is explained as following:

  • --model The model name. In this example, we only have VGG model from image classification, so please let it as it is. In future, MXnet may provide multiple other models, like Google Inception, since they share the same framework.
  • --content-image Path to the content image, a.k.a the cat image.
  • --style-image Path to the style image, a.k.a the Van Gogh image.
  • --stop-eps The model use eps for evaluating the difference between the content and the style. One can see the eps value converging during the training. The less eps, the more similar style. stop-eps is the threshold for stopping the training. Usually a smaller stop-eps gives stronger style, but it needs longer training time. The default 0.005 value is good, and one can change to 0.004 for better results.
  • --content-weight --style-weight The weight of content and style. By default, it is 10:1. If one thinks the style is too strong, for example, the painting feels strange and harsh, please reduce it to 20:1 or 30:1.
  • --max-num-epochs The max number of epochs, by default it is 1000 epochs. Usually MXnet can converge to a good eps value around 200 epochs, and we can leave this parameter alone.
  • --max-long-edge The max length of the longer edge. MXnet adjust the content image to this size and keeps the aspect ratio. The runtime is almost proportional to the number of pixels (aread) of the image, because the convolution network input size is defined by the number of pixels, and each convolution is on each image block. In short, 700px image may double the memory cost and runtime to that in 500px image. In the following benchmark, one can see that, a 512 px image needs about 1.4GB memory, which is good for a 2014 Macbook Pro or other 2GB CUDA devices; a 850-900 px image is good for 4GB memory CUDA card; if one wants a 1080p HD image, one may need to get a 12GB memory Titan X. Meanwhile, the computing time is related to the number of CUDA cores: the more cores, the faster. I think my dear readers now understand why these free Neural Art services/apps needs hours to weeks of waiting time.
  • --lr logistic regression learning ratio eta for SGD. Mxnet uses SGD for finding a image which has similar content to cat and similar style to Van Gogh. As in other machine learning projects, larger eta converges faster, but it jumps around the minimum. The default value is 0.1, and 0.2 or 0.3 work too.
  • --gpu GPU ID. By default it is 0 for using the first GPU. For people who have multiple GPUs, please specify which ones would be used.--gpu -1 means using CPU-only mxnet, which takes 40-50 minutes per image.
  • --output Filename and path for the output.
  • --save-epochs If save the tempory results. By default, it saves output for each 50 epochs.
  • -remove-noise Gaussian radius for reducing image noise. Neural art starts with white noise images for converging to the neural art from content + style, so it artificially introduces some unnecessary noise. Mxnet can simply smooth this noise. The default value is 0.2, and one can change it to 0.15 for less blur.

Troubleshooting

Out of memory

Since the runtime memory cost is proportional to the size of the image. If --max-long-edge was set too large, MXnet may give this out of memory error:

terminate called after throwing an instance of 'dmlc::Error' what():  [18:23:33] src/engine/./threaded_engine.h:295: [18:23:33] src/storage/./gpu_device_storage.h:39: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: out of memory

To solve it, one needs to have smaller --max-long-edge: for 512px image, MXnet needs 1.4GB memory; for 850px image, MXnets needs 3.7GB. Please notices these two items:

  1. GTX 970 memory issue: GTX 970 can only support up to 3.5GB memory, otherwise it goes crazy. It is an known problem from nVidia, please refer to this link for more details.
  2. The system GUI costs some memory too. In ubuntu, one can press ctrl+alt+f1 for shutting down the system graphic interface, and save some about 40MB memory.

Out of workspace

If the image size is larger than 600 to 700 pixels, the default workspace parameter in model_vgg19.py may not be enough, and MXnet may give this error:

terminate called after throwing an instance of 'dmlc::Error' what():  [18:22:39] src/engine/./threaded_engine.h:295: [18:22:39] src/operator/./convolution-inl.h:256: Check failed: (param_.workspace) >= (required_size)
Minimum workspace size: 1386112000 Bytes
Given: 1073741824 Bytes

The reason is MXnet needs a buffer space which is defined in model_vgg19.py as workspace for each CNN layer. Please replace all workspace=1024 in model_vgg19.py with workspace=2048.

Benchmark

In this benchmark, we choose this Lua (Torch 7) implementation https://github.com/jcjohnson/neural-style and compare it with MXnet for learning Van Gogh style and painting pogo the cat. The hardware include a single GTX 960 4GB, a 4-core AMD CPU and 16GB memory.

512px

Memory Runtime
MXnet (w/o cuDNN) 1440MB 117s
MXnet (w/ cuDNN) 1209MB 89s
Lua Torch 7 2809MB 225s

MXnet has efficient memory usage, and it costs only half of the memory as that in the Lua/Torch7 version.
´┐╝mxnet-lua-512.png

850px

Lua/Torch 7 is not able to run with 850px image because of no enough memory, while MXnet costs 3.7GB memory and finishes in 350 seconds.

Memory Runtime
MXnet (w/o cuDNN) 3670MB 350s
MXnet (w/ cuDNN) 2986MB 320s
Lua Torch 7 Out of memory Out of memory

mxnet-850

MXnet magic to squeeze memory (12.21.2015 update)

With some invaluable discussion from reddit, and special thanks to┬áalexjc┬á(the author of DeepForger) and┬ájcjohnss┬á(the author of Lua Neural-artstyle), I have this updated benchmark with MXnet’s new magic┬áMXNET_BACKWARD_DO_MIRROR to squeeze memory (github issue). Please update to the latest MXnet github and re-compile. To add this magic, one can simply do:

MXNET_BACKWARD_DO_MIRROR=1 python run.py --content-image input/pogo.jpg --style-image input/vangogh.jpg

512px

Memory Runtime
MXnet (w/o cuDNN) 1440MB 117s
MXnet (w/ cuDNN) 1209MB 89s
MXnet (w/o cuDNN + Mirror) 1116MB 92s

850px

Memory Runtime
MXnet (w/o cuDNN) 3670MB 350s
MXnet (w/ cuDNN) 2986MB 320s
MXnet (w/ cuDNN+Mirror) 2727MB 332s

The mirror magic slows down a little bit and gains memory saving. With this Mirror magic, a 4GB GPU can process up to 1024px image with 3855MB memory!

Some comments about improving the memory efficiency: currently in the market, the Lasagne version (with Theano) is the most memory efficient Neural Art generator (github link, thanks to alexjc) which can process 1440px images with a 4GB GPU. antinucleon, the author of MXnet, has mentioned that, gram matrix uses imperative mode while symbolic mode should save more memory by reusing it. I will update the benchmark when the symbolic version is available.

In short, MXnet can save more memory than that in the Lua version, and has some speed up with CuDNN. Considering the price difference between a Titan X (1000$) and a GTX 960 4GB (220$), MXnet is also eco-friendly.

A note about the speed comparision: Lua version uses L-BFGS for the optimal parameter search while MXnet uses SGD, which is faster but needs a little bit tune-ups for best results. To be honest, the comparision above doesn’t mean MXnet is always 2x faster.
´┐╝
For readers who want to know MXnet’s secret of efficient memory usage, please refer to MXnet’s design document where all dark magic happens. The link is http://mxnt.ml/en/latest/#open-source-design-notes

Till now, my dear readers can play with Neural art in MXnet. Please share your creative artwork on twitter or instagram with #mxnet and I will check out your great art!

How machine learns the artwork style?

Quantize the “style”

“Style” itself doesn’t have a clear definition, it might be “pattern” or “texture” or “method of painting” or something else. People believe it can be described by some higher order statistical variables. However, different art styles have different representations, and for a general approach of “learning the style”, it becomes very difficulty to extract these higher order variables and apply to some new images.

Fortunately, Convolution Network (CNN) has proved its power of extracting high-level abstract features in the image classification, for example, computers can tell if a cat is in the image by using CNN. For more details, please refer to Yann Lecun’s deep learning tutorial. The power of “extracting high-level abstract features” is used in Neural Art: after couple of layers of convolution operations, the image has lost its pixel-level feature, and only keeps its high-level style. In the following figure from the paper, the author has defined a 5-layer CNN, where the staring night by Van Gogh keeps some content details in the 1st, 2nd and 3rd layer, and becomes “something looks like staring night” in the 4th and 5th layer:

paper-fig1.png
´┐╝

And the author has reached the “Aha!” moment: if we put a Van Gogh image and one more other image to the same CNN network, some clever adjustment may make the second image closer to Van Gogh, but keeps some content in the first 3 layers. It is the way to simulate Van Gogh painting style! Moreover, there is a VGG model for image classification in the market for it!

Learn style and generate a new image

Now the problem becomes an optimization problem: I want the generated picture looks like my cat (the content feature should be kept for the first 3 layers), and I want Van Gogh style (the style feature for the 4th and 5th layer), thus the solution is an intermediate result which has a similar content representation to the cat, and a similar style representation to Van Gogh. In the paper, the author uses a white noise image for generating a new image closer to the content using SGD, and the other white nose image for being closer to the style. The author has defined a magical gram matrix for describing the texture and has used this matrix to defind the loss function which is a weighted mixture of these two white noise image. Mxnet uses SGD for converging it into a image which meets both of the content and style requirement.

For exmaple, in these 200+ steps of painting pogo the cat, the generated image changes like this:

steps.png

where we can see, in the first 50 epoches, the generated image looks like a simple texture overlap in between the content and the style; with more epoches, the program gradually learns the color, the pattern etc, and becomes stable around 150th epoches, and finally paints pogo the cat in Van Gogh style.

Further reading: other methods of simulating art style

Neural art is not the only method of simulating artwork style and generating new images. There are many other computer vision and graph research papers, for example:

Summary

Neural art is a nice demo for convolution network, and people can generate artwork from their own images. Let’s have fun with MXnet neural art. Please share your creative artwork on twitter or instagram and add hashtag #mxnet.

A reminder about the style: if the content image is a portrait, please find a portrait artwork for learning the style instead of a landscape one. It is the same with landscape images, always landscape to landscape. Because the landscape artwork uses different paiting techniques and it doesn’t look good on portrait images.

In the next blog, I will have detailed introduction to the convolution network for image classification, a.k.a, the dog vs the cat.

Deep learning for hackers with MXnet (1) GPU installation and MNIST

 

I am going to have a series of blogs about implementing deep learning models and algorithms with MXnet. The topic list covers MNIST, LSTM/RNN, image recognition, neural artstyle image generation etc. Everything here is about programing deep learning (a.k.a. deep learning for hackers), instead of theoritical tutorials, so basic knowledge of machine learning and neural network is a prerequisite. I assume readers know how neural network works and what backpropagation is. If difficulties, please review Andew Ng’s coursera class Week 4:┬áhttps://www.coursera.org/learn/machine-learning.

Surely, this blog doesn’t cover┬áeverything about deep learning. It is very important to understand the fundamental deep learning knowledge. For readers who want to know in-depth theoritical deep learning knowledge, please read some good tutorials, for example, http://deeplearning.net/reading-list/tutorials/.

MXnet: lightweight, distributed, portable deep learning toolkit

MXnet is a deep learning toolkit written in C++11, and it comes with DMLC (Distributed (Deep) Machine Learning Common
http://dmlc.ml/). You might have known MXnet’s famous DMLC-sibling xgboost https://github.com/dmlc/xgboost, a parallel gradient boosting decision tree which dominates most Kaggle competitions and is generally used in many projects.

MXnet is very lightweight, dynamic, portable, easy to distribute, memory efficient, and one of the coolest features is, it can run on portable devices (e.g. image recognition on your Android phone ) MXnet also has clear design plus clean C++11 code, let go star and fork it on github: https://github.com/dmlc/mxnet

Recently MXnet has received much attention in multiple conferences and blogs for its unique features of speed and efficient memory usage. Professionals are comparing MXnet with Caffe, Torch7 and Google’s TensorFlow. These benchmarks show that MXnet is a new rising star. Go check this recent tweet from Quora’s Xavier Amatriain: https://twitter.com/xamat/status/665222179668168704

Install MXnet with GPU

MXnet natively supports multiple platforms (Linux, Mac OS X and Windows) and multiple languages (C++, Java, Python, R and Julia, plus a recent support on javascript, no joking MXnet.js). In this tutorial, we use Ubuntu 14.04 LTS and Python for example. Just a reminder that, since we use CUDA for GPU computing and CUDA hasn’t yet support ubuntu 15.10 or newer (with gcc 5.2), let’s stay with 14.04 LTS, or, at latest 15.04.

The installation can be done on physical machines with nVidia CUDA GPUs or cloud instance, for example AWS GPU instance g2.2xlarge or g2.8xlarge. The following steps mostly come from the official installation guide http://mxnt.ml/en/latest/build.html#building-on-linux, with some CUDA modification.

Please note: for installing CUDA on AWS from scratch, some additional steps are needed for updating linux-image-extra-virtual and disabling nouveau, for more details, please refer to Caffe’s guide: https://github.com/BVLC/caffe/wiki/Install-Caffe-on-EC2-from-scratch-(Ubuntu,-CUDA-7,-cuDNN)

Install dependency

MXnet only needs minimal dependency: gcc, BLAS, and OpenCV (optional), that is it. One can install git just in case it hasn’t been installed.

sudo apt-get update
sudo apt-get install -y build-essential git libblas-dev libopencv-dev

Clone mxnet

git clone --recursive https://github.com/dmlc/mxnet

Just another reminder that --recursive is needed: MXnet depends on DMLC common packages mshadow, ps-lite and dmlc-core, where --recursive can clone all necessary ones. Please don’t compile now, and we need to install CUDA firstly.

Install CUDA

CUDA installation here is universal for other deep learning packages. Please go to https://developer.nvidia.com/cuda-downloads for selecting the CUDA installation for the corresponding system. For example, installing CUDA for Ubuntu 14.04 should looks like this, and deb(network) is suggested for fastest downloading from the closest Ubuntu source.

yymd1dsdtnkfbhjuck2o_cuda-selection

Or, here it is the command-line-only solution´╝Ü

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_7.5-18_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1404_7.5-18_amd64.deb
sudo apt-get update
sudo apt-get install cuda

If everything goes well, please check the video card status by nvidia-smi, and it should look like this´╝Ü

zhbzdjy3sbeoc6x6hcyu_idle-gpu

CPU info may vary, and I am using a GTX 960 4GB (approximately 200$ now). MXnet has very efficient memory usage, and 4GB is good for most of the problems. If your video card has only 2GB, MXnet is fine with it with some small parameter tunes too.

Optional: CuDNN. Mxnet supports cuDNN too. cuDNN is nVidia deep learning toolkit which optimizes operations like convolution, poolings etc, for better speed and memory usage. Usually it can speed up MXnet by 40% to 50%. If interested, please go apply for the developer program here https://developer.nvidia.com/cudnn, and install cuDNN by the official instruction when approved,

Compile MXnet with CUDA support

MXnet needs to turn on CUDA support in the configuration. Please find config.mk from mxnet/make/, copy to mxnet/, and edit these three lines:

USE_CUDA = 1
USE_CUDA_PATH = /usr/local/cuda
USE_BLAS = blas

where the second line is for CUDA installation path. The path usually is /usr/local/cuda or /usr/local/cuda-7.5. If readers prefer other BLAS implementations. e.g. OpenBlas or Atlas, please change USE_BLAS to openblas or atlas and add the blas path to ADD_LDFLAGS and ADD_CFLAGS.

We can compile MXnet with CUDA (-j4 for multi-thread compiling):

make -j4

One more reminder that, if one has non-CUDA video cards, for example Intel Iris or AMD R9, or there is not video card, please change USE_CUDA to 0. MXnet is dynamic for switching between CPU and GPU: instead of GPU version, one can compile multi-theading CPU version by setting USE_OPENMP = 1 or leave it to 0 so BLAS can take care of multi-threading, either way is fine with MXnet.

Install Python support

MXnet natively supports Python, one can simply do:

cd python; python setup.py install

Python 2.7 is suggested while Python 3.4 is also supported. One might need setuptools and numpy if not yet installed. I personally suggest Python from Anaconda or Miniconda

wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh
bash Miniconda-latest-Linux-x86_64.sh
(answer some installation questions)
conda install numpy

Let’s run MNIST, a handwritten digit recognizer

Now we have a GPU-ready MXnet, let’s have the first deep learning example: MNIST. MNIST is a handwritten digit dataset with 60,000 training samples and 10,000 testing samples, where each sample is a 28X28 greyscale picture of digits, and the goal of MNIST is training a smart┬ámachine learning model for recognizing hand-writing digits, for example, it recognizes the zip code that people write┬áon envelops and helps our post masters distribute the mails.

Let’s run the example:

cd mxnet/example/image-classification/
python train_mnist.py

train_mnist.py will download MNIST dataset for the first time, please be patient while downloading. The sample code will print out the loss and precision for each step, and the final result should be approximately 97%.

train_mnist.py by default uses CPU only. MXnet has easy switch between CPU and GPU. Since we have GPU, let’s turn it on by:

python train_mnist.py --gpus 0

That is it. --gpus 0 means using the first GPU. If one has multiple GPUs, for example 4 GPUs, one can set --gpus 0,1,2,3 for using all of them. While running with GPU, the nvidia-smi should look like this´╝Ü

oa8r4sairgmijrrest1k_mnist-gpu

where one can see python is using GPU. Since MNIST is not a heavy task, with MXnet efficient GPU meomory usage, GPU usage is about 30-40% while memory usage at 67MBŃÇé

Trouble shooting

When run with GPU for the first time, readers may encounter┬ásomething like this´╝Ü

ImportError: libcudart.so.7.0: cannot open shared object file: No such file 

It is because of the PATH of CUDA dynamic link lib, one can add this to ./bashrc´╝Ü

export LD_LIBRARY_PATH=/usr/local/cuda-7.5/targets/x86_64-linux/lib/:$LD_LIBRARY_PATH

Or compile it to MXnet by adding in config.mk:

ADD_LDFLAGS = -I/usr/local/cuda-7.5/targets/x86_64-linux/lib/
ADD_CFLAGS =-I/usr/local/cuda-7.5/targets/x86_64-linux/lib/

MNIST code secrete revealed: design a simple MLP

In train_mnist.py, there is an function get_mlp(). It implements a multilayer perceptron (MLP). In MXnet, a MLP needs some structure definition, like this in the code:

data = mx.symbol.Variable('data')
fc1 = mx.symbol.FullyConnected(data = data, name='fc1', num_hidden=128)
act1 = mx.symbol.Activation(data = fc1, name='relu1', act_type="relu")
fc2 = mx.symbol.FullyConnected(data = act1, name = 'fc2', num_hidden = 64)
act2 = mx.symbol.Activation(data = fc2, name='relu2', act_type="relu")
fc3 = mx.symbol.FullyConnected(data = act2, name='fc3', num_hidden=10)
mlp = mx.symbol.Softmax(data = fc3, name = 'mlp')

Let’s understand what is going on for this neural network. Samples in MNIST look like these:
hqdefault

  • Each same (a digit) is a 28X28 pixel grey scale image, which can be represented as a vector of 28X28=784 float value where each value is the grey scale of the pixel.
  • In MLP, each layer needs a layer structure. For example in the first layer, fc1 is a full connected layer mx.symbol.FullyConnected which takes input from data. This first layer has 128 nodes, defined as num_hidden.
  • Each layer also need an activation function Activation to connect to the next layer, in other words, transferring values from the current layer to the next layer. In this example, the Activation function for connecting layer fc1 and fc2 is ReLu, which is short for rectified linear unit or Rectifier, a function as f(x)=max(0,x). ReLU is a very commonly used activation function in deep learning, mostly because it is easy to calculate and easy to converge in gradient decent. In addition, we choose ReLU for the MNIST problem because MNIST has sparse feature where most values are 0. For more information about ReLU, please check wikipedia ) and other deep learning books or tutorials.
  • The second layer fc2 is similar to the first layer fc1: it takes the output from fc1 as input, and output to the third layer fc3.
  • fc3 works similar to the previous layer, and the only difference is that, it is an output layer, which has 10 nodes where each node outputs the probability of being one of the 10 digits, thus num_hidden=10.

With this network structure, MXnet also needs the stucture of input. Since each sample is 28X28 grey scale, MXnet takes the grey scale value vector of 28X28=784 elements, and give a python iterator get_iterator() for feeding data to the network defined above. The detailed code is in the example which is very clean, so I don’t copy-n-paste here.

The final step is running the model. If readers know scikit-learn, MXnet’s python looks very familiar, right?

train_model.fit(args, net, get_iterator)

Congratulations! We can implement a MLP! It is the first step of deep learning, not hard, right?

It is Q&A time now. Some of my dear readers may ask, “Do I need to design my MNIST recognizer or some other deep learning network exactly like what you did here?” “hey, I see another function get_lenet() in the code, what is that?”

The answer to these two questions can be, most real life problems on deep learning are about designing the networks, and, no, you don’t have to design your network exactly like what I did here. Designing the neural network is art, and each problem needs a different network, and a single problem may have multiple solutions. In the MNIST example code, get_lenet() implements Yann Lecun’s convolution network LeNet for digit recognition, where each layer needs Convolution Activation and Pooling where the kernel size and filter are needed, instead of FullyConnected and ReLU. FYI: the detailed explanation of super cool convolution network (ConvNet) can be found at Yann Lecun’s tutorial: http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf . Another good reference can be “Understanding ConvNet” by Colah. I may later on write a blog post for explaining ConvNet, since convnet is my personal favorite.

I have a fun homework for my dear readers. Let’s tune up some network parameters like the number of nodes and the activation function in get_mlp(), and see how it helps the precision and accuracy. One can also try changing num_epoch (number of iterations of learning from the data) and learning_rate (the speed of gradient decent, a.k.a the rate of learning) for better speed and precision. Please leave your comment for your network structure and precision score. Kaggle also has a MNIST competition, one can also go compete with mxnet MNIST models, please mention it is MXnet model. The portal: https://www.kaggle.com/c/digit-recognizer

Outlook

Thanks for reading the┬áfirst blog of “Deep learning for hackers with MXnet”. The following blogs will include some examples in MXnet, which may include RNN/LSTM for generating a Shakespeare script (well, looks like Shakespeare), generative models of simulating Van Gogh for painting a cat, etc. Some of these models are available now on MXnet github https://github.com/dmlc/mxnet/tree/master/example. Is MXnet cool? Go star and fork it on github https://github.com/dmlc/mxnet