Eli
Definition
Machine learning is the art of studying algorithms that learn and improve from examples and experiences.
A typical machine learning tasks are to provide recommendations. For example, for those who have social media accounts such as Facebook, Twitter, and Instagram, all recommendations and advertisements that pop up when you visit these sites are based on the user's historical data. Technical companies use machine learning techniques to improve the user experience with personalized recommendations. Machine learning is used for enormous tasks, such as fraud detection, predictive maintenance, portfolio optimization, and lots of other automation.
Traditional Programming versus machine learning
In traditional programming, a programmer codes all the rules for which software is being developed. In this approach, a programmer writes a logical step-by-step program/instructions to do something. The computer will then execute an output following the logical statements given to it. In contrast to traditional programming, the machine learns how the input and output data are correlated and it writes a rule. The programmers do not need to write new rules each time there is new data. The algorithms adapt and respond to new data and new experiences to improve effectiveness over time. As the machine learns and experiences more data, it is likely to predict more accurately when it did given new data.
Within machine learning are neural networks inspired by the brain (neuroscience), and then deep learning. In principle, deep learning is a subset/sub-field of machine learning and machine learning is one part of Artificial Intelligence (AI). An example of deep learning networks is convolutional neural networks (CNNs), developed by Yann LeCun. Deep learning has been there since the 1990s, but recent advancements in deep learning and its ever-growing popularity, are propelled by more powerful hardware, data availability and more sophisticated accurate algorithms.
Deep Learning
Deep learning is a sub-field of machine learning. Deep learning in the context of AI means the machine uses different layers to learn and improve from the data. The depth of the model is represented by the number of layers in the model. The learning phase in deep learning takes place through a neural network. A neural network is an architecture where the layers are stacked on top of each other.
AI versus machine learning
Although machine learning is one part of AI, the daily use of AI connotes that AI is the science of training machines to apply cognitive functions such as perceiving, learning, reasoning in order to perform human tasks. AI started in 1950 when scientists began to explore ways to make computers mimic human aptitudes and solve problems on their own. Their primary intention which led to the invention of intelligent machines/computers was to develop an ability to "tackle every aspect of learning or any other feature of intelligence that can in principle be so precisely described, that the machine can be made to simulate it."
AI has three different levels, which are, the general state when AI can perform any intellectual task with the same accuracy level as a human, a level when it can perform a specific task better than a human, and a level when AI can beat humans in many tasks.
On the other hand, machine learning (ML) is a distinct sub-field/subset of AI that trains a machine on how to learn. ML is based on the idea that machines can learn on their own from examples/data to recognize patterns and develop rules that it can use for future predictions. Recently, applications of AI have been very successful, and examples of areas where AI rocks are in reducing or avoiding repetitive tasks and improving an existing product by adding new features or enhancing their functionalities. AI is used in almost all industries from marketing to supply chain, to finance and food-processing sector. Recent surveys show that financial services and high technology communications are frontiers of the AI fields. AI is a cutting-edge technology that can deal with complex data which is impossible to handle by a human being.
How does it work?
Way back in 1950s engineers decided to write a computer program that would try to imitate human intelligence. This attempt led to the emergence of a new field of study within AI, called Machine Learning. This is the field of study where machines are trained through examples and experiences -- they learn through lots of data about something so that they can automate a process that dissects out various features/behaviors/characteristics and make accurate predictions. Just like we humans, the more we know or have experience about something/situation, the more easily we can predict similar situations in the future and vice versa. Machines are trained the same.
In summary, a machine learns through an example; when we feed the machine with a similar example, it can figure out the outcome and make an accurate prediction. On the other side, the machine has difficulties to predict if you give it an example that it has never experienced before.
The machine learns by discovering patterns in the given data and make an inference -- a conclusion reached on the basis of evidence and reasoning. Machine learning uses mathematically sophisticated algorithms to simplify a physical reality of some phenomenon and transform this discovery into a model.
Features Vector
The data used to train the machine is very crucial, and should thus be chosen very carefully. This data is usually associated with the list of attributes called a features vector -- a vector that contains information that characterizes an object's important attributes/features. We can think of a features vector as a subset of data that is used to tackle a problem. A features vector is simply a collection of attributes (features) extracted from the input data, usually in the form of a matrix.
Machine Learning Phase
The machine learning phase orderly includes Training data, Features vector, Algorithm, and then Model.
We have seen that machine learning is the general term that implies computers learn from data. Computers learn different algorithms (ways) which can be grouped into supervised, unsupervised, and reinforcement algorithms:-
A. Supervised learning
The data that is fed to a machine learning algorithm can be in the form of input-output pairs or just inputs. Supervised learning algorithms require input-output pairs for their training. It uses labeled data to train the model.
B. Unsupervised learning
Here machine only requires the input data. unsupervised learning algorithm takes an example input data repeatedly, the algorithm learns the data, clusters/classifies the inputs into groups and eventually, it picks up a pattern between the inputs and outputs. The algorithm will then be able to predict the output when it is fed with a brand new input.
C. Reinforcement algorithms
Here machine learns and improves through feedback. It concerns with training an agent to interact with the real-world and maximize its reward. This model of learning in combination with deep learning is on the bleeding edge and massively drives AI. Example implementations in this area include self-driving cars and machines that can play video games.
TensorFlow
Google's TensorFlow is currently the world's most popular open-source ML (deep learning) framework/library for research and production. TensorFlow has a very large API stack, it powers almost every machine learning project at Google, that's, it runs at scale and as well as on small devices like mobile phones. This very large ML framework has been developed rapidly, has a lot of different pieces, and runs on devices, ranging from CPU, GPU, Cloud TPU (tensor processing unit), to android, iOS, and embedded systems. According to this post, "for the most part, the TensorFlow core is written in a combination of highly-optimized C++ and CUDA (Nvidia's language for programming GPUs). Much of that happens, in turn, by using Eigen (a high-performance C++ and CUDA numerical library) and NVidia's cuDNN (a very optimized DNN library for NVidia GPUs, for functions such as convolutions)."
Google uses AI and machine learning to take advantage of its massive data sets to improve efficiency and give users, such as researchers, scientists, and programmers/developers the best experience. Most Google products use machine learning to improve the search engine, translation, image captioning, recommendations, and email communications. TensorFlow library was initially developed by the Google Brain Team to accelerate machine learning and deep neural network research. TensorFlow has several wrappers in a number of languages such as Python, C++ or Java, and is confidently built to run on CPUs or GPUs.
TensorFlow has been around for several years and was first made public in late 2015, with the first stable version appearing in 2017. By July 2018 it had around 1500 active developers, with about 1000 contributors coming outside Google. TensorFlow is licensed under the Apache License 2.0.
At high level, almost every programmer writes TensorFlow code in Python. TensorFlow is massively powered by C++ back-end. Under this TensorFlow introduction, we will only focus on Python front-end. There is also something called TensorFlow.js, which is fantastic for writing really useful TensorFlow projects in JavaScript. TensorFlow thus supports other several languages, such as Java, JavaScript, and R.
TensorFlow is not just a Python API for training certain models, but, it contains a massive collection of projects - a giant codebase, which includes everything from-to putting the code into production. According to Google, TensorFlow has no secret source, the open-source is what exactly used internally by Google.
Why the term TensorFlow?
TensorFlow takes inputs as multi-dimensional arrays, known as tensors. A tensor is an N-dimensional array, a vector or matrix that represents all types of data. Values in a tensor carry identical data type with a known (or partially known) dimensionality (shape of a matrix or array).
It is possible to construct a set of operations about input in TensorFlow that proceed in a flowchart model also called graph - a set of successive computations. The input/tensors enter the system at one end and flow through a series of operations coming out at the other side as output, and hence the name TensorFlow.
A tensor may originate from the input data or as the result of a computation. Thus, computations in TensorFlow involve tensors, and all the operations are carried inside a graph (by connecting tensors together). Each operation is called an "op node" and are connected to each other. The edge of the nodes is the tensor, which is a way of populating the operation with data.
Deep learning relies on a lot of algebraic operations/computations, especially, matrix multiplication.
API Styles
There are several API styles, they include 1. Keras 2. Estimators 3. Eager execution 4. Deferred execution. Here we will focus on tf.keras which is a TensorFlow (tf) implementation of Keras API.
When you define a neural network, one task you have to do is to lay down a sequence of layers. Keras is broadly an API for defining and training neural networks using lego-like building blocks. Keras is an API stack, traditionally runs on top of other deep learning libraries. You can run Keras on top of TensorFlow, or on top of other libraries, such as MXNet and CNTK. In addition, we can incorporate Keras in the tf, which means we can write a good Keras code inside the tf.
Keras: A high-level API specification
Used to build and train neural networks using lego-like building blocks. If we want to use Keras outside the tf, this is how we would install and call it:
- pip install keras
- import keras
By default, this will also install tf behind the scenes and use it as a back-end.
tf.keras: TensorFlow's implementation of the Keras API
This is a superset of the Keras API and is useful in the following ways:
- Adds support for tf specific functionality. eg., eager execution
- Makes tf much easier to use
- pip install tensorflow
- from tensorflow import keras
In this way, there is no need to install Keras separately, everything is bundled-in.
In this machine learning introductory topic, we will use and fully adopt some materials from Google Colaboratory, a free web-based Jupyter notebook environment which runs entirely in the cloud. It comes with TensorFlow pre-installed on it, and thus anyone can jump in and start writing the code. Colaboratory is great for education demos and comes with free GPU and lots of other resources. With Colaboratory, one can write and run code, save and share analyses, and access powerful computing resources, all these freely, just from the browser. We introduce how TensorFlow works with the image classification "Hello World' program.
Image classification (MNIST)
We will use tf.keras, a high-level API to build and train models in TensorFlow, specifically, to train a neural network model to classify images of clothing, like sneakers and shirts MNIST dataset. Here are a few things to note:
To run or execute a cell use Shift/Ctrl + Enter. Cells need to be run in order, just as we would with any Jupyter notebook. To restore a notebook to a clean start, just select Runtime -> Restart runtime. To enable the GPU, go to Edit -> Notebook settings and then choose the Hardware accelerator to GPU and Save the selected changes. Click connect on the top right corner to connect to Google Compute Engine backend (GPU, TPU, etc.,). On the left bar near the top, you can easily navigate the Table of contents, show code snippet panel and filter/find code snippets, and upload files, refresh or even Mount the Google Drive. We take advantage of Colaboratory in the sense that we do not need to install anything because all requirements are satisfied, however, what we are going to show can be done with TensorFlow locally. This notebook can be saved to Google Drive or Github.
We first need to import the following libraries/routines:
- from __future__ import absolute_import, division, print_function, unicode_literals
- # TensorFlow and tf.keras
- import tensorflow as tf
- from tensorflow import keras
- # Helper libraries
- import numpy as np
- import matplotlib.pyplot as plt
- print(tf.__version__)
It is important to print the TensorFlow version to find out if you are using the required version, as for now print(tf.__version__) yields tf version 2.1.0-rc1.
Importing the Fashion MNIST dataset
The Fashion MNIST dataset contains 70,000 grayscale images in 10 categories. The images show individual articles of clothing at low resolution (28 by 28 pixels), see here.
Here, 60,000 images are used to train the network and 10,000 images to evaluate how accurately the network learned to classify the images. We can access the Fashion MNIST directly from TensorFlow. Importing and loading the Fashion MNIST data directly from TensorFlow goes as follows:
- fashion_mnist = keras.datasets.fashion_mnist
- (train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
This execution results to downloading/loading the data set:
- Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-labels-idx1-ubyte.gz
- 32768/29515 [=================================] - 0s 0us/step
- Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/train-images-idx3-ubyte.gz
- 26427392/26421880 [==============================] - 0s 0us/step
- Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-labels-idx1-ubyte.gz
- 8192/5148 [===============================================] - 0s 0us/step
- Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/t10k-images-idx3-ubyte.gz
- 4423680/4422102 [==============================] - 0s 0us/step
Loading the dataset returns four NumPy arrays:
- The train_images and train_labels arrays are the training set -- the data the model uses to learn.
- The model is tested against the test set, the test_images, and test_labels arrays.
Label Class
0 T-shirt/top
1 Trouser
2 Pullover
3 Dress
4 Coat
5 Sandal
6 Shirt
7 Sneaker
8 Bag
9 Ankle boot
Each image is mapped to a single label. Since the class names are not included with the dataset, we store them here for a later use when plotting the images:
- class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
- 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
Exploring the data
Let's explore the format of the dataset before training the model. Executing
- train_images.shape
yields (60000, 28, 28), which shows that there are 60,000 images in the training set, with each image represented as 28 x 28 pixels.
Similarly, there are 60,000 labels in the training set:
- len(train_labels)
- 60000
Each label is an integer between 0 and 9:
- train_labels
- array([9, 0, 0, ..., 3, 0, 5], dtype=uint8)
There are 10,000 images in the test set. Again, each image is represented as 28 x 28 pixels:
- test_images.shape
- (10000, 28, 28)
And the test set contains 10,000 images labels:
- len(test_labels)
- 10000
TensorFlow Architecture
TensorFlow architecture works in three steps, they are:
- Preprocessing the data
- Building the model
- Training and estimating the model
Preprocessing the Data
The data must be preprocessed before training the network. If we inspect the first image in the training set, we will notice that the pixel values fall in the range of 0 to 255:
- plt.figure()
- plt.imshow(train_images[0])
- plt.colorbar()
- plt.grid(False)
- plt.show()
- Preprocessed data image
- image_inspec.png (7.39 KiB) Viewed 9626 times
- Preprocessed data image
- image_inspec.png (7.39 KiB) Viewed 9626 times
We scale these values to a range of 0 to 1 before feeding them to the neural network model. To do so, we divide the values by 255. It's important that the training set and the testing set be preprocessed in the same way:
- [train_images = train_images / 255.0
- test_images = test_images / 255.0
To verify that the data is in the correct format and that we're ready to build and train the network, let's display the first 25 images from the training set and display the class name below each image:
- plt.figure(figsize=(10,10))
- for i in range(25):
- plt.subplot(5,5,i+1)
- plt.xticks([])
- plt.yticks([])
- plt.grid(False)
- plt.imshow(train_images[i], cmap=plt.cm.binary)
- plt.xlabel(class_names[train_labels[i]])
- plt.show()
Building the Model
Building the neural network requires configuring the layers of the model, then compiling the model.
Setting up the layers
The basic building block of a neural network is the layer. Layers extract representations from the data fed into them. Hopefully, these representations are meaningful for the problem at hand.
Most of the deep learning consists of chaining together simple layers. Most layers, such as tf.keras.layers.Dense, have parameters that are learned during training.
- model = keras.Sequential([
- keras.layers.Flatten(input_shape=(28, 28)),
- keras.layers.Dense(128, activation='relu'),
- keras.layers.Dense(10, activation='softmax')
- ])
The first layer in this network, tf.keras.layers.Flatten, transforms the format of the images from a two-dimensional array (of 28 by 28 pixels) to a one-dimensional array (of 28 * 28 = 784 pixels). Think of this layer as unstacking rows of pixels in the image and lining them up. This layer has no parameters to learn; it only reformats the data.
After the pixels are flattened, the network consists of a sequence of two tf.keras.layers.Dense layers. These are densely connected, or fully-connected, neural layers. The first Dense layer has 128 nodes (or neurons). The second (and last) layer is a 10-node softmax layer that returns an array of 10 probability scores that sum to 1. Each node contains a score that indicates the probability that the current image belongs to one of the 10 classes.
As previously introduced, a tensor has a node and an edge. The node carries the mathematical operation and produces endpoints outputs. The edges explain the input/output relationships between nodes.
Compiling the model
Before the model is ready for training, it needs a few more settings. These are added during the model's compile step:
- Loss function -- This measures how accurate the model is during training. We want to minimize this function to "steer" the model in the right direction.
- Optimizer -- This is how the model is updated based on the data it sees and its loss function.
- Metrics -- Used to monitor the training and testing steps. The following example uses accuracy, the fraction of the images that are correctly classified.
- model.compile(optimizer='adam',
- loss='sparse_categorical_crossentropy',
- metrics=['accuracy'])
Training the Model
Training the neural network model requires the following steps:
- Feed the training data to the model. In this example, the training data is in the train_images and train_labels arrays.
- The model learns to associate images and labels.
- You ask the model to make predictions about a test set -- in this example, the test_images array.
- Verify that the predictions match the labels from the test_labels array.
To start training, call the model.fit method -- so called because it "fits" the model to the training data:
- model.fit(train_images, train_labels, epochs=10)
- Train on 60000 samples
- Epoch 1/10
- 60000/60000 [==============================] - 7s 109us/sample - loss: 0.4965 - accuracy: 0.8271
- Epoch 2/10
- 60000/60000 [==============================] - 4s 73us/sample - loss: 0.3744 - accuracy: 0.8655
- Epoch 3/10
- 60000/60000 [==============================] - 4s 73us/sample - loss: 0.3374 - accuracy: 0.8771
- Epoch 4/10
- 60000/60000 [==============================] - 4s 72us/sample - loss: 0.3106 - accuracy: 0.8858
- Epoch 5/10
- 60000/60000 [==============================] - 4s 72us/sample - loss: 0.2927 - accuracy: 0.8919
- Epoch 6/10
- 60000/60000 [==============================] - 5s 75us/sample - loss: 0.2812 - accuracy: 0.8962
- Epoch 7/10
- 60000/60000 [==============================] - 4s 74us/sample - loss: 0.2686 - accuracy: 0.8996
- Epoch 8/10
- 60000/60000 [==============================] - 4s 73us/sample - loss: 0.2566 - accuracy: 0.9043
- Epoch 9/10
- 60000/60000 [==============================] - 4s 72us/sample - loss: 0.2486 - accuracy: 0.9064
- Epoch 10/10
- 60000/60000 [==============================] - 4s 72us/sample - loss: 0.2377 - accuracy: 0.9113
- <tensorflow.python.keras.callbacks.History at 0x7f4be233ac88>
As the model trains, the loss and accuracy metrics are displayed. This model reaches an accuracy of about 0.91 (or 91%) on the training data.
Evaluating accuracy
Next, we compare how the model performs on the test dataset:
- test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
- print('\nTest accuracy:', test_acc)
- 10000/10000 - 1s - loss: 0.3385 - accuracy: 0.8786
- Test accuracy: 0.8786
It turns out that the accuracy on the test dataset is a little less than the accuracy on the training dataset. This gap between training accuracy and test accuracy represents overfitting. Overfitting is when a machine learning model performs worse on new, previously unseen inputs than on the training data. An overfitted model "memorizes" the training data -- with less accuracy on testing data. For more information, see Overfit and underfit, and Strategies to prevent overfitting.
Making Predictions
With the model trained, we can now use it to make predictions about some images.
- predictions = model.predict(test_images)
Here, the model has predicted the label for each image in the testing set. Let's take a look at the first prediction:
- predictions[0]
- array([3.5061245e-09, 1.7832109e-12, 2.8170513e-09, 7.0614859e-12,
- 4.2359991e-09, 3.8344726e-05, 2.7043097e-09, 3.8953245e-04,
- 5.1749037e-08, 9.9957210e-01], dtype=float32)
A prediction is an array of 10 numbers. They represent the model's "confidence" that the image corresponds to each of the 10 different articles of clothing. We can see which label has the highest confidence value:
- np.argmax(predictions[0])
- 9
So, the model is most confident that this image is an ankle boot, or class_names[9]. Examining the test label shows that this classification is correct:
- test_labels[0]
- 9
We graph this to look at the full set of 10 class predictions:
- def plot_image(i, predictions_array, true_label, img):
- predictions_array, true_label, img = predictions_array, true_label[i], img[i]
- plt.grid(False)
- plt.xticks([])
- plt.yticks([])
- plt.imshow(img, cmap=plt.cm.binary)
- predicted_label = np.argmax(predictions_array)
- if predicted_label == true_label:
- color = 'blue'
- else:
- color = 'red'
- plt.xlabel("{} {:2.0f}% ({})".format(class_names[predicted_label],
- 100*np.max(predictions_array),
- class_names[true_label]),
- color=color)
- def plot_value_array(i, predictions_array, true_label):
- predictions_array, true_label = predictions_array, true_label[i]
- plt.grid(False)
- plt.xticks(range(10))
- plt.yticks([])
- thisplot = plt.bar(range(10), predictions_array, color="#777777")
- plt.ylim([0, 1])
- predicted_label = np.argmax(predictions_array)
- thisplot[predicted_label].set_color('red')
- thisplot[true_label].set_color('blue')
Verifying predictions
Let's look at the 0th image, predictions, and prediction array. Correct prediction labels are blue and incorrect prediction labels are red. The number gives the percentage (out of 100) for the predicted label:
- i = 0
- plt.figure(figsize=(6,3))
- plt.subplot(1,2,1)
- plot_image(i, predictions[i], test_labels, test_images)
- plt.subplot(1,2,2)
- plot_value_array(i, predictions[i], test_labels)
- plt.show()
- Prediction with TensorFlow
- Ankle_boot.png (4.85 KiB) Viewed 9626 times
- Prediction with TensorFlow
- Ankle_boot.png (4.85 KiB) Viewed 9626 times
- i = 12
- plt.figure(figsize=(6,3))
- plt.subplot(1,2,1)
- plot_image(i, predictions[i], test_labels, test_images)
- plt.subplot(1,2,2)
- plot_value_array(i, predictions[i], test_labels)
- plt.show()
- Prediction with TensorFlow
- Sneaker.png (4.52 KiB) Viewed 9626 times
- Prediction with TensorFlow
- Sneaker.png (4.52 KiB) Viewed 9626 times
Let's plot several images with their predictions. Note that the model can be wrong even when very confident.
- # Plot the first X test images, their predicted labels, and the true labels.
- # Color correct predictions in blue and incorrect predictions in red.
- num_rows = 5
- num_cols = 3
- num_images = num_rows*num_cols
- plt.figure(figsize=(2*2*num_cols, 2*num_rows))
- for i in range(num_images):
- plt.subplot(num_rows, 2*num_cols, 2*i+1)
- plot_image(i, predictions[i], test_labels, test_images)
- plt.subplot(num_rows, 2*num_cols, 2*i+2)
- plot_value_array(i, predictions[i], test_labels)
- plt.tight_layout()
- plt.show()
Using the Trained Model
Finally, we can use the trained model to make a prediction about a single image.
- # Grab an image from the test dataset.
- img = test_images[1]
- print(img.shape)
- (28, 28)
tf.keras models are optimized to make predictions on a batch, or collection, of examples at once. Accordingly, even though we're using a single image, we need to add it to a list/array:
- # Add the image to a batch where it's the only member.
- img = (np.expand_dims(img,0))
- print(img.shape)
- (1, 28, 28)
Now, let's predict the correct label for this image:
- predictions_single = model.predict(img)
- print(predictions_single)
- [[6.8552431e-04 4.7085687e-14 9.9778843e-01 2.8135696e-09 1.3103909e-03
- 2.6281535e-09 2.1459218e-04 2.8115356e-17 1.1111447e-06 7.8943376e-14]]
- # Plot
- plot_value_array(1, predictions_single[0], test_labels)
- _ = plt.xticks(range(10), class_names, rotation=45)
- Verifying image prediction
- Image_prediction.png (2.24 KiB) Viewed 9626 times
- Verifying image prediction
- Image_prediction.png (2.24 KiB) Viewed 9626 times
- Road_map.png (9.05 KiB) Viewed 9626 times
- Road_map.png (9.05 KiB) Viewed 9626 times
model.predict returns a list of lists -- one list for each image in the batch of data. We grab the predictions for our (only) image in the batch:
- np.argmax(predictions_single[0])
- 2
We see that the model predicts a label as expected.
In our next lesson, we will learn about Text Classification with TensorFlow.
