Object Oriented Neural Network in Java Script

Learn How to Code an Object Oriented Neural Network in Java Script

Posted by Heikki Kupiainen/Oppikone on 25.11.2016 07:10:04

Neural Networks, Cars and OOP

In my previous post I introduced some easy techniques that enable you to program with JavaScript in an object oriented manner. I used some examples with imaginary cars to demonstrate inheritance. However, it would be great to have examples that already have practical value as such.

Therefore I'll try to demonstrate JavaScript OOP (object oriented programming) with some more useful examples than just cars. I am going to implement a neural network. But first of all, just what is a neural network?

Neural Network - What Is It Good For?

"Neural Network" is a very brutally abused term. It makes people think of all kinds of crazy sci-fi movie fantasies. In reality, a neural network is just a set of some useful algorithms. Let's say that you have a table in an SQL database that contains rows of customer data. What is their turnover, how many employees, how much profit they made and so on and did they buy your product or not. There is probably some kind of correlation between all attributes and if they bought the product. You feed the database table into a neural network and it will learn that correlation.

Next time, you have some new companies on your list and you are worrying which ones might be potential customers. Well, you just feed the data you have about them into the neural network and it will - based on previous examples - tell which ones probably would buy the product. This information will save from quite many fruitless phone calls!

Creating Neuron And Neural Net

Sounds great! Now, let's get down to business. I will try to create a simple neural network in JavaScript. The files of this exercise will be available in a GitHub project. Needless to say, there are many neural network implementations in many programming languages and there are many online services providing neural network functionality. But if you really want to understand how something works you must create one by yourself.

Animal and human brains consist of billions of neurons that are somehow interconnected and somehow enable those animal and human brains to think.

So that's why I guess that we need to start by coding a neuron:

var Neuron = {};

That's the neuron class. Now I'll add a constructor method so we can get many instances of neurons. I guess, to create a clever thing we need to have tons of neurons.

Neuron.new = function() {
    //let's figure out later what a neuron contains
   return {};

And to have a really cluster of neurons we need a neural network to contain all neurons:

var NeuralNet = {};

A very basic neural network known as multilayer perceptron typically has a layer of input neurons, hidden neurons and output neurons. Let's add a constructor that takes those values as parameters:

NeuralNet.new = function(inputNeuronCount, hiddenNeuronCount, outputNeuronCount) {
   return {
      inputNeuronCount: inputNeuronCount,
      hiddenNeuronCount: hiddenNeuronCount,
      outputNeuronCount: outputNeuronCount

The parameters passed above instruct the neural network on configuration of its layers. A multilayer perceptron (MLP) often only contains really three layers of neurons. It is possible to have many hidden layers but it depends on the case. Adding many hidden layers does not automatically make a network "more clever". Therefore, "multilayer" perceptron might be initially disappointing to those who wish to see a software equivalent to brain neurons: brain neurons appear to form endlessly deep networks. A "multilayer" neuron that only can afford three layers hardly resembles the construction of thousands or billions of layers that can be found in a biological neural network.

But no need to be discouraged! Despite of their hollowness, multilayer perceptrons are apparently still very useful for machine learning purpose. They can be used as building blocks for truly deep networks. It just does not happen by adding more hidden layers but rather chaining them up into higher level algorithms instead. With MLP, you can have one or two hidden layers, depending on the case. But adding more than that won't make the network any better. In that respect recurrent neural networks are particularly interessant extending the depth by repetition, providing solutions that help memorize frequencies. Some recurrent neural network models are Elman, Jordan and Hopfield. I believe that most probable pathway to increase artificial intelligence, though, comes through the growth of ensemble algorithms. That means combining several useful algorithms to create a new algorithm that is more powerful than its sub-components. An example of this development taking place is "The Random Forest" algorithm that combines two age old machine learning algorithms, namely "ID3 Decision Tree" algorithm and "Bagging" algorithm, into an all-powerful ensemble classifier that challenges much hyped neural network based deep learning algorithms in both performance, simplicity and readibility. While random forests and decision trees are worthy of more study and at least one more blog post, it is noteworthy that also the multilayer perceptron is a great building block for future ensemble algorithms.

Creating Layers

NeuralNet class serves as a container for layers that are created when the neural net is instantiated. Let us create constructors for layers. We create first a super class NeuronLayer that will contain the common features of a neuron layer. Then we create input, output and hidden layers that extend the base class.

Now the classes for layers have been introduced. Please check my previous post for extend function. The next step is to modify NeuralNet's constructor to initialize the layers:

In case the neural network needs to have more than one hidden layer it is useful to modify the constructor to allow multiple hidden layers:

Since JavaScript is a weakly typed language you don't need (and you can't) define data types in parameters. Therefore it is recommendable for clarity to name variables in a way that suggests their data type. For instance, "outputNeuronCount" intuitively indicates an integer value and "hiddenLayerNeuronCounts" refers to an array of integers. In my opinion!

Using the constructor I described above, let's create a neural net with two input neurons, one output neuron and one hidden layer with four neurons:

var neuralNet = NeuralNet.new(2,1,[4]);

For comparison, if we wanted a neural net with two input neurons, one output neuron, one hidden layer with four neurons and one hidden layer with three neurons, we would code:

var neuralNet = NeuralNet.new(2,1,[4,3]);

Now it's time to have a new iteration of neuron layers. Each layer contains an array of neurons. We'll modify NeuronLayer to initialize its neurons instead of jus holding a dummy number to tell the total number of neurons.

Training Data

Neural network's purpose is to learn a certain dataset. The dataset is basically a table of numbers. Each column of the table represents a certain property and each row represents a possible value combination for those properties. This goes well for all datasets that already have continuous values. In case of discrete values (such as categories) the case is more tricky because then the categories must be normalized which will then cause a lot more columns. But it works, anyway. A neural network consisting of multilayer percetrons is useful in classifying tasks. For instance, you can teach the network labeled examples of different species of flowers. Then you can let the trained network use its "wisdom" and label some new unlabeled flower data. A classical example of this is the Iris dataset that contains variations of Iris -flower subspecies pedal lengths.

The simplest and probably most common example training data sample, however, is XOR operator. XOR operator data is already in normalized form (1's and 0's) and its training data is easy to generate. XOR operator takes two parameters and returns true if they differ from each other and false if they are same. The task is to teach the neural network to understand the logic of a XOR operator. Let's create training data for XOR operator:

Each entry of our training data is an array containing two more arrays. First array contains a sample set of parameters 7 and the second array contains an obseved result of that parameter combinations. Now, we get our training data for XOR operator:

var trainingData = XORTrainData.generate();

Training Procedure

When training a neural network with train data, the data matrix is being fed into the neural network row by row. On each iteration, the error margin decreases. Training is ready when the error is smaller than a threshold value. Training can also be stopped until a desired number of iterations is reached. This means that two barrier values must be set before the training starts: maximum number of iterations and error threshold value. For example:

This procedure can be compared with a blind man with a golf club whom you want to teach hitting the hole. Every time you shout "fire!" the blind fellow launches. You check by how wide margin he misses the hole and then you tell him something like "adjust left 2 meters". The blind fellow then launches again. You repeat this procedure until the ball hits the hole (threshold value reached) or you just run out of golf balls (maximum number of iterations reached).

Given that the blind man is a neural network (actually, he is!) and his brain cells are neurons (in deed, they are!) then let NeuralNet.train() be the function to to teach him to hit the hole. Training consists of many repetitions until the targeted skill level is reached (threshold) or the resources reserved for training are used up without reaching the objective (max iterations). Basically almost any training has one thing in common. It is a procedure of exhaustive iterations. Think about teaching a dog to do a trick, a ski jumper pursuing a new record, just anything. Training a neural network can be modeled in JavaScript program code similarly:

Notice that I declared Neural network's train function as a static method. Is it a bad practice for object oriented programming? Absolutely not, if you know that your application is only going to handle one neural network at time! And absolutely yes, if your application must handle many neural networks simultaneously. For simplicity, we are using static methods here. We only need one neural network at any given time.

This is all very simple so far. But miracles start to unravel when we dive in to explore what happens inside all those countless iterations of learning. Warning, this is going to be very complicated, almost rocket science:

Just kidding with rocket science! Each iteration really consists of these very primitive steps. To understand the next part, however, it's important to understand first the nature of a neuron.

Neuron Is Like A Raindrop

When you were little, did you ever observe how raindrops behave on a window glass when it is raining? I believe you did. You stared at a window glass and saw how a raindrop would stay in one place, growing a little bit every time another raindrop flowed down and joined it. And when the raindrop was big enough, it would suddenly divide in two! One part remained where it was while the other part rapidly sprinted downward until it joined another raindrop further below. That's exactly how neurons are! Nature repeats itself everywhere. The same neural pattern can be observed in every place. When you fill a coffee cup and it spills over when it's full, that has something in common with a neuron!

Neuron is a container that absorbs many stimuli and fires one big stimulus when it's full.

My description of a neuron, as you may have noticed, is not very mathematical. It almost sounds too easy, right? But many things we see around us that can be easily understood, for instance raindrops on window, would require extremely complicated mathematical equations to be described formally. Luckily, it works also the other way round: when you look at some mathematical equations you may easily think that is something hopelessly beyond your intellectual reach. But in reality, this is often the case only because mathematical equations are not very intuitive for human mind. I suspect that if some very complex equations and algorithms were described in an object oriented way that is more natural to humans, they would suddenly become children's play for the masses.

Pragmatical programmers who are familiar with object oriented programming have always known it. Almost any overly complex looking formal equation can become relatively easily accessible when converted into object oriented programming code. While OOP might not be yet as intuitive as playing with Legos, it is still a great way to model complex real world interactions in a manner that will help create computer applications to handle them.

Grids Aren't Natural But Trees Are

There are many tutorials about implementing a neural network in any given computer language. Often they start nicely by showing some charts that describe neurons having many inputs and one output, explaining how stimuli enter a neuron through synapses and how the weights are calculated, summed and fed into the activation function. However, those tutorials then degrade - after a neat object friendly start - into explaining how you need matrix calculations to sum up the weights of each neuron layer. Mathematically, it is perfect. Matrix calculation isn't that difficult to understand either. And it probably is the computationally fastest way to update neuron weights. But it is not an object oriented way to model neural interaction. If you were to observe neurons through a microscope, you wouldn't see any matrices or tables or grids where neurons are placed by an invisible hand like pralines in a christmas calendar. You would rather see a tree-like structure. Neurons connect to each other in a fashion that resembles trees.

Tree-like structures are natural. They are everywhere. In river deltas, in blood vessels, under the ground where plants form their Internet of roots, on the roads where highways split into smaller roads, in software code where high level functions split into ever smaller instructions finally reaching the physical level of magnetic zeros and ones. Therefore, I will try to implement a different neural network. A neural network that is a bit more object oriented. I will try to take a more recursive, tree-like approach instead of the typical matrix approach. Something that is a little bit more faithful to what we see in a microscope. This is going to be a hybrid of true biological neurons and their computational abstractions known as multilayer perceptrons.

Anatomy of The Neuron

A neuron contains many input channels that deliver input signals from other neurons. Those channels are called dendrites. When a neuron is being stimulated by activity coming from dendrites, it wakes up and checks whether the incoming total stimulation exceeds the threshold required for outbound stimulus. When that is the case, the neuron then forwards the signal through a tube that is called axon. So a neuron has a multitude of dendrites for inbound signal traffic and one axon for outbound signal traffic. Axon is a broadcast object. It gets an input signal from only one neuron and broadcasts the signal to all dendrites that it is connected to. Therefore neurons are not directly connected to each other but rather through bridges:

Neuron -> Axon -> Dendrites -> Neurons

Since no neuron is useful if it is isolated from others, we'll modify neuron in a fashion that input neurons can be injected into it: generates corresponding dendrites when instantiated.

Anatomy Of The Dendrite

Dendrites really are a very integral part of the biological neuron. They are bridges delivering inbound signal traffic into the neuron. But typically, software implementations of neural networks don't contain dendrites. Typically, they don't really have that much in common with true biological neural networks. They don't introduce any good object oriented models of real neurons. But since my objective is to demostrate OOP, my computational neuron model will in deed include the dendrite.

In this model, it is the dendrites that will contain the "wisdom" of the neural network. The dendrites will contain a weight value that simulates the signal strength of biological neurons. The purpose of a neural network is to learn correlations between input data and their observed result. The correlation is stored in neuron's dendrites and it is being further tuned toward the optimum each time the neuron is fed with a new data sample. When a dendrite is created, it will have a random initial weight between 0 and 1. A dendrite must also have a reference to the host neuron so it can inform the neuron when it's being invoked by its axon. Axon invokes a dendrite by calling its fire function. A dendrite can be fired only if it is loaded. And it cannot be fired twice without being reloaded in between. Only neuron can reload its dendrites. Neuron only reloads its dendrites once it has invoked its axon. This is a safety measure that prevents dendrites from going crazy and "voting" many times.

Please note that Dendrite is now calling neuron's method addDendriteValue that I didn't define yet. Let's implement that function for Neuron class. It's purpose is to add a new weighed value to Neuron. These values will be flowing in to join the neuron like raindrops on a rainy day!

Neuron's Response to Dendritic Activity

Neuron will keep track of these droplets joining in and when all dendrite channels have spitted out their values then the neuron will sum up those values and invoke its outbound signal system, the axon:

Neuron's new instance methods described above will wake up neuron every time a dendrite pours in its content. Neuron checks whether all dendrites have given their value. If so then those values are summed up and passed over to the axon. After that all dendritic values are cleaned up and dendrites reloaded so that they can fire again.

How About Modularity?

Note that I could have used here "The Event Listener" design pattern to implement the interaction between the neuron and its dendrites. It would mean that I would, instead of allowing a dendrite to call directly addDendriteValue of its host neuron, rather let the dendrite fire an "onchange" event when it's been invoked. Some say, that would increase the maintainability through modular isolation, allowing to separate Dendrite from Neuron. Changes made to Neuron's implementation then wouldn't affect the Dendrite's implementation.

While that sort of "modularity" provides a certain theoretical advantage, experience shows that it comes with an enormous cost. Overuse of events makes it difficult to follow the program flow. If you try to follow the program execution you are, generally speaking, lost almost every time the code fires an event. You don't know what parts of the software are listening to the event. You can't be sure of the execution order of those listeners. It's going to be a maintainer's nightmare. Of course you can make it easier by using an IDE that automatially searches for occurences that add a listener to the component. But that only proves that there should be a direct shortcut from the event shooter to the listener. And that shortcut has a name, ladies and gentlemen, and the name is direct invocation.

For reasons described above, the event pattern should only be applied if there are very heavy arguments for it. Such might be some asynchronous cases, for instance when catching user events in the UI. Or cases where the result of an action is highly mutable, having many different possible listeners that could be expected to be catching outbound events. But it is another story and it isn't the case with dendrites of this article: firing a dendrite will always be only a concern of it's very host neuron.

Anatomy Of The Axon

Axon is the output channel of a neuron. Neuron fires an outbound signal when the pressure in the dendrites "spills over". Axon is also a broadcast object. It broadcasts its signal to all target dendrites it ís connected to. Those target dendrites then deliver the signal further to their host neurons.

Since our neural network is going to be essentially an adaptation of multilayer perceptron with a more object oriented approach than usual and with a more biology-mimicking touch than usual, we'll use a classic Sigmoid-activation function that is often used with multilayer perceptrons and place it inside Axon class. Axon will muffle the signal before sending it on so it won't break the system. Just like any reasonable drummer who would want to wear earmuffs when banging their drums.


Now that all moving parts of a neuron have been modeled as JavaScript objects, it is time to put these parts together to create a neural network. Neural network is basically a construction containing three or four layers of neurons. When the neural network is trained with data samples, then those samples travel through all those layers filling each neural cell with different values. Each neuron takes data in from all previous neurons in the previous layer, makes a calculation and passes data on to the neurons in the next layer. But if neurons in each layer get their data from neurons in the previous layer so where can the neurons in the first layer get their data from? The neurons in the first layer can't have any input neurons because they are in the front line. For this reason we need to have another way to feed the first neurons. Luckily we already implemented method addInboundDendrite() in the constructor. It will be particularly useful when feeding data into the first neuron layer of the neural network!

Let's start by creating the first neuron layer, the input layer. For a network supposed to learn XOR operator logic we only need two input values. XOR operator, after all, has only two input parameters:

var inputLayer = InputLayer.new(2);

I just used input layer's constructor method to create a new layer containing two neurons. Those neurons must be fed with data explicitly since they don't have any anterior neurons. We want to make it easy by adding a special fire function to the input layer. It will take an array of numbers as parameters. Those numbers are the train data sample we want the neural network to study. For example, I want the layer to absorb two zeros, which is one possible input value combination for the XOR operator:


Now, let's have a look at InputLayer with fire function implementation:

The upgraded InputLayer class described above is now capable of feeding its neurons directly by using their dendrites as entrances. Next step is to create the next layer in the neural network. Creating the input layer was straightforward because the input neurons don't have any previous neurons to depend on. The next layer, which is called a hidden layer, however needs the neurons if the input layer as input channels. We need the input layer to create the next hidden layer. For this reason it is convenient to have a creation method in NeuronLayer class for creating follow-up layers:

var hiddenLayer = inputLayer.createNextLayer(2);

Now let's have look at createNextLayer() method:

Method described above is capable of creating a new layer. The constructor creates a new neuron layer and injects the input neurons into it. I'll implement the neuron injection method. All neurons of the preliminary layer will be injectd as input neurons to each neuron of the following layer:

The Github Project

I have introduced here some classes that make up the neural network in the scope of this blog post. To test out this construction, however, it is not enough to sketch the parts in an essay but really to implement a project to contain the source code with necessary auxiliatory structures to run the demo on command line, desktop client or web browser. Therefore I structured a GitHub project to contain these elements. The project is available here. The actual source code for the neural network is available in folder src/lib while classes needed for demonstration, UI and tests are available in src/app folder. I added also linux-nw folder to contain NW.js that is a nice tool to execute browser-friendly JavaScript code as a desktop app. NW.js is available at https://nwjs.io/ and it helps run JavaScript based apps like native GUI apps on all major operating systems.

Neural Network Implementation In ES6 Programming Language

This article demonstrated that it is in deed possible to write object oriented code in JavaScript and implement classes, instances and inheritance. But I forgive you if I could not make you love the syntax of my OOP style. But the world is changing rapidly and we are witnessing the rise of new programming languages. EcmaScript Version 6, also called ES6 and ES2015, is an attractive language for web developers who want to see tidier syntax for OOP than just some non-standard hacks. Thanks to emerging transpiling engines such as Babel, it is now possible to write software for browsers in ES6. Therefore I have started a new project to implement a neural network in ES6 language. The project is hosted at https://github.com/develprr/multilayer-perceptron. I'll be back to you soon!