{"version":"https:\/\/jsonfeed.org\/version\/1","title":"mathspp.com feed","home_page_url":"https:\/\/mathspp.com\/blog\/tags\/nnfwp","feed_url":"https:\/\/mathspp.com\/blog\/tags\/nnfwp.json","description":"Stay up-to-date with the articles on mathematics and programming that get published to mathspp.com.","author":{"name":"Rodrigo Gir\u00e3o Serr\u00e3o"},"items":[{"title":"Neural networks fundamentals with Python \u2013 student-teacher","date_published":"2021-07-26T00:00:00+02:00","id":"https:\/\/mathspp.com\/blog\/neural-networks-fundamentals-with-python-student-teacher","url":"https:\/\/mathspp.com\/blog\/neural-networks-fundamentals-with-python-student-teacher","content_html":"

In this article of the NNFwP series<\/a> we'll do the\n“student-teacher” experiment with two neural networks,\nwhere one network will learn directly from the other.<\/p>\n\n

\"A
Original photo by JJ Ying on Unsplash.<\/figcaption><\/figure>

Purpose of this article<\/a><\/h2>\n

The purpose of this article is to try and perform an experiment\nthat is called the “student-teacher” experiment,\nin which we use one large neural network to train another smaller network directly,\ninstead of using conventional training data.<\/p>\n

In the field of machine learning,\nthis is also commonly referred to as “knowledge distillation”.\nThe “student-teacher” experiment we will do in this article\nwill be smaller than similar experiments you might find done in the field,\nbut the principles will be more or less the same;\nour experiment will only be smaller because we are dealing with a small and simple problem,\nthe MNIST classification problem, and because the networks we are using are fairly small.<\/p>\n

If you want to learn more about “student-teacher” experiments\/knowledge distillation,\nmaybe take a look at this survey<\/a>.<\/p>\n

\n

The code for this article, and for the all articles of the series,\ncan be found in this GitHub repository<\/a>.\nThis article will build upon v1.2<\/a> of that code.<\/p>\n<\/div>\n

The layout of the experiment<\/a><\/h2>\n

Let me explain to you how the “student-teacher” experiment works,\nand why we bother doing it.<\/p>\n

As you have seen, neural networks are composed of several layers\nthat are stacked together.\nThe number of layers is variable, and so is the shape of each layer.\nAs it turns out, for more complex tasks, like speech recognition,\nor object recognition, or text translation, among others,\nwe usually have networks much larger than the network we used for\nthe MNIST dataset.<\/p>\n

However, a network being larger brings a couple of downsides with it,\nnamely the computation power it takes to run it and the memory usage\nof having it loaded in memory.\nTo counteract these issues, researchers looked for a way of “compressing”\nthese large networks, and they came up with a really interesting method.\nThey observed that, if you trained a smaller network right from the get-go,\nthe smaller network wouldn't be as good as the larger one.\nBut<\/em>, if they started by training the larger one, then<\/em> they could train\nthe smaller one by teaching it to mimic<\/em> the larger one.\nThey realised that the smaller network\nwould work as well as the original one, or sometimes even better!<\/p>\n

Here are the detailed steps of how this experiment works:<\/p>\n

  1. create, train, and test, a large network in the conventional way – this will be the teacher;<\/li>\n
  2. create a smaller network – this will be the student;<\/li>\n
  3. train the student as follows:\n
    1. traverse the training data;<\/li>\n
    2. feed the input x<\/code> to the teacher and obtain output o<\/code>; and<\/li>\n
    3. train the student on the input data x<\/code> and use the teacher output o<\/code> as the target.<\/li>\n<\/ol><\/li>\n
    4. test the student.<\/li>\n<\/ol>

      Let us do this experiment together, with the MNIST data.<\/p>\n

      Benchmarking the student<\/a><\/h2>\n

      Before we begin, let...<\/p>","date_modified":"2025-12-19T21:31:00+01:00","tags":["machine learning","nnfwp","numpy","programming","python"],"image":"\/user\/pages\/02.blog\/neural-networks-fundamentals-with-python-student-teacher\/thumbnail.webp"},{"title":"Neural networks fundamentals with Python \u2013 subtleties","date_published":"2021-04-30T00:00:00+02:00","id":"https:\/\/mathspp.com\/blog\/neural-networks-fundamentals-with-python-subtleties","url":"https:\/\/mathspp.com\/blog\/neural-networks-fundamentals-with-python-subtleties","content_html":"

      In the fifth article of this short series we will be handling\nsome subtleties that we overlooked in our experiment to classify handwritten\ndigits from the MNIST dataset.<\/p>\n\n

      \"A
      Original photo by JJ Ying on Unsplash.<\/figcaption><\/figure>

      Purpose of this article<\/a><\/h2>\n

      The purpose of this article is to go over the code\nwe have written so far and, in particular, the code\nwe wrote in the previous article<\/a>,\nand fix a couple of little inconsistencies that are related\nto some subtleties I overlooked in the previous articles.<\/p>\n

      \n

      The code for this article, and for the all articles of the series,\ncan be found in this GitHub repository<\/a>.\nThis article will build upon v1.1<\/a> of that code.<\/p>\n<\/div>\n

      Interpreting outputs as probabilities<\/a><\/h2>\n

      In the article where we classified handwritten digits<\/a>\nthere was a point in which we had to create the target\ncolumn vector outputs for the images of the digits:\nwe only knew what digit each image referred to, we didn't\nhave the column vector we wanted the network to output.<\/p>\n

      What we decided to do was create a vector with 0s, except\nin the position of the correct digit:<\/p>\n

      >>> digit = 3\n>>> t = np.zeros((10, 1))\n>>> t[digit] = 1\n>>> t\narray([[0.],\n       [0.],\n       [0.],\n       [1.],\n       [0.],\n       [0.],\n       [0.],\n       [0.],\n       [0.],\n       [0.]])<\/code><\/pre>\n
      I then proceeded to saying we would more or less interpret\nthese numbers as the probabilities that the network assigns to\neach digit; e.g. if the network receives an image of a three,\nwe want the network to say that there's 0% chance that image\nis a 0, 0% chance it is a 1, or 0% chance it is an 8, but we\nwould want the network to say there's 100% chance that that image\nis an image of a 3.\nHowever<\/em>, the activation function we used in the last\nlayer was the leaky ReLU, and the leaky ReLU can output numbers\noutside the range \\([0, 1]\\)<\/span>, which is the range where probabilities\nlie.<\/p>\n
      As we have seen, we can still create something that works if we\noverlook this subtlety, but we should actually be careful to\nmake sure everything is consistent with our assumptions, otherwise\nwe might get a problem later down the road that is caused by\nthese inconsistencies... and if you kick this can down the road,\nit becomes really hard to trace back those future problems\nto this present inconsistency.<\/p>\n
      Fixing this can be as simple as using an activation function\nthat only produces values in the range \\([0, 1]\\)<\/span> in the last layer.\nThe sigmoid function<\/a> is an appropriate alternative,\nas it takes any real number and always produces a value in \\([0, 1]\\)<\/span>.\nThe sigmoid function looks like this:<\/p>\n
      \"\"
      Graph of the sigmoid function.<\/figcaption><\/figure>
      The formula for the sigmoid function is the following:<\/p>\n
      \\[\nf(x) = \\frac1{1 + e^{-x}}\\]<\/p>\n
      If we want to implement a Sigmoid<\/code> activation function, all we need\nto do is figure out what its derivative is, and then implement it\nas a class...<\/p>","date_modified":"2025-12-19T21:31:00+01:00","tags":["machine learning","mathematics","nnfwp","numpy","programming","python"],"image":"\/user\/pages\/02.blog\/neural-networks-fundamentals-with-python-subtleties\/thumbnail.webp"},{"title":"Neural networks fundamentals with Python \u2013 MNIST","date_published":"2021-03-13T00:00:00+01:00","id":"https:\/\/mathspp.com\/blog\/neural-networks-fundamentals-with-python-mnist","url":"https:\/\/mathspp.com\/blog\/neural-networks-fundamentals-with-python-mnist","content_html":"
      In the fourth article of this short series we will apply\nour neural network framework to recognise handwritten digits.<\/p>\n\n
      \"A
      Original photo by JJ Ying on Unsplash.<\/figcaption><\/figure>
      Purpose of this article<\/a><\/h2>\n
      The purpose of this article is to take the neural network\nframework you built in the previous three articles and apply it\nto an actual machine learning problem.\nIn particular, we will take the MNIST dataset – a dataset that\ncontains images of handwritten digits – and train a neural\nnetwork to be able to recognise them.<\/p>\n
      The images we will be working with are like the ones below:<\/p>\n
      \"\"
      Some example images from the MNIST dataset.<\/figcaption><\/figure>
      By the time you are done with this article, you will have a neural\nnetwork that is able to recognise the digit in an image\n9 out of 10 times.<\/p>\n
      \n
      The code for this article, and for the all articles of the series,\ncan be found in this GitHub repository<\/a>.\nThis article will build upon v1.0<\/a> of that code.<\/p>\n
      If you need a refresher on what we built last time, have a quick read\nat the previous article<\/a>.<\/p>\n<\/div>\n
      Getting the data<\/a><\/h2>\n
      In the world of machine learning, the MNIST dataset is very well-known\nand it is common to play around with this dataset when you are experimenting\nwith a new machine learning model.\nThink of it as the “Hello, World!” of machine learning.<\/p>\n
      For your convenience, I compressed the MNIST dataset\nit and made it available for download here<\/a>\n(direct download link<\/a>).<\/p>\n
      After you decompress the data folder, you should get two files:<\/p>\n