![\"SOS](\"https://indico.in2p3.fr/event/37891/logo-2009395760.png\"){kind=logo}
\n",
" ![\"Hands](\"https://perso.ip2i.in2p3.fr/ruppin/hands_on_dl.jpg\")
\n",
"
Figure 1: Left: Examples of images in the MNIST database. Right: Schematic representation of the neural network to be implemented.
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Import the data**\n", "\n", "The lines below allow you to import the data that will be used during this hands-on session.\n", "- How many images are there in the training and test samples?\n", "- How many neurons will there be in the input layer of our network?\n", "- Print the 10 first class labels of the training sample" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2024-05-16 16:03:47.079425: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n" ] } ], "source": [ "from tensorflow import keras\n", "from keras.datasets import mnist\n", "\n", "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n", "\n", "# Normalize the data\n", "x_train, x_test = x_train / 255.0, x_test / 255.0 \n", "\n", "# Reshape training and test sample to match input layer\n", "x_train = x_train.reshape(x_train.shape[0], x_train.shape[1] * x_train.shape[2])\n", "x_test = x_test.reshape(x_test.shape[0], x_test.shape[1] * x_test.shape[2])" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(60000, 784)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "x_train.shape" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the class labels are numbers $i$ between 0 and 9. However, we will need the class labels to take the form of vectors of size 10 with a 1 at the $i^{th}$ position and zeros elsewhere. For example, the number 7 becomes (0,0,0,0,0,0,0,1,0,0).\n", "- Complete the following function that will transform a single digit label into a vector label that can be compared to the output of our newtwork" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "def one_hot(Yi):\n", " \"\"\"transform image figure into one-hot vector\n", "\n", " Parameters\n", " ----------\n", " Yi: int\n", " the figure value on the image\n", "\n", " Returns\n", " -------\n", " array\n", " a one-hot encoded vector with 1 at the location of the input figure\n", " \"\"\"\n", " one_hot_Yi = np.zeros(10)\n", " one_hot_Yi[Yi] = 1\n", " return one_hot_Yi" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Code the network**\n", "\n", "The list of functions that we will need in order to train our neural network is the following:\n", "- init_params -- initializes our weights and biases randomly\n", "- ReLU\n", "- Softmax -- the two activation functions that we will consider\n", "- ReLU_deriv -- derivative of the ReLU function\n", "- forward_prop -- forward propagation for a given image\n", "- backward_prop -- backward propagation for a given pair (image, label)\n", "- update_params -- update weight and bias values after data-set full pass\n", "\n", "The init_params, ReLU, Softmax, and ReLU_deriv functions are quite easy to code. They are given in the cells bellow.\n", "- Complete the following cells based on the algorithm that we introduced this morning in order to define forward_prop, backward_prop, and update_params" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def init_params(N_hidden_neurons):\n", " \"\"\"Weight and bias initialization\n", "\n", " Parameters\n", " ----------\n", " N_hidden_neurons: int\n", " number of neurons in hidden layer\n", "\n", " Returns\n", " -------\n", " array, array, array, array\n", " the weight and bias matrices\n", " \"\"\"\n", "\n", " # inverse order of line/column for faster memory access\n", " W0 = np.random.rand(784,N_hidden_neurons) - 0.5\n", " b0 = np.random.rand(N_hidden_neurons) - 0.5\n", " W1 = np.random.rand(N_hidden_neurons,10) - 0.5\n", " b1 = np.random.rand(10) - 0.5\n", " return W0, b0, W1, b1" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def ReLU(Z):\n", " \"\"\"ReLU function for the hidden layer\n", "\n", " Parameters\n", " ----------\n", " Z: array\n", " linear combination from previous layer + bias\n", "\n", " Returns\n", " -------\n", " array\n", " output from hidden neuron\n", " \"\"\"\n", " return np.maximum(Z, 0)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def softmax(Z):\n", " \"\"\"softmax function for the output layer\n", "\n", " Parameters\n", " ----------\n", " Z: array\n", " linear combination from previous layer + bias\n", "\n", " Returns\n", " -------\n", " array\n", " output from output neuron\n", " \"\"\"\n", " A = np.exp(Z) / sum(np.exp(Z))\n", " return A" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "def ReLU_deriv(Z):\n", " \"\"\"derivative of ReLU function\n", "\n", " Parameters\n", " ----------\n", " Z: array\n", " linear combination from previous layer + bias\n", "\n", " Returns\n", " -------\n", " array\n", " array of boolean (interpreted as 0 in 1 in operations)\n", " \"\"\"\n", " return Z > 0" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "def forward_prop(W0, b0, W1, b1, Xi):\n", " \"\"\"forward propagation for a given image\n", "\n", " Parameters\n", " ----------\n", " W0, b0, W1, b1: array\n", " the weight and bias matrices of the network\n", " Xi: array\n", " an image\n", "\n", " Returns\n", " -------\n", " array, array, array, array\n", " the entry and output for each activation function\n", " \"\"\"\n", " Z0 = W0.T.dot(Xi) + b0 # no transpose for W0 because axis swap\n", " A0 = ReLU(Z0)\n", " Z1 = W1.T.dot(A0) + b1 # idem\n", " A1 = softmax(Z1)\n", " return Z0, A0, Z1, A1" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "def backward_prop(Z0, A0, A1, W1, Xi, Yi):\n", " \"\"\"backward propagation for a given (image, class)\n", "\n", " Parameters\n", " ----------\n", " Z0: array\n", " the entry of hidden activation function\n", " A0, A1: array\n", " the output of all activation functions\n", " Xi, Yi: array\n", " an image and its associated class\n", "\n", " Returns\n", " -------\n", " array, array, array, array\n", " derivatives of cross-entropy loss wrt weights and biases\n", " \"\"\"\n", " one_hot_Yi = one_hot(Yi)\n", " delta1i = A1 - one_hot_Yi\n", " delta0i = ReLU_deriv(Z0) * W1.dot(delta1i)\n", " dJdW1 = A0[:, None].dot(delta1i[None, :]) #np.outer(A0,delta1i)\n", " dJdW0 = Xi[:, None].dot(delta0i[None, :]) \n", " dJdb1 = delta1i\n", " dJdb0 = delta0i\n", "\n", " return dJdW0, dJdb0, dJdW1, dJdb1" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "def update_params(W0, b0, W1, b1, dJdW0_arr, dJdb0_arr, dJdW1_arr, dJdb1_arr, alpha):\n", " \"\"\"update weight and bias values after data-set full pass\n", "\n", " Parameters\n", " ----------\n", " W0, b0, W1, b1: array\n", " previous weight and bias matrices of the network\n", " dJdW0_arr, dJdb0_arr, dJdW1_arr, dJdb1_arr: list\n", " lists of derivatives of cross-entropy loss wrt weights and biases\n", " for each image in the training sample\n", " alpha: float\n", " network learning rate\n", "\n", " Returns\n", " -------\n", " array, array, array, array\n", " updated weight and bias matrices of the network\n", " \"\"\"\n", " D = len(dJdW0_arr) # dJdW0_arr.shape[0]\n", " W0 = W0 - (alpha / D) * np.sum(dJdW0_arr, axis=0)\n", " b0 = b0 - (alpha / D) * np.sum(dJdb0_arr, axis=0)\n", " W1 = W1 - (alpha / D) * np.sum(dJdW1_arr, axis=0)\n", " b1 = b1 - (alpha / D) * np.sum(dJdb1_arr, axis=0)\n", " return W0, b0, W1, b1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Prediction and accuracy**\n", "\n", "You will need a function that returns the predicted class label for a given image as well as a function that returns the accuracy of the network in order to record the evolution of the network performance during training. These functions are quite straightforward to code and are given below." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "def get_predictions(A1):\n", " \"\"\"get predicted value for a given image\n", "\n", " Parameters\n", " ----------\n", " A1: array\n", " the output for each neuron in output layer\n", "\n", " Returns\n", " -------\n", " int\n", " the predicted class of the image\n", " \"\"\"\n", " return np.argmax(A1)\n", "\n", "\n", "def get_accuracy(predictions, Y):\n", " \"\"\"get the accuracy of the network\n", "\n", " Parameters\n", " ----------\n", " predictions: list\n", " the predicted classes for a list of images\n", " Y: list\n", " the actual classes for the same images\n", "\n", " Returns\n", " -------\n", " float\n", " the accuracy of the network for the considered set of images\n", " \"\"\"\n", " return np.sum(predictions == Y) / Y.size" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Gradient descent**\n", "\n", "You now have all the tools needed to code your own gradient descent function that will train this neural network to recognize the MNIST images.\n", "- Complete the following cell so that the function returns the weight matrices and bias vectors of the trained network. You will use the get_predictions and get_accuracy functions in order to print the accuracy of the network on the training sample every 10 epochs." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "def gradient_descent(\n", " X, Y, alpha, N_hidden_neurons, iterations\n", "):\n", " \"\"\"train the neural network\n", "\n", " Parameters\n", " ----------\n", " X, Y: array, array\n", " the images and corresponding classes for the\n", " training sample\n", " alpha: float\n", " the learning rate\n", " N_hidden_neurons: int\n", " number of neurons in hidden layer\n", " iterations: int\n", " number of iterations for the wight and bias updates\n", "\n", " Returns\n", " -------\n", " array, array, array, array\n", " weight and bias matrices of the trained network\n", " \"\"\"\n", " W0, b0, W1, b1 = init_params(N_hidden_neurons)\n", " for i in range(iterations):\n", " dJdW0_list, dJdb0_list, dJdW1_list, dJdb1_list, A1_list = [], [], [], [], []\n", " for j in range(X.shape[1]):\n", " Z0, A0, Z1, A1 = forward_prop(W0, b0, W1, b1, X[j,:])\n", " dJdW0, dJdb0, dJdW1, dJdb1 = backward_prop(Z0, A0, A1, W1, X[j,:], Y[j])\n", " dJdW0_list.append(dJdW0)\n", " dJdb0_list.append(dJdb0)\n", " dJdW1_list.append(dJdW1)\n", " dJdb1_list.append(dJdb1)\n", " A1_list.append(A1)\n", " W0, b0, W1, b1 = update_params(\n", " W0, b0, W1, b1, dJdW0_list, dJdb0_list, dJdW1_list, dJdb1_list, alpha\n", " )\n", " if i % 10 == 0:\n", " print(\"Iteration: \", i)\n", " predictions = [get_predictions(A1i) for A1i in A1_list[:100]]\n", " print(get_accuracy(predictions, Y[:100]))\n", "\n", " return W0, b0, W1, b1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- Test your gradient_descent function using a learning rate of 1 and 100 epochs. What is the final accuracy of your network?" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Iteration: 0\n", "0.15\n", "Iteration: 10\n", "0.54\n", "Iteration: 20\n", "0.65\n", "Iteration: 30\n", "0.63\n", "Iteration: 40\n", "0.75\n", "Iteration: 50\n", "0.72\n", "Iteration: 60\n", "0.83\n", "Iteration: 70\n", "0.73\n", "Iteration: 80\n", "0.94\n", "Iteration: 90\n", "0.92\n" ] } ], "source": [ "N_hidden_neurons = 10\n", "learning_rate = 1.0\n", "W0, b0, W1, b1 = gradient_descent(\n", " x_train, y_train, learning_rate, N_hidden_neurons, 100\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "-Check that your network works by running the following cell" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Prediction: 6\n", "Actual class label: 6\n" ] }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAGdCAYAAAC7EMwUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAZpElEQVR4nO3df2jU9x3H8ddp49UfyY2gyd3VmGZDt1FF5o+qaeuPrgYDZvXHwNqyxX9cnT9AUuma2WE6mBGH0j8yHXMj062uUmadoGhTNNHOWlKnKK6TdCaaoSEo7i5GG2f97A/x6DVp6ve88527PB/whebu++59/PaLz35zl298zjknAAAMDLBeAACg/yJCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADAzCPWC/iyO3fu6NKlS8rOzpbP57NeDgDAI+ecOjo6FA6HNWBA79c6fS5Cly5dUkFBgfUyAAAPqLW1VSNHjux1nz737bjs7GzrJQAAkuB+/j5PWYS2bNmioqIiPfroo5o4caKOHj16X3N8Cw4AMsP9/H2ekgjt2rVLq1ev1tq1a3Xy5Ek988wzKi0t1cWLF1PxcgCANOVLxV20p0yZogkTJmjr1q2xx7773e9q3rx5qq6u7nU2Go0qEAgke0kAgIcsEokoJyen132SfiV069YtnThxQiUlJXGPl5SU6NixY9327+rqUjQajdsAAP1D0iN05coVff7558rPz497PD8/X21tbd32r66uViAQiG18Mg4A+o+UfTDhy29IOed6fJOqsrJSkUgktrW2tqZqSQCAPibpPyc0fPhwDRw4sNtVT3t7e7erI0ny+/3y+/3JXgYAIA0k/Upo0KBBmjhxourq6uIer6urU3FxcbJfDgCQxlJyx4SKigr96Ec/0qRJkzRt2jT97ne/08WLF7Vs2bJUvBwAIE2lJEKLFi3S1atX9ctf/lKXL1/W2LFjtX//fhUWFqbi5QAAaSolPyf0IPg5IQDIDCY/JwQAwP0iQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM49YLwDA/RkxYoTnmW3btiX0Wo8//rjnmYULF3qe+fe//+15BpmFKyEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAw3MAXSxLJlyzzPlJWVJfRaW7Zs8Txz4cKFhF4L/RtXQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGW5gChh4/fXXPc+sWbPG88yf/vQnzzOSVFFR4Xnm9u3bCb0W+jeuhAAAZogQAMBM0iNUVVUln88XtwWDwWS/DAAgA6TkPaEnnnhC77//fuzrgQMHpuJlAABpLiUReuSRR7j6AQB8rZS8J9TU1KRwOKyioiK98MILOn/+/Ffu29XVpWg0GrcBAPqHpEdoypQp2rFjhw4ePKht27apra1NxcXFunr1ao/7V1dXKxAIxLaCgoJkLwkA0EclPUKlpaVauHChxo0bp+eee0779u2TJG3fvr3H/SsrKxWJRGJba2trspcEAOijUv7DqkOHDtW4cePU1NTU4/N+v19+vz/VywAA9EEp/zmhrq4uffLJJwqFQql+KQBAmkl6hNasWaOGhgY1Nzfro48+0g9/+ENFo1GVl5cn+6UAAGku6d+O+89//qPFixfrypUrGjFihKZOnarjx4+rsLAw2S8FAEhzPuecs17EF0WjUQUCAetlAPftySef9Dxz4MABzzNDhgzxPDN37lzPM5LiftgcSFQkElFOTk6v+3DvOACAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADATMp/qR2Q6X72s595nknkJr2vvfaa5xluRIq+jishAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmOEu2sAXLF682PPM/PnzPc+88847nmd+/etfe54B+jquhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM9zAFBlp2LBhCc1t2LDB84xzzvNMbW2t5xkgE3ElBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCY4QamyEhTp05NaG7kyJGeZ/bv3+955ujRo55ngEzElRAAwAwRAgCY8RyhI0eOqKysTOFwWD6fT3v27Il73jmnqqoqhcNhDR48WDNnztTZs2eTtV4AQAbxHKHOzk6NHz9eNTU1PT6/ceNGbd68WTU1NWpsbFQwGNTs2bPV0dHxwIsFAGQWzx9MKC0tVWlpaY/POef05ptvau3atVqwYIEkafv27crPz9fOnTv18ssvP9hqAQAZJanvCTU3N6utrU0lJSWxx/x+v2bMmKFjx471ONPV1aVoNBq3AQD6h6RGqK2tTZKUn58f93h+fn7suS+rrq5WIBCIbQUFBclcEgCgD0vJp+N8Pl/c1865bo/dU1lZqUgkEttaW1tTsSQAQB+U1B9WDQaDku5eEYVCodjj7e3t3a6O7vH7/fL7/clcBgAgTST1SqioqEjBYFB1dXWxx27duqWGhgYVFxcn86UAABnA85XQ9evX9emnn8a+bm5u1qlTp5Sbm6tRo0Zp9erVWr9+vUaPHq3Ro0dr/fr1GjJkiF588cWkLhwAkP48R+jjjz/WrFmzYl9XVFRIksrLy/XHP/5Rr776qm7evKnly5fr2rVrmjJlit577z1lZ2cnb9UAgIzgc84560V8UTQaVSAQsF4G0ty//vWvhOa++c1vep5J5Gap//jHPzzPAOkmEokoJyen1324dxwAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMJPU3qwKp8Nhjj3meCYfDCb3W+fPnPc9wR2wgcVwJAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmuIEp+rwdO3Z4nhk2bFhCr7Vq1aqE5gAkhishAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMNzBFnzdr1izPM865hF6rubk5obmH4Rvf+IbnmW9/+9sJvdbjjz/ueeajjz7yPNPS0uJ5BpmFKyEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAw3MMVDlZOT81Be54MPPkho7sKFC0leSc+eeuopzzNvv/2255msrCzPM5K0Y8cOzzM/+MEPPM98+OGHnmdqamo8z6Dv4koIAGCGCAEAzHiO0JEjR1RWVqZwOCyfz6c9e/bEPb9kyRL5fL64berUqclaLwAgg3iOUGdnp8aPH9/r92XnzJmjy5cvx7b9+/c/0CIBAJnJ8wcTSktLVVpa2us+fr9fwWAw4UUBAPqHlLwnVF9fr7y8PI0ZM0ZLly5Ve3v7V+7b1dWlaDQatwEA+oekR6i0tFRvvfWWDh06pE2bNqmxsVHPPvusurq6ety/urpagUAgthUUFCR7SQCAPirpPye0aNGi2D+PHTtWkyZNUmFhofbt26cFCxZ027+yslIVFRWxr6PRKCECgH4i5T+sGgqFVFhYqKamph6f9/v98vv9qV4GAKAPSvnPCV29elWtra0KhUKpfikAQJrxfCV0/fp1ffrpp7Gvm5ubderUKeXm5io3N1dVVVVauHChQqGQWlpa9POf/1zDhw/X/Pnzk7pwAED68xyhjz/+WLNmzYp9fe/9nPLycm3dulVnzpzRjh079N///lehUEizZs3Srl27lJ2dnbxVAwAygucIzZw5U865r3z+4MGDD7QgZLaysrKH8jrvvPNOQnP/+9//PM8MGOD9u9rr1q3zPBMIBDzP9PRhoPvx/vvve55J5L/t73//e88z3MA0s3DvOACAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJhJ+W9WBb6opaXFeglJ973vfc/zzPe//33PM9OnT/c88/e//93zTKKam5s9zwwfPjwFK0E64UoIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADDDDUzxUCVyQ02fz+d55qWXXvI8I0k1NTWeZ5566inPMzt37vQ88zBvRvqwXLp0yXoJMMaVEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghhuYos9rbGz0PDNhwoSEXmvSpEmeZ4YOHep5xjnneaavmzVrlueZw4cPp2AlSCdcCQEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZriBKfq8n/zkJ55njh49mtBrvf76655nXnrpJc8zZWVlnmcef/xxzzMtLS2eZyRp8eLFnmcqKys9z/zqV7/yPIPMwpUQAMAMEQIAmPEUoerqak2ePFnZ2dnKy8vTvHnzdO7cubh9nHOqqqpSOBzW4MGDNXPmTJ09ezapiwYAZAZPEWpoaNCKFSt0/Phx1dXV6fbt2yopKVFnZ2dsn40bN2rz5s2qqalRY2OjgsGgZs+erY6OjqQvHgCQ3jx9MOHAgQNxX9fW1iovL08nTpzQ9OnT5ZzTm2++qbVr12rBggWSpO3btys/P187d+7Uyy+/nLyVAwDS3gO9JxSJRCRJubm5kqTm5ma1tbWppKQkto/f79eMGTN07NixHv8dXV1dikajcRsAoH9IOELOOVVUVOjpp5/W2LFjJUltbW2SpPz8/Lh98/PzY899WXV1tQKBQGwrKChIdEkAgDSTcIRWrlyp06dP6y9/+Uu353w+X9zXzrluj91TWVmpSCQS21pbWxNdEgAgzST0w6qrVq3S3r17deTIEY0cOTL2eDAYlHT3iigUCsUeb29v73Z1dI/f75ff709kGQCANOfpSsg5p5UrV2r37t06dOiQioqK4p4vKipSMBhUXV1d7LFbt26poaFBxcXFyVkxACBjeLoSWrFihXbu3Km//e1vys7Ojr3PEwgENHjwYPl8Pq1evVrr16/X6NGjNXr0aK1fv15DhgzRiy++mJI/AAAgfXmK0NatWyVJM2fOjHu8trZWS5YskSS9+uqrunnzppYvX65r165pypQpeu+995SdnZ2UBQMAMofPOeesF/FF0WhUgUDAehlIc4ncTFOS3njjDc8zp0+ffiivk4iFCxcmNDd37lzPM5s2bfI8U11d7XkG6SMSiSgnJ6fXfbh3HADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwk9JtVgb4u0bsz5+bmep758Y9/7Hlmz549nmd8Pp/nmcbGRs8zkvTcc895njl16lRCr4X+jSshAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMCMzznnrBfxRdFoVIFAwHoZAIAHFIlElJOT0+s+XAkBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZjxFqLq6WpMnT1Z2drby8vI0b948nTt3Lm6fJUuWyOfzxW1Tp05N6qIBAJnBU4QaGhq0YsUKHT9+XHV1dbp9+7ZKSkrU2dkZt9+cOXN0+fLl2LZ///6kLhoAkBke8bLzgQMH4r6ura1VXl6eTpw4oenTp8ce9/v9CgaDyVkhACBjPdB7QpFIRJKUm5sb93h9fb3y8vI0ZswYLV26VO3t7V/57+jq6lI0Go3bAAD9g8855xIZdM7p+eef17Vr13T06NHY47t27dKwYcNUWFio5uZm/eIXv9Dt27d14sQJ+f3+bv+eqqoqvfHGG4n/CQAAfVIkElFOTk7vO7kELV++3BUWFrrW1tZe97t06ZLLyspyf/3rX3t8/rPPPnORSCS2tba2OklsbGxsbGm+RSKRr22Jp/eE7lm1apX27t2rI0eOaOTIkb3uGwqFVFhYqKamph6f9/v9PV4hAQAyn6cIOee0atUqvfvuu6qvr1dRUdHXzly9elWtra0KhUIJLxIAkJk8fTBhxYoV+vOf/6ydO3cqOztbbW1tamtr082bNyVJ169f15o1a/Thhx+qpaVF9fX1Kisr0/DhwzV//vyU/AEAAGnMy/tA+orv+9XW1jrnnLtx44YrKSlxI0aMcFlZWW7UqFGuvLzcXbx48b5fIxKJmH8fk42NjY3twbf7eU8o4U/HpUo0GlUgELBeBgDgAd3Pp+O4dxwAwAwRAgCYIUIAADNECABghggBAMwQIQCAGSIEADBDhAAAZogQAMAMEQIAmCFCAAAzRAgAYIYIAQDMECEAgBkiBAAwQ4QAAGaIEADADBECAJghQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwEyfi5BzznoJAIAkuJ+/z/tchDo6OqyXAABIgvv5+9zn+tilx507d3Tp0iVlZ2fL5/PFPReNRlVQUKDW1lbl5OQYrdAex+EujsNdHIe7OA539YXj4JxTR0eHwuGwBgzo/VrnkYe0pvs2YMAAjRw5std9cnJy+vVJdg/H4S6Ow10ch7s4DndZH4dAIHBf+/W5b8cBAPoPIgQAMJNWEfL7/Vq3bp38fr/1UkxxHO7iONzFcbiL43BXuh2HPvfBBABA/5FWV0IAgMxChAAAZogQAMAMEQIAmEmrCG3ZskVFRUV69NFHNXHiRB09etR6SQ9VVVWVfD5f3BYMBq2XlXJHjhxRWVmZwuGwfD6f9uzZE/e8c05VVVUKh8MaPHiwZs6cqbNnz9osNoW+7jgsWbKk2/kxdepUm8WmSHV1tSZPnqzs7Gzl5eVp3rx5OnfuXNw+/eF8uJ/jkC7nQ9pEaNeuXVq9erXWrl2rkydP6plnnlFpaakuXrxovbSH6oknntDly5dj25kzZ6yXlHKdnZ0aP368ampqenx+48aN2rx5s2pqatTY2KhgMKjZs2dn3H0Iv+44SNKcOXPizo/9+/c/xBWmXkNDg1asWKHjx4+rrq5Ot2/fVklJiTo7O2P79Ifz4X6Og5Qm54NLE08++aRbtmxZ3GPf+c533GuvvWa0oodv3bp1bvz48dbLMCXJvfvuu7Gv79y544LBoNuwYUPssc8++8wFAgH329/+1mCFD8eXj4NzzpWXl7vnn3/eZD1W2tvbnSTX0NDgnOu/58OXj4Nz6XM+pMWV0K1bt3TixAmVlJTEPV5SUqJjx44ZrcpGU1OTwuGwioqK9MILL+j8+fPWSzLV3Nystra2uHPD7/drxowZ/e7ckKT6+nrl5eVpzJgxWrp0qdrb262XlFKRSESSlJubK6n/ng9fPg73pMP5kBYRunLlij7//HPl5+fHPZ6fn6+2tjajVT18U6ZM0Y4dO3Tw4EFt27ZNbW1tKi4u1tWrV62XZubef//+fm5IUmlpqd566y0dOnRImzZtUmNjo5599ll1dXVZLy0lnHOqqKjQ008/rbFjx0rqn+dDT8dBSp/zoc/dRbs3X/7VDs65bo9lstLS0tg/jxs3TtOmTdO3vvUtbd++XRUVFYYrs9ffzw1JWrRoUeyfx44dq0mTJqmwsFD79u3TggULDFeWGitXrtTp06f1wQcfdHuuP50PX3Uc0uV8SIsroeHDh2vgwIHd/k+mvb292//x9CdDhw7VuHHj1NTUZL0UM/c+Hci50V0oFFJhYWFGnh+rVq3S3r17dfjw4bhf/dLfzoevOg496avnQ1pEaNCgQZo4caLq6uriHq+rq1NxcbHRqux1dXXpk08+USgUsl6KmaKiIgWDwbhz49atW2poaOjX54YkXb16Va2trRl1fjjntHLlSu3evVuHDh1SUVFR3PP95Xz4uuPQkz57Phh+KMKTt99+22VlZbk//OEP7p///KdbvXq1Gzp0qGtpabFe2kPzyiuvuPr6enf+/Hl3/PhxN3fuXJednZ3xx6Cjo8OdPHnSnTx50klymzdvdidPnnQXLlxwzjm3YcMGFwgE3O7du92ZM2fc4sWLXSgUctFo1HjlydXbcejo6HCvvPKKO3bsmGtubnaHDx9206ZNc4899lhGHYef/vSnLhAIuPr6enf58uXYduPGjdg+/eF8+LrjkE7nQ9pEyDnnfvOb37jCwkI3aNAgN2HChLiPI/YHixYtcqFQyGVlZblwOOwWLFjgzp49a72slDt8+LCT1G0rLy93zt39WO66detcMBh0fr/fTZ8+3Z05c8Z20SnQ23G4ceOGKykpcSNGjHBZWVlu1KhRrry83F28eNF62UnV059fkqutrY3t0x/Oh687Dul0PvCrHAAAZtLiPSEAQGYiQgAAM0QIAGCGCAEAzBAhAIAZIgQAMEOEAABmiBAAwAwRAgCYIUIAADNECABghggBAMz8H8rFsdbGBzEdAAAAAElFTkSuQmCC", "text/plain": [ "