{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Avoiding overfitting using regularisation\n", "\n", "[Overfitting](regression_overfitting.ipynb) can plague the maximum likelihood approach to model fitting. For example, polynomial fits to the simple 1D regression dataset showed pathalogical behaviour for $D>7$ (see plots below). In this section we will discuss how to mitigate overfitting using regularisation. " ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "import numpy as np\n", "import pandas\n", "import matplotlib.pyplot as plt\n", "\n", "from IPython.display import HTML, set_matplotlib_formats\n", "\n", "set_matplotlib_formats('pdf', 'svg')\n", "css_style = open('../../_static/custom_style.css', 'r').read()\n", "HTML(f'')\n", "\n", "%config InlineBackend.figure_format = 'svg'" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "hide_input": true, "tags": [ "hide-input", "center-output" ] }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDU2OC43ODQzNzUgNTY4LjI2ODc1IF0gL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSCi9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTEgMCBSID4+CnN0cmVhbQp4nL2dS68kx3GF9/0ramkv2Mz3Y+GFBNoEBHghibDXAk3RIjg0JELW3/c5J6ozs/p2950ajinDwEywuiorH5FxIr7K8dsPly9/47fvf97c9gP+/x+b377evvzqu//9y7ff/eHr327f/nxxsH+45NKutaVYM/764/pX/jGUhj/9iGvXv/335fLTBXfHL77Gjb+/XEK79qhfxR6vMbacAm7ue7q2mpz3q/3H1Z5zvYbq7cbzNgcznvfny1+3Bw/xpZS6+ZLxX3qJGaaIv0b89fa/7W/fbf+5/bR9+ZvA/vCbw9v87Xv84YfNX7O6hl3xh6/3P/zt+8tfL/ZHv/mQr7Uk34L3MWwp9WuNrrdcck3btx+2L//dbV/9z7j+i9SuaHXL2eewlXDNobvYa+rh0dU+XYvrzrkec8Vf6zWlEoJPOdRH1+PmzuF1U0q4PLirizGX1H1+1BjvcL8WQou+tcJfhxB6ryW7+LA1uD760HPsvYXNd1yP10i9+ketj+nau6uxhhoKr47Re7x8SQ/vHt01JPzXEkNDP1097u1C6eXRm7IproXWS4i16Mfd+9BweW+P7l7S1Tlc7GtskY1x2dXuQiplufz3l99vf71NBbfZLHDXhlnAuQzLs6mMeZDemH+8XH6LJvxjnzBfsCWYt1ts/WqTU/Pz2w+X336zfflvfK3tmz9f2jXW1tFarrlv/uvyT+Gft29+uPzrN7gPhtTeafwBv8ZPv/jqux/+9B9//+Offvr5iw9/+envP++vc1Hrl1ajldcYtHLWVi/mZ61O6OSMNr3bXP85m1vzNbocXDs2d5pHcw+N8vmadR3WsGu3tc7WObZuQ+vuH5V9uYb2ZjwX88tHZR+uWGm3QR0d8fBR030dHrV6tVePyvGKHphzg8/gzP0sLjBcfrd4P/dmY1jfwwWsu5Cxvj9cMD2Kdxn3ozkVX3zCe8SA7iule5kdVq5jT0a0oISWOsx4Uu4VV8CcryXC41WZvWutdprpeTASfCQekyvdAsxoSYfjjTIH3Dqz6xIuKTU1v6UOh5zRELYkhSuajdvLzOkT2ZIE1+Dx+AJzu7J/ZEVDeu5wnLQmF5NjQxIaUuHssRx6vXZ5P5rhUOBKq6xoObqZu6W71pAS/BnN8DilRprRO87lxgeWa6bH5RMz2lHR7CSzT83ugQ2GmwQfyCbVHjnnMVMSrEXWkNCnbEbu6LKMZQxzuuKVvOc9CtrRPB0lzRG7h2M7sP0kTKfIB8Zr67gh2wFX6WMsPcicUtQDMYEaNzYYMeYdjlNW7iDFuSYzZlvOhWa0owRMPZmxg5XEdlSsA7hfDBcWCse5Rraj4pcYIQ0Xxrw17KY0ox34pYYLLxBtgmDhYzbVXmWlx+9sSEW3o/0Y/gTn2mJxjQ2p2KM8doYoM9aD19pqaAjGruKJTZM5FDakoSHV4zYyJ7iNzEe2xB7OrcrsYkiRU6FhSWH3SXwknt6wFbIl/HPHxhdl9tFhIdCMltSCv8BcrnwBxzmJd0BD0EaZsRdjfGkOVyyOWPlITIBaW2NLOHou5Zhljtwd2RJMgd5c6XwkI6eCyUVzxUopHDJYU0AfsyGc+jFqyBrWYE0hsiEeyw1xhsYMdqyNGBSLYdHiHnhp2R0mtPeyoy1YWAgE8P9orceikr2gi7DJy+yxIluVuV4rpmDnYz3+iAVcZOew9J6L7Awhi5rj0ZyC+cv7YLQ8YiY1B14WTUZIIXuEY0lqDkIlhGHwbQljjtdwQa2B83Y95dZkTyV7r+YglKrVVfQI7Q6D7rLsnSFWx9ik2q4ZwUhXcwImZsBYBtnRitjUnIDmYDXQjN5Gf+juIdE/a/hgDliTWa0JWNeI/Dh+tWBd97JfX69wHhpA2KNrNao1AXGC46zVhtdybUGtiVwmDAtlTxh6r9bA1yL09IXPRfSlteixs2P4YsgyZrih1mTHfHMNgyq7w+Ktagx8bWsJQS3sEfEnppkaE9GYgqCpy47ZHjOnqk8YNriLzIeiN9CXFsQn7g2RHoj2gAfb5WhD66FGbc64XfFqTkJzELlGL3vE4nFqPjwuhhg/gZ3Lttau5sDlFrQ4VdkxaK3p/hnNwVbHEYQbwsxCeCo7mlOdRrAw/Mp2e3jdwphaVkwVzGiZMQwBqxvzBsoGISteUXa0Bl4Qf6Hdd/Ur7WgNZlPnUzG1cBuv1hS8Cjo8B9nhsbE5yI6lFNBXMBe4q873oxle20VfmuwRYXlVc/gXeIbAx3Inr4iTZYc/yhnrW3a4EGxfslOH4G5BdhdzT2oOejY5vRRHAXNLjcFwOrVBdkwYLAPZ0ZoCFcanYibAcyve8pij2EdDTbKjC0JXa2xlaAixt3DLaWoNVhgcmIYQdjgfTAjaseDxQA0h+gmuD3NXdkwvbLdYkbRj/4SEkT3REWBLgB1+E24xasbCc2KrgheVHdFIC2oPvDV6Ah0nu8N6U0TiuUMkBFF4LscNU9OZ7EQggl0L84t2vL7TluOxDSJAwKjDDv1LFdGPMpV2+KSgqOTyx1VGhO13i4Q4hlLPxMRjnYv7PlbMH54qZvzkpPY+/OIQp754huM77nLJ602/H4FrYOAKR9aji5hCnbMGewfmZLDAXGHsV9u/bCNc3nMTlkKw/MRMJXjGk9jo6/F9sFSxfXaM1fF9FvsxlzDv8zaZoFh6XIA9r31qQH2xnMJ2IqcQsSViAUamFcrDlII/lVLwJ1MK/lxKwZ9LKfiTKQV/KqXgT6UU3jT9nZSCP5dS8DOl8PkmE93Ja3U2JzYj5JgYLdOpeCouxLdZdkQJIdpCgKzAVOOGDDu3XXmx4Dk0jVuEwnKHYYyyU3ThB132iN1Ce07A3CjYvOjN6UixZSpQxARgIzK9OewYPK+FGRAkOXjMYmoAE6pr6w2BMkcaQOIBKkahHGI+7IGYGV12zDEvgYT5hZ5HiG9iI1J3yh6pvirjN9oTwzS1JwbJOW7J1CyYqvL+CIP1MHtsqQhc1A2RAizXYhIH8sw1s2OmRwQA9tjm0CHqnkgN1lyzx7YEiSV7oghLIdljG0Sk5AIM2ATdfjW3JbNyHy78i9nhCuwuCAMRLvr9qQWu1OxcdxDL+33wTtKlWEnYArH47Km1RROK2PLQNG7zMvveJHqh5CHFKoIv6wQMrY1hphaL2ZVd50FtqI8z1Vi3McR7xJ662SnH8m0MId2rxEHIWuC3MUQg0RXfhMLIjhLVxhxjZ+2hO6vUdZoj2IG7IoSAtYdXSHlXkoUeRnZqMsoG2REYJusexF5QBHic7Gh2t+4pVGWMRmWHb0gK5xCZIXpm3GbaE1GHzUEEsiUyxJEdwQK8kewUZtwbTZUiECwyU5hFBv+UnwhvmkLvQK3AuLTKDlFSFGiESo+WuRFJrkZMAT0WgZLzEO3R9CpiWmmzQOGFuWF6tUgY0kplljSIjWEJVr06AWGSS06DCGWK/m5mpjKDH0qmY1242dGCjkcE2REKSYIG9lNp0nHQsRCYkh5BayMj4tuFbLIO61SenvGhhGzhEpCdugw/SLLjr8HetFOXRd5TUtZBVthTqcu69LC6uko4ReaIMKmCiVl4X5tnmOX4bWAOgPaALlVEiMgTkwIiytQp13iw+1CYYWmbmIW2CFLneBDWktvFKZpSlV/iDs1kjsaPgejeN9iMsBUHGz+4Xng6dSVmDOZu1/gxFkYPSQ9GT2mWEXfK7rEF2W3QgXBoqUtswoEUDSx2GniCil1GshLzNkmUwAtRqWIsZIcC9xoTOGAE59hXTVYi8C9yIoiU5W6byUqIFlOE0PhX9KVnxoXyEdNmt1OatdBNVnKHLpbZozSDg8tShAgjbhk/BtVoRZTywwzNSphgPnOkazWJB9UfdzulGWZQlx1j7uw+kdqs099S4mE17c+NFGdZo1g4LxAd2X2ghvBTjiKEEaL9YO2PFGeVo0hthkVu3ZCozTBIJrYakwPqtkRt1lvdxVb2STIjJooz6GETWy5504TY3KDU4AqCySf0rEYxUZs1an3s55g5ycZc8RzUZ5MYws9suWEaXBOGjvkXiB44bq8MA1qIsKRQ+FPEhMLwS3aKs8CoiWIFnjIpg4H/Sj/MvAk1SVLsQzO1GUJDPJbhLhy7zXz8FtM3sXPwSwhKXU3ZVWrRXTJmVI1SYJFvgt7gCDJdyYWoRhYqs64RpM6O5hYxJSHMigYQr4c9JsgdR0pnqCWmzpgjxQZsXVYozFpi7gzdhI0z2ERgFh9Xc29Gr8KhOBspzLrgsGjwWGyAAbJcIQHCBGbBGt0FBgcXWKYXcd0Vss9xTBIv4Q571AjYL+DUMAXDOSH3RJU8Uxm482PB8uGpYKEwO6l9jj85iLkXT3lfzCGSDNAb2ICZy3+i5sLHqLnInDbGDn7g8EYYakRmb+tpi/3Q7OU+j9XcvABzvf6Kas5iT6yWEpSFfCDnwik5F07KuXBOzoVzci6clHPhlJwLp+Tcm6a/I+fCOTkXppz7fLPpXTm3rhBWwbDDV65fBEpMx2FP5s4BcRYsBOmMD1UHo1NGYNkUWCWWzehP4SLoBfGSxUplVGcdHnbDZs9tNVtBi4safwlcmJ07SlRIi02GKd5Cl44oCB7fdrvEka+I7uBa0XfNO9tMeZHLeDpcK4VaL0l7NbY6rgNPl86EM+STai2JVVSHGJsuHd2I32ufSpjoGM7ETTmzPNGc1CIUCuQZxJFtjxnbvLaYxJWAzuqWePUOe4La4xVndQtZEFN3+YkUGOVjI2oKuTABJf5SoDhD/GNxJHaXpk0cMg7yrCcVRQi+BG+XU51BtJnaQhc3FbgSAiUssWqVPegEL9XDDvGIiLknqyrTLIBnoR26kWEnS6Yez7GSZKQ3cNwT4XohG3JRMpm1e4/phq0YUSm6tdgQQrki7s0sjjKKxFuo7ZHarHGjz64roaoIAUPMsB1h0Ja9GhBVoUFDrt15jiDEIH7anVVTE7VZ5QjS3lOOdjmlWeQAsiKOLcL2WI0m5ABmLy9veKqULjfugleNWXZ0ldWR6CgxpVkOy5orsSr5zIABogBRmTXfaU7TTmmGadX1uvRQKncxTqHCL6woV+6FyTo5U5tJ72XO6Zi0fhgcFbrqqD52bn+ohgcDZyPivGWeGakh4EQkqwI05rzlMBjZQdK5bAVhLDXTp4wPY4H0z6rbYqvK3uyUZnjFqJqrx6zWOzFc9T2b2ixRoTnNVGYUk1aQhAqQblDwjJ0vmdhiDUShosoyECTOVE9hNKmnVoozOK1gRTP83pm9cJeIVtTCwiw2Qag54JtZPuAvMXWDWsnCGmYXEzBoAfaWqvIBJZNnQTMpHO9cMGpOozhLXTUR1aCqKpCqUjZP7Q6PiTgtmQdp1GbYIJpl/emi9bbUsJGlW0W6VId2d2qznpl/ySxFOsvXWGmZyQQ5EGi3Yp2gNArdTWaQWexa6rKm4cM0xPtY/ZqrF6tPw4dpm+Hs93tTTjsNHxM0UBjmPjqFGWSUV7wJc9+5AAb4eCM6OTzeYAaDKVpjAi0xUZa7lj3nYYYmYgKNSaReteq51BHneOZeYHbwElJHnOUYysjcC4UPXFswO1UZXaLsikho9hRl9B4yw2p8gacm4zqTuUOTanlnBhzZ5iQxi8DOkZ2azEYvqbavRFD2lGSJ4p5v2llf1rvCDRdMAiZe0DMs4mrV50BNVjR6iNyhQNqOTEQW3jV6dAyuK2eH0AK7UTPVgSklv4dBZKWfNXAucrycy3Yx2lIQNNiE4SZpbQwMyfB/Vs5y1RJ5GDXMccgv20HQLVFIAXwbdiIW3iVuWSG2y6nH0HDbQbA4uhwKvCLmJlRKUeqC7y/nj/mFjah5pl24uJk8sttTkCUW3zmtqsMObXSI5bOa3BUHXosSux3cReGgZYTlsXXT+Jn6JjD3s+XALGq0ajDGF1tRI9fCF4RTi/JkmWEnvBGLO3iSR8Rm443exs7MAUREitfFUlZztFoqBzCz1kdZpesJt2QqPdj7QEI/Xkk9UwTPInzc+bFY+PBULFB8ndQdx58clNSLp7yrpBBT4Z4YnMCo5YmQih9VFhs1u8Dd70bMLTW7xX4oXUbiN7f3XIqFq3kWxR48Ro09xdheTimoO8aWuV9JEARF2I8VmsdTCiqeVFDxnIKK5xRUPKmg4ikFFU8pqDdNf0dBxXMKKp5mbA+TeYKSh7n8mFYNjunjQX7+qowtqy75LUm6mJ+2GuFD7R9PBn9O1DawovEWSl3ML6FUlhfqDqU+ZWyZAs75zXgu5pfPYMYZam/FeZ8ytosDOzxq9WuvHsUvHdIB532C2v4CT3gCtaUXhJeCFzmgtqSsque2tK2orc9sU7fy0URtSfrwh+WI2nqWgFQwWknbQCKQqKDMg7Rl9g8h6R5tD9KWZVvfyBxtK2kbIgs9phMGZxvIeiHKKybcbqAtq4nEOU1s3Dhb+ntE1Xsa4MbZBkGjzUcLhW+cLSto7HwLMgdnq7kJrZAt1LlxtixadfjXapHLDbQNvTHBEBTzD9A2uqi0STyCtuieK8u9yoNP0JYlH1etsjw528jJVdSKSdliZ4BbKO5A2bJ2AxHohORNyhYDgCGmNNhWyhbxA+HNKBRyUrYs3MQITcGbTMqWqyj00iSvJmWrcg4mmNimidliq7jSo28rZCtjvJkHZKv1CZWn156QLc2eKaEjZKvmOcXgE7Hlq3iEztI4E7Hli0OtV+N0B2LLUhbubnz3RGwjWQRWc4XBDsSWPQ25kDRpJmLLelspkKfsponYsjqHUN6LL56ILUcWkXKQFB+ILUuCFcpatxiELeuHCJJjN5J2ELasQ6IhzmC9SdiybukoXQyZHYStyp8Blyt2G4Qtq6gEHAxBnIQtq66OlSK7/SBsWbtF/xaj7yZhG1Q99tkAyknYBvZ/q1Zzn4Qteo37UDJgdgC2gQOOPjfScAK2AXLICl4HvlbFcUgow3onXxvoo9Ke2pp8bSC6mAkxbwfAliV8SHWjLSdfy4o/BKqJm4WvJSHAKN66YPK1QcFLagY+Tr5WBELckZKFrw1F1KvRFwtiS5KBSQ977kRsjXyAs7X7D8Q25Kacnr3tRGxpR8u6AZQTsSVZUdsOeC+IreycUrrPZGx1HwfZb+ztYGxpT+0GXE7GVu1M3hndPBnboNojveJ2YGzZD5kZRLVzMraBlUJ4FUOHB2TL7ocb6IY9T8iWw8XkrjOEd0C2HF2stX2hTMhW7p95VDVnQracVBWXG1s+IFtuC9h6LIWxQLacydBA1prJ2BKJoIC3GT4ZW66fokzUAbHlMsQccdHg2B2xJZvAndGeORFb+oRIxsKuHohtZFWtIoJSYyZiq+o+loCyqAtiq6o8VE2y+w/EFq4H71G8Qd4TsY2J+0J3xrNPxJZ16kJHbejtQGxZSO4RXWn3GYhtLNrDvJHoE7Fl0RXb+I7vT8Q2Njqj1Kz9E7Glvm7EsO8RW6rgVoyDWhFb+B8Stkp7fApie9QSj4XuEbF9JqXvf3JSfB+h3DVMffGMk4htoBZ0itpveYR0Eq89vMtSLz68y1p6PiQSllL140zCkYj8xCD6F+O1j7IJ6VQ2IZ3MJqRz2YR0LpuQTmYT0qlsQjqVTXjT9HeyCelcNiE9xWt/wWQ6g9fCGWG1ejrvA15LIlVMn+GyA68leYog0hmeOvFaaqRaMf/EDE68lsgoho6J6QNeyyRBYrFG9594LaFOljiiYbQDrw1CeLLBZQteq1hFdcHtgNdyk8a96o7jDryWVGSszj6KWPBabrrJvnBa4FqGg9h6ioGfA65lTIkZ3gy5HWwtswyOKtWuHmwtQ1m8nNX+FrY2qM6NrVvXT7aWKg16c8dHb2wt+b5+43wnWgsLqVm7xQBruRGjI5J11wRrJRN633nbwdVyf8YCCAamDq6WAq6wIKKGT66WQiaz3q5BnVwtdQ/GMiiLvnC13Oazj67Z9YOrpaZKpdkHrwtXS2UGB9QNv5xcLe2R8KDZB1crO2t2uv/kanUfVteMzx1crZ7r9kLWwtWynQnLU1WGhavle0HuWBi0cLUm6LyRewtXS0XHZWFc8ORqKen4oGS87QBrqek8wzkDaAdYS6kN6aJoZ+FqObh4KYMHFq6WU6FZeXrlaqnuscDtG8YFrI2swWCVRsNzB1hLAQdf6+/JWuNXW5dOXchaho6p3BjXG1kb+N0tXsLmziRruUgSC41mH2gtH8SgWqHXgtaywdRjxjpPtJbvTZbCFslAaxUlO2m2A1obGIxiize2bqK1HB2HVu4o7kBrKdXg/LqhgROt5axAM61wNdFaTiI4BEsELGgtJ13t2RnuONFaTVJMQ/swfKK1nNSYctkwyInWUqJUZkT02MnWcnF0T2p1O7C1kjrd2dtOtJZrDOI9e8u0DbQ2iKCMORhyO9BaXQ/vX+/QWrs9Fq6aP9Faa07tO7o70Fo1v0ZDQxa0Vq+bU9mR2IHWsnuYNLAc30RrpXztg7wFrGXn8xNl4yAnWMuxwtwyKGIBazm2Tp8XbCtYS9HOr/2UiVjAWi5UJiKsMROs5UJN8Hbqm8nVcgZm4TzbgavlhOUM0WqeYC3lW+HH5XdgLfcYag9r/ARruU4Y/zgDaAdYq/wHJmk34HaAtVyouH9Ldp8bWMtliBip73zuAGuVKW97tXkBa2knGm0zaqK18gMsNxsrO9BaNrOknUxa0Fq+FtTo3pmTrWXvVEx8dc5Ea9XHXgHRAa3lEKJhVdm4Ba3lBPHUhxbXD7SWmw6cpIuG4g60lltjrtXtOmCgtQxnEjfbO30QuPMlPfWTwNqjfnuiL45g7TMJ80bAnVQ9dyzuKuFePOUsWPtAw+WTUO3hbZZK9eFt1qL32uS1SP5Qw91hkL+ehruDah+JuHxKxOWTIi6fE3H5nIjLJ0VcPiXi8ikR96bp74i4fE7E5adQ7f+niCN3AX/Js1u+8EeaNjCWa1FJ/wWm5eOS2KwDS9tFCxYjECdKy4Qz2lnvUVomQpkPNjJ2kLSenrpjI7CS2iBpEXJRqiRDVCdJG3QkEXZ+1domSUv1FTyCcyNgB0nLepvnejECdpC0TAJH8qq6/yRpg3C9YCm7haQNyj42O/dhkrSKf3tMzUp/g6RVeQ2zXOWWBaWVdvMKVg4obRRw2XO5Y2kZhEeynUbq3lhaldjKXp1YWFpG/oFH8Kg5k6VViAfZkYyZHSwtlQXrqtbLk6VljAf9ZgeATJhWJTj81N3BtAzxsCiyXT1YWuk6zzMMtgNLSzsV0g7BDpaWESF2W4tuF5hWkWLY+2CytLocjdxR3cHS6vYYQyuxTpZWzanOPuVdWFoFqPyfQbCDpeXbEtk25HKytAx0c1aa4sDSKjBm3fwOpo2Cwv1Okk6cVvKtINQxsHXgtIzTEVMnhWoLTsu4nhrMltDEaTkZENdbtmHBaakn0ON7dw6eluojMcFutxk8LacgFrnptIWnpepBJwfjtidPS/GEuNsU2cLTUoRFOg7df/C0FHaIxKy+t/C0iijR+t0+eFoFrJilGvSJ01LXYR0m6cMFp5WeRU8YyTxwWgaCLKd442MHTsvgHNGbfY694LRU454ZDbv94GlVaqs7Jz1xWixpnlDkbardcNqgsYG21c0nUEvZ4lltvQNqJYv4uacGagK1ViDzhmouQK1FpcVGagFqQ5YDd0qOLUgtUyrMQxjyOZFapmBYu1a5ZGFqaW/NSxwuSC1vwySNt8sHUkt7h5DqBs8OppbNxA5op8EsUK3qbzFbyXWBapWT680+4FygWnYbAnQrdC9ULbuZctOY0knVUtdhJXljGCZVyzHEBmrfIi5UrTIwUukLVavEHrYWA1wnVmv7gLdvcBeslik/fSRr9oHVKp/CmS775Go5j+Us1JjB1UrnsLxoPOzgamkPPI9MfTC5Wt3GB0tzLFyt0hxQy5pqC1er9YB+1UYwuVpNe36zYebB1UpQw40Z9zq5WjhmhgGGgC9cLbORCJFs+SxcLfUPwkr7pHHhatFAbjl2TNHC1bLWlzH42Y6Box5j8LOxKovZ6fRpSeY0RZzBkIizCBPQhjZTj2W4vo2f7EKP2Wvxm1BseJmDy4+J9olGngWLAGP1hT+lz55pjWfa4YjrPpMnbyTdSUVzR/iu+uzFU07iug/kWTmJ6kbWD96crjPNhwIoKa12e8MFDFvNj0DdcbeT5+Bu7vKpjG5jwoNzDkpk1wXllB4rJ/VYOafHyjk9Vk7qsXJKj5VTeuxN09/RY+WcHisfiegyQqQWqYdJvFh1/MFzaJQffTQeW8ePYjZVCzCHDB693uGjJJU6t+54XDGL+Z2nebhYRLlUC+Sa7h6X7x6HsI9pNqaI18ct5vcex1PRMCUCVhrCtcPj/P3bPXYHC3+7eINnqDPPNvt1EWdmJd9yvNP6rK2M8PPHU9mfk2+GfiAYeQ8EL+aXQDAxPXcEgp+OJEGr9vaQ5cX8+lFKlOfXfPPi9w/PWLeDV88glnM8GfkZ3/xJG4hyMJcXORjiwPy0sFgKZsGAK+nbHI7H7TKL34J9IzUo4MqzgBHVW5B+g4AbpTr+u4W4NwZY32GWbFHTQIDpQfErbzHTjQD2qh65rkTHIIAhy/X91t1Ru4QRs6+mLCYBLOiwMQuwrQiw5yna0IsqI08ImGhhrjHq3IxBARMsDLHZZ5mTAvZEt60OvlLAxA2rReIrBUzaEN3RVOafFDBCXdwEXXM8a5cMImPjdDxs1+sjQNcVwk8O2CPWFW59PGyX5uqqsWaTA/aGoClAnRiwLt7D38kB0yrJJTx4cMBqB9ypjueZHDDNnvGAwOPBAfMdmdpR8mVywOyocqOXJwfM3u6sbPMmkwP2qqEV+75tcsBeAIQrGsjBAXuC0cQrjiCw1wKNzs6lnCAw+daU94NkFhCYnCy28WonuE4QGGqA517aN+4TBCazC31nxZUFBCb7i1mcDKucIDB9Hg/FNoJ3cMBsgK8MKLYDB8xjsRHwGLE4MGB+wNzsA/eFAtbxYKEY5zohYIwJfYjhoxMCJmhJmXLHAJNCbDU6Y31vCDCmDFln44IHANx4WGxMRohOABizjsD5fkrv4H85R2PZzz2d+C+kVmW6yajdQf9SgTHNZufoDviXV7P2a2fxDvaXlAF06H5a7kB/2RLCLHb1IH+ZzkH8Yzzw4H4b5wNUqswT+23ywXZ220L9kqVpVuZcmN9WOI1DvUN+G17G2cekC/DLc8laczsHPHhfkuIFI3N3pC5HF38y64B9O3OEeCeDem+sL3M+dEZGDA/Ul+eVO3d3mC6MlZV4u/HAfC09ZCczT8iX1PvtY9/J+DKFj8DBRnYivp0xRQ/5DvH1/FyfmRJjeQfiyyWQGSnacbyD8eVaQ0Psi+zB+HKNYAXa8pqILzcK3NzZ+pqIL11GL+RitwPi6/nVl8NyNCR4IL7cb703AGIhfD3rydh5kpG2g/AlCkbpYut0Er7MHSWMq7snfL1S8vZt+0r41miHAdjhuvMQXag2vKC3Q3En4cvUSiB4YYfiOn7UoRP8P+kD4Cfx8GO9e8frPtbT9784JcAP1x9irhdPOAnr0oWUcXzMLZlQz/K6y8ushevlZRbzMZsw7/IknXAHWH5aTuHcd79HUPdBSqGeSinUkymFei6lUM+lFOrJlEI9lVKop1IKb5r+Tkqhnksp1I/96hdRbed3CEfdvZrf0d2BqegUEIlERgavswqBGxFJzDtHs5jfe1wTO4UHvE4okCjSaflHD7Wa33kSUajMoyByQU8/TSh8tpX6/nepkWlwH/JeQF9PyUVoWZm33g4gL51lz67ecbzcKrHvux3NHRwvg7BApvFwRq5Ccf7bI3a27QB5ubFysfW7M3IVD0J4ubszchV2o4nV0N/B8rI16I1sMOxgeSX8WHAz1nawvIxYeRK/sbCT5WV0iilrUebC8jLgRDCWvZ2FO1hexpAO+3419newvJJorVsFYGF59dFv7fbvjSwsL+OIhqVX7OzcwfIy3KssvtvZuYPlhaS4hv0UlAXl9TpLsSd3d0Qub89jjv3dEbmMUxFc1GQI7kB5GexCptrbTpLXs9DTi/1rJAvJS+mUeYTjHcnrVVj39g+6LCgvRwVxcLZjmifKi4FCJJPLztreUF6KEfxwh2Enyks7Rt+lO5RXKsU7q0ItKC/nTi7w5nZexkB5OdcQIlv5ZbK8nLKcO/HulFxT5Yje1JmT5WXuh77TpvhkeRWyt2b/2s3C8io4xxbpjP29sbyMzglbGPQ6WV54fj4pGeQ3WV7KBxICBvNNlpdTE3plZxQny8ve6Vyex2Ny8Xw03RtIOVFeBJTERMP9IblwVdHvvO7geFnki3aONGltDDz13Bc3ghTLB55z53YmOsrkze0flljIUbYTAi81O6n1Ro5S6UB++P1g10GOen1NEZMhzZMcResKnmOHo7JEx2L96WLZk2DtWfR1xzI+Du/eBLenAsLjDw7h7YtnnAUZH8e37SzLuBYalsLf8jqL+dDm5S6P49t79uzXiG/vIMYHAW47FeC2kwFuOxfgtnMBbjsZ4LZTAW47FeC+afo7AW47F+C2jwxwScwhPuAeuM7l1fxe1Qy+k99VsNSKfel1gKuDLDHed55gNb/3OObXcW0MiiVfR7mZqS2Sf0dXtZrfeRyJEX7n3Pi5g38Z5X6W9fpulJu8lhzju5nL+LCQj6TvmOzaT/8c5KMSOjHav5G3kI9KpGY7jn8BH7WaeNiHnS06wEfmLV3fP82e4CNzi3CtBpdO7pHRZYVnM+Jqco/M+4YeLR+4cI+IjES03R8hildyaT/tdVKP2i8tglyhR+b4mLLeDtAjc5Eu7dzqZB71L2aRntkOyCNzbnBS1pBJPDLKxbrbT1ydyKMyXu12VuVEHoO3pF+9Oz5U3yOgFYa5TeSRqqIiVrEzMifyGHgYUUIkYGjjQB4DoyVMTKPuJvLIfCXez9u5sxN5ZNq+IG7Od8gjtVyI9g9K8ERLBKeMKxVLkFDMVlayqtggFgshK/soaeEVmZbkP5BimOTkFSk64L29nTq6nP+ZBRFmO4gTYRl2S/zvkKrTv94S7MTUddbrH0sRZ6ymZh3nGwmVqan8eyQifbwbsS5ycceb0UqSq/wi4GiNb57s8He80eMQ4k3UdSroOP7gEEO9eMZJ2OhxCNWHT7z8HxWX4UcKZW5kc3RyZWFtCmVuZG9iagoxMSAwIG9iago5ODU4CmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE2MCA+PgpzdHJlYW0KeJw9kEsSwyAMQ/ecQkfA+H+edLpK7r+tDZ1ssBiE9MB9YiKjFieCr8SHBqXDJPBsFYR7MNkRcoTkBE2GsoMkcQ0NBqXCpmOZ78mmddJKrLzRftl3NGaddIotRYd2If/n9SLco+Aa6xk8D2AxyNpKpeyZMFplpq7yqOi1H9PhPQ9Eq8Xl9Qau8NpHN6koKkvq/kR3NNj+kbf7Ht8fmWU4JAplbmRzdHJlYW0KZW5kb2JqCjI2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzkyID4+CnN0cmVhbQp4nD1SS24FMQjbzym4QKXwTXKeqd7u3X9bm8xUqgovA7YxlJcMqSU/6pKIM0x+9XJd4lHyvWxqZ+Yh7i42pvhYcl+6hthy0ZpisU8cyS/ItFRYoVbdo0PxhSgTDwAt4IEF4b4c//EXqMHXsIVyw3tkAmBK1G5AxkPRGUhZQRFh+5EV6KRQr2zh7yggV9SshaF0YogNlgApvqsNiZio2aCHhJWSqh3S8Yyk8FvBXYlhUFtb2wR4ZtAQ2d6RjREz7dEZcVkRaz896aNRMrVRGQ9NZ3zx3TJS89EV6KTSyN3KQ2fPQidgJOZJmOdwI+Ge20ELMfRxr5ZPbPeYKVaR8AU7ygEDvf3eko3Pe+AsjFzb7Ewn8NFppxwTrb4eYv2DP2xLm1zHK4dFFKi8KAh+10ETcXxYxfdko0R3tAHWIxPVaCUQDBLCzu0w8njGedneFbTm9ERoo0Qe1I4RPSiyxeWcFbCn/KzNsRyeDyZ7b7SPlMzMqIQV1HZ6qLbPYx3Ud577+vwBLgChGQplbmRzdHJlYW0KZW5kb2JqCjI3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzEgPj4Kc3RyZWFtCnicszC2UDBQMDQwUzA0N1IwNzZSMDE1UUgx5AIJgZi5XDDBHDDLGKgsByyLYEFkQSwjU1OoDhALosMQrg7BgsimAQDr5xgyCmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDcgPj4Kc3RyZWFtCnicTVG7bUQxDOvfFFzgAOtreZ4LUl32b0PJCJDCIKEvKaclFvbGSwzhB1sPvuSRVUN/Hj8x7DMsPcnk1D/muclUFL4VqpuYUBdi4f1oBLwWdC8iK8oH349lDHPO9+CjEJdgJjRgrG9JJhfVvDNkwomhjsNBm1QYd00ULK4VzTPI7VY3sjqzIGx4JRPixgBEBNkXkM1go4yxlZDFch6oCpIFWmDX6RtRi4IrlNYJdKLWxLrM4Kvn9nY3Qy/y4Ki6eH0M60uwwuileyx8rkIfzPRMO3dJI73wphMRZg8FUpmdkZU6PWJ9t0D/n2Ur+PvJz/P9CxUoXCoKZW5kc3RyZWFtCmVuZG9iagoyOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDkwID4+CnN0cmVhbQp4nE2NQRLAIAgD77wiT1BE0P90etL/X6vUDr3ATgKJFkWC9DVqSzDuuDIVa1ApmJSXwFUwXAva7qLK/jJJTJ2G03u3A4Oy8XGD0kn79nF6AKv9egbdD9IcIlgKZW5kc3RyZWFtCmVuZG9iagozMCAwIG9iago8PCAvQkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNwovU3VidHlwZSAvRm9ybSAvVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJzjMjQwUzA2NVXI5TI3NgKzcsAsI3MjIAski2BBZNMAAV8KCgplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzIwID4+CnN0cmVhbQp4nDVRu3HFMAzrNQUX8J34lTSPc6/K278NQDsVYRoEQKq8ZEq5XOqSVbLC5EeH6hRN+T5gpvwO9ZDj6B7ZIbpT1pZ7GAjLxDyljlhNlnu4BYEvDE2JuYXz9wjoKwajMBOBusXfP0CzJDBpcPBTkGutWmKJDjwsFlizK8ytGilUyFV8Oza5BwVycbPQpxyaFLfcgvBliGRHarGvy2Up8rv1CRiEFeaITxSJheeBDmYi8ScDYnv22WJXVy+qERnWSYcHUgTSbG4SMDRFsuqDG9hXxzU/T0fZwclBv4rB+DY4mS9JeV8FoRCPF/4Oz9nIsZJDJBTyfbXAiCNsgBGhT+0jEGUgNEX37plSPiZViu8ARiEcfapXMrwXkdlqhs3/GV3ZKgoGVVkfn0ZwJoNJOPNkowrTUrXTv/vc4/MHY2N6gAplbmRzdHJlYW0KZW5kb2JqCjMyIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggODAgPj4Kc3RyZWFtCnicRYy7DcAwCER7pmAEfiZmnyiVs38bIErccE+6e7g6EjJT3mGGhwSeDCyGU/EGmaNgNbhGUo2d7KOwbl91geZ6U6v19wcqT3Z2cT3Nyxn0CmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0OSA+PgpzdHJlYW0KeJwzNrRQMFAwNDAHkkaGQJaRiUKKIRdIAMTM5YIJ5oBZBkAaojgHriaHKw0AxugNJgplbmRzdHJlYW0KZW5kb2JqCjM0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNjggPj4Kc3RyZWFtCnicMzM2UzBQsDACEqamhgrmRpYKKYZcQD6IlcsFE8sBs8wszIEsIwuQlhwuQwtjMG1ibKRgZmIGZFkgMSC60gBy+BKRCmVuZHN0cmVhbQplbmRvYmoKMzUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMTcgPj4Kc3RyZWFtCnicNVJLckMxCNu/U3CBzpi/fZ50smruv62EJyuwLUBCLi9Z0kt+1CXbpcPkVx/3JbFCPo/tmsxSxfcWsxTPLa9HzxG3LQoEURM9+DInFSLUz9ToOnhhlz4DrxBOKRZ4B5MABq/hX3iUToPAOxsy3hGTkRoQJMGaS4tNSJQ9Sfwr5fWklTR0fiYrc/l7cqkUaqPJCBUgWLnYB6QrKR4kEz2JSLJyvTdWiN6QV5LHZyUmGRDdJrFNtMDj3JW0hJmYQgXmWIDVdLO6+hxMWOOwhPEqYRbVg02eNamEZrSOY2TDePfCTImFhsMSUJt9lQmql4/T3AkjpkdNdu3Csls27yFEo/kzLJTBxygkAYdOYyQK0rCAEYE5vbCKveYLORbAiGWdmiwMbWglu3qOhcDQnLOlYcbXntfz/gdFW3ujCmVuZHN0cmVhbQplbmRvYmoKMzYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNyA+PgpzdHJlYW0KeJwzNrRQMIDDFEMuABqUAuwKZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMzOCA+PgpzdHJlYW0KeJw1Ujmu3UAM630KXSCAds2c5wWpfu7fhpRfCkO0VoqajhaVafllIVUtky6/7UltiRvy98kKiROSVyXapQyRUPk8hVS/Z8u8vtacESBLlQqTk5LHJQv+DJfeLhznY2s/jyN3PXpgVYyEEgHLFBOja1k6u8Oajfw8pgE/4hFyrli3HGMVSA26cdoV70PzecgaIGaYlooKXVaJFn5B8aBHrX33WFRYINHtHElwjI1QkYB2gdpIDDmzFruoL/pZlJgJdO2LIu6iwBJJzJxiXTr6Dz50LKi/NuPLr45K+kgra0zad6NJacwik66XRW83b309uEDzLsp/Xs0gQVPWKGl80KqdYyiaGWWFdxyaDDTHHIfMEzyHMxKU9H0ofl9LJrookT8ODaF/Xx6jjJwGbwFz0Z+2igMX8dlhrxxghdLFmuR9QCoTemD6/9f4ef78Axy2gFQKZW5kc3RyZWFtCmVuZG9iagozOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0OCA+PgpzdHJlYW0KeJwtUTmSA0EIy+cVekJz0++xy5H3/+kKygGDhkMgOi1xUMZPEJYr3vLIVbTh75kYwXfBod/KdRsWORAVSNIYVE2oXbwevQd2HGYC86Q1LIMZ6wM/Ywo3enF4TMbZ7XUZNQR712tPZlAyKxdxycQFU3XYyJnDT6aMC+1czw3IuRHWZRikm5XGjIQjTSFSSKHqJqkzQZAEo6tRo40cxX7pyyOdYVUjagz7XEvb13MTzho0OxarPDmlR1ecy8nFCysH/bzNwEVUGqs8EBJwv9tD/Zzs5Dfe0rmzxfT4XnOyvDAVWPHmtRuQTbX4Ny/i+D3j6/n8A6ilWxYKZW5kc3RyZWFtCmVuZG9iagozOSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIxMCA+PgpzdHJlYW0KeJw1UMsNQzEIu2cKFqgUAoFknla9df9rbdA7YRH/QljIlAh5qcnOKelLPjpMD7Yuv7EiC611JezKmiCeK++hmbKx0djiYHAaJl6AFjdg6GmNGjV04YKmLpVCgcUl8Jl8dXvovk8ZeGoZcnYEEUPJYAlquhZNWLQ8n5BOAeL/fsPuLeShkvPKnhv5G5zt8DuzbuEnanYi0XIVMtSzNMcYCBNFHjx5RaZw4rPWd9U0EtRmC06WAa5OP4wOAGAiXlmA7K5EOUvSjqWfb7zH9w9AAFO0CmVuZHN0cmVhbQplbmRvYmoKMjMgMCBvYmoKPDwgL0Jhc2VGb250IC9EZWphVnVTYW5zIC9DaGFyUHJvY3MgMjQgMCBSCi9FbmNvZGluZyA8PAovRGlmZmVyZW5jZXMgWyAzMiAvc3BhY2UgNDYgL3BlcmlvZCA0OCAvemVybyAvb25lIC90d28gL3RocmVlIC9mb3VyIC9maXZlIC9zaXggL3NldmVuCi9laWdodCAvbmluZSA2MSAvZXF1YWwgNjggL0QgXQovVHlwZSAvRW5jb2RpbmcgPj4KL0ZpcnN0Q2hhciAwIC9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnREZXNjcmlwdG9yIDIyIDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zCi9TdWJ0eXBlIC9UeXBlMyAvVHlwZSAvRm9udCAvV2lkdGhzIDIxIDAgUiA+PgplbmRvYmoKMjIgMCBvYmoKPDwgL0FzY2VudCA5MjkgL0NhcEhlaWdodCAwIC9EZXNjZW50IC0yMzYgL0ZsYWdzIDMyCi9Gb250QkJveCBbIC0xMDIxIC00NjMgMTc5NCAxMjMzIF0gL0ZvbnROYW1lIC9EZWphVnVTYW5zIC9JdGFsaWNBbmdsZSAwCi9NYXhXaWR0aCAxMzQyIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoyMSAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzQyIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjMgNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjEyIDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTIgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwNQo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTgyIDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoyNCAwIG9iago8PCAvRCAyNSAwIFIgL2VpZ2h0IDI2IDAgUiAvZXF1YWwgMjcgMCBSIC9maXZlIDI4IDAgUiAvZm91ciAyOSAwIFIKL25pbmUgMzEgMCBSIC9vbmUgMzIgMCBSIC9wZXJpb2QgMzMgMCBSIC9zZXZlbiAzNCAwIFIgL3NpeCAzNSAwIFIKL3NwYWNlIDM2IDAgUiAvdGhyZWUgMzcgMCBSIC90d28gMzggMCBSIC96ZXJvIDM5IDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMjMgMCBSID4+CmVuZG9iago0IDAgb2JqCjw8IC9BMSA8PCAvQ0EgMCAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+Ci9BMiA8PCAvQ0EgMSAvVHlwZSAvRXh0R1N0YXRlIC9jYSAxID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9GMS1EZWphVnVTYW5zLW1pbnVzIDMwIDAgUiAvTTAgMTIgMCBSIC9NMSAxMyAwIFIgL00yIDE0IDAgUiAvTTMgMTUgMCBSCi9NNCAxNiAwIFIgL001IDE3IDAgUiAvTTYgMTggMCBSIC9NNyAxOSAwIFIgL004IDIwIDAgUiA+PgplbmRvYmoKMTIgMCBvYmoKPDwgL0JCb3ggWyAtMTAuNSAtMTAuNSAxMC41IDEwLjUgXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM2Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nDNUyOIyUPDi0jVWAKJcLmMFY4UcEA/C0QXxuJy4AI05BuYKZW5kc3RyZWFtCmVuZG9iagoxMyAwIG9iago8PCAvQkJveCBbIC0xMC41IC0xMC41IDEwLjUgMTAuNSBdIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzYKL1N1YnR5cGUgL0Zvcm0gL1R5cGUgL1hPYmplY3QgPj4Kc3RyZWFtCnicM1TI4jJQ8OLSNVYAolwuYwVjhRwQD8LRBfG4nLgAjTkG5gplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjw8IC9CQm94IFsgLTEwLjUgLTEwLjUgMTAuNSAxMC41IF0gL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNgovU3VidHlwZSAvRm9ybSAvVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJwzVMjiMlDw4tI1VgCiXC5jBWOFHBAPwtEF8bicuACNOQbmCmVuZHN0cmVhbQplbmRvYmoKMTUgMCBvYmoKPDwgL0JCb3ggWyAtMTAuNSAtMTAuNSAxMC41IDEwLjUgXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM2Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nDNUyOIyUPDi0jVWAKJcLmMFY4UcEA/C0QXxuJy4AI05BuYKZW5kc3RyZWFtCmVuZG9iagoxNiAwIG9iago8PCAvQkJveCBbIC0xMC41IC0xMC41IDEwLjUgMTAuNSBdIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzYKL1N1YnR5cGUgL0Zvcm0gL1R5cGUgL1hPYmplY3QgPj4Kc3RyZWFtCnicM1TI4jJQ8OLSNVYAolwuYwVjhRwQD8LRBfG4nLgAjTkG5gplbmRzdHJlYW0KZW5kb2JqCjE3IDAgb2JqCjw8IC9CQm94IFsgLTEwLjUgLTEwLjUgMTAuNSAxMC41IF0gL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNgovU3VidHlwZSAvRm9ybSAvVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJwzVMjiMlDw4tI1VgCiXC5jBWOFHBAPwtEF8bicuACNOQbmCmVuZHN0cmVhbQplbmRvYmoKMTggMCBvYmoKPDwgL0JCb3ggWyAtMTAuNSAtMTAuNSAxMC41IDEwLjUgXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM2Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nDNUyOIyUPDi0jVWAKJcLmMFY4UcEA/C0QXxuJy4AI05BuYKZW5kc3RyZWFtCmVuZG9iagoxOSAwIG9iago8PCAvQkJveCBbIC0xMC41IC0xMC41IDEwLjUgMTAuNSBdIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzYKL1N1YnR5cGUgL0Zvcm0gL1R5cGUgL1hPYmplY3QgPj4Kc3RyZWFtCnicM1TI4jJQ8OLSNVYAolwuYwVjhRwQD8LRBfG4nLgAjTkG5gplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9CQm94IFsgLTEwLjUgLTEwLjUgMTAuNSAxMC41IF0gL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNgovU3VidHlwZSAvRm9ybSAvVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJwzVMjiMlDw4tI1VgCiXC5jBWOFHBAPwtEF8bicuACNOQbmCmVuZHN0cmVhbQplbmRvYmoKMiAwIG9iago8PCAvQ291bnQgMSAvS2lkcyBbIDEwIDAgUiBdIC9UeXBlIC9QYWdlcyA+PgplbmRvYmoKNDAgMCBvYmoKPDwgL0NyZWF0aW9uRGF0ZSAoRDoyMDIxMDIxNjAwNDMxNCswMycwMCcpCi9DcmVhdG9yIChNYXRwbG90bGliIHYzLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZykKL1Byb2R1Y2VyIChNYXRwbG90bGliIHBkZiBiYWNrZW5kIHYzLjMuMykgPj4KZW5kb2JqCnhyZWYKMCA0MQowMDAwMDAwMDAwIDY1NTM1IGYgCjAwMDAwMDAwMTYgMDAwMDAgbiAKMDAwMDAxNzg0NyAwMDAwMCBuIAowMDAwMDE1OTk2IDAwMDAwIG4gCjAwMDAwMTYwMjggMDAwMDAgbiAKMDAwMDAxNjEyNyAwMDAwMCBuIAowMDAwMDE2MTQ4IDAwMDAwIG4gCjAwMDAwMTYxNjkgMDAwMDAgbiAKMDAwMDAwMDA2NSAwMDAwMCBuIAowMDAwMDAwMzk4IDAwMDAwIG4gCjAwMDAwMDAyMDggMDAwMDAgbiAKMDAwMDAxMDMzMSAwMDAwMCBuIAowMDAwMDE2MzE3IDAwMDAwIG4gCjAwMDAwMTY0ODcgMDAwMDAgbiAKMDAwMDAxNjY1NyAwMDAwMCBuIAowMDAwMDE2ODI3IDAwMDAwIG4gCjAwMDAwMTY5OTcgMDAwMDAgbiAKMDAwMDAxNzE2NyAwMDAwMCBuIAowMDAwMDE3MzM3IDAwMDAwIG4gCjAwMDAwMTc1MDcgMDAwMDAgbiAKMDAwMDAxNzY3NyAwMDAwMCBuIAowMDAwMDE0NzM4IDAwMDAwIG4gCjAwMDAwMTQ1MzggMDAwMDAgbiAKMDAwMDAxNDE0NSAwMDAwMCBuIAowMDAwMDE1NzkxIDAwMDAwIG4gCjAwMDAwMTAzNTIgMDAwMDAgbiAKMDAwMDAxMDU4NSAwMDAwMCBuIAowMDAwMDExMDUwIDAwMDAwIG4gCjAwMDAwMTExOTMgMDAwMDAgbiAKMDAwMDAxMTUxMyAwMDAwMCBuIAowMDAwMDExNjc1IDAwMDAwIG4gCjAwMDAwMTE4NDUgMDAwMDAgbiAKMDAwMDAxMjIzOCAwMDAwMCBuIAowMDAwMDEyMzkwIDAwMDAwIG4gCjAwMDAwMTI1MTEgMDAwMDAgbiAKMDAwMDAxMjY1MSAwMDAwMCBuIAowMDAwMDEzMDQxIDAwMDAwIG4gCjAwMDAwMTMxMzAgMDAwMDAgbiAKMDAwMDAxMzU0MSAwMDAwMCBuIAowMDAwMDEzODYyIDAwMDAwIG4gCjAwMDAwMTc5MDcgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyA0MCAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgNDEgPj4Kc3RhcnR4cmVmCjE4MDY0CiUlRU9GCg==\n", "image/svg+xml": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-02-16T00:43:14.200635\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.3.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Load data for nonlinear regression task\n", "x_data = np.load('reg_nonlin_x.npy')\n", "y_data = np.load('reg_nonlin_y.npy')\n", "\n", "# 100 points equispaced between 0 and 1\n", "x_plot = np.linspace(-0.2, 1.2, 100)\n", "\n", "# Figure to plot models on\n", "fig = plt.figure(figsize = (8, 8))\n", "\n", "# Loop over different values of D, fit a different polynomial model for each\n", "for D in range(1, 10):\n", "\n", " # Design matrix corresponding to the input data\n", " phi = np.array([[x ** d for d in range(D + 1)] for x in x_data])\n", " \n", " # Solve for maximum-likelihood weights via Moore-Penrose pseudoinverse\n", " w = np.linalg.solve((phi.T).dot(phi), (phi.T).dot(y_data))\n", " \n", " # Design matrix at the prediction points\n", " phi_pred = np.array([[x ** d for d in range(D + 1)] for x in x_plot])\n", " \n", " # Mean model predictions\n", " y_plot = phi_pred.dot(w)\n", " \n", " # Plot data and predictions separately for each model\n", " plt.subplot(3, 3, D)\n", " \n", " # Plot data and maximum a posteriori polynomial models\n", " plt.plot(x_plot, y_plot, color = 'black')\n", " plt.scatter(x_data, y_data, marker = 'x', color = 'red')\n", " \n", " # Format plots\n", " plt.text(0.1, -1.8, s = 'D = ' + str(D), color = 'black', fontsize = 12,\n", " horizontalalignment = 'right', verticalalignment = 'bottom')\n", " plt.xlim([-0.2, 1.2])\n", " plt.ylim([-2, 2])\n", " \n", " if D % 3 is not 1:\n", " plt.yticks([])\n", " else:\n", " plt.yticks(np.linspace(-2, 2, 5))\n", " \n", " if D < 7:\n", " plt.xticks([])\n", " else:\n", " plt.xticks(np.linspace(0, 1, 3))\n", " \n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Overfitted weight estimates\n", "\n", "In order to motivate the new approach, let's view effects of overfitting from another perspective by inspecting the estimated weights for each model fit:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "hide_input": true, "tags": [ "hide-input", "center-output" ] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
D\\(w_0\\)\\(w_1\\)\\(w_2\\)\\(w_3\\)\\(w_4\\)\\(w_5\\)\\(w_6\\)\\(w_7\\)\\(w_8\\)
00.230.000.000.000.000.000.000.000.00
10.52-0.780.000.000.000.000.000.000.00
20.430.11-0.920.000.000.000.000.000.00
3-0.039.83-30.0520.610.000.000.000.000.00
4-0.0711.05-36.7232.37-6.380.000.000.000.00
50.007.98-11.57-43.3487.70-40.800.000.000.00
6-0.0812.52-64.96193.72-387.43395.19-149.150.000.00
7-0.1617.64-154.12796.59-2322.573541.54-2645.98767.180.00
8-1.25102.88-1982.2617260.67-77552.28191604.56-261551.39184026.39-51922.83
\n", "
" ], "text/plain": [ "D \\(w_0\\) \\(w_1\\) \\(w_2\\) \\(w_3\\) \\(w_4\\) \\(w_5\\) \\(w_6\\) \\\n", "0 0.23 0.00 0.00 0.00 0.00 0.00 0.00 \n", "1 0.52 -0.78 0.00 0.00 0.00 0.00 0.00 \n", "2 0.43 0.11 -0.92 0.00 0.00 0.00 0.00 \n", "3 -0.03 9.83 -30.05 20.61 0.00 0.00 0.00 \n", "4 -0.07 11.05 -36.72 32.37 -6.38 0.00 0.00 \n", "5 0.00 7.98 -11.57 -43.34 87.70 -40.80 0.00 \n", "6 -0.08 12.52 -64.96 193.72 -387.43 395.19 -149.15 \n", "7 -0.16 17.64 -154.12 796.59 -2322.57 3541.54 -2645.98 \n", "8 -1.25 102.88 -1982.26 17260.67 -77552.28 191604.56 -261551.39 \n", "\n", "D \\(w_7\\) \\(w_8\\) \n", "0 0.00 0.00 \n", "1 0.00 0.00 \n", "2 0.00 0.00 \n", "3 0.00 0.00 \n", "4 0.00 0.00 \n", "5 0.00 0.00 \n", "6 0.00 0.00 \n", "7 767.18 0.00 \n", "8 184026.39 -51922.83 " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# List for storing maximum likelihood weights for each polynomial\n", "weights = []\n", "\n", "# Loop over degree of polynomial\n", "for D in range(9):\n", " \n", " # Design matrix at training data\n", " phi = np.array([[x ** d for d in range(D + 1)] for x in x_data])\n", " \n", " # Solve for the maximum likelihood weights\n", " w = np.linalg.solve((phi.T).dot(phi), (phi.T).dot(y_data))\n", " \n", " # Pad with zeroes for unused weights - this is used for the table below\n", " w = np.pad(w, (0, 9 - w.shape[0]), 'constant', constant_values = (0, 0))\n", " weights.append(w)\n", "\n", "# Row and column names for table\n", "row_names = [d for d in range(9)]\n", "column_names = ['\\(w_{}\\)'.format(d) for d in range(9)]\n", "\n", "# Create table and show it\n", "table = pandas.DataFrame(weights, row_names, column_names).round(2)\n", "table.columns.name = \"D\"\n", "table" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The estimated weights contain interesting structure: for low order polynomials, the weights take modest values. However, as the order of the polynomial increases the magnitudes of the high-order weight estimates increase dramatically and the signs oscillate. The large magnitude weights lead to polynomials with extreme gradients and the oscillating signs enable many zero crossings in the region of the observed data. The model is contorting to go through the observed data.\n", "\n", "## Regularising least squares\n", "\n", "The observation that the well-fit models have estimated weight values whose magnitudes are modest, suggests that overfitting could be mitigated by penalising large weight magnitudes. For example, a simple approach takes the sum-of-square-errors cost and adds a quadratic term $\\frac{\\alpha}{2}\\mathbf{w}^\\top\\mathbf{w}$, \n", "\n", "\\\\[C_2^{(\\text{reg})} = \\big|\\big|\\mathbf{y} - \\boldsymbol{\\Phi}\\mathbf{w}\\big|\\big|^2 +\\frac{\\alpha}{2}\\mathbf{w}^\\top\\mathbf{w}. \\\\]\n", "\n", "Here $\\alpha > 0$ is a constant. Now, if a weight magnitude becomes large, the quadratic term penalises the fit with $\\alpha$ controlling the size of the penalty. This method discourages the weights from becoming large and is known as **regularisation**. Again, there is some arbitrariness in our choice of regularisation term $-$ why use $\\mathbf{w}^\\top\\mathbf{w} = ||\\mathbf{w}||^2$ and not $||\\mathbf{w}||$ or in fact $||\\mathbf{w}||^p$ for arbitrary $p$? Regularization using different powers $p$ is known as $Lp$ regularisation. With $L1 \\implies ||\\mathbf{w}||$ and $~L2 \\implies ||\\mathbf{w}||^2$.\n", "\n", "## Interpretation as maximum a posteriori fitting\n", "\n", "Probabilistic approaches to fitting non-linear regression place a prior distribition on the weights, $p(\\mathbf{w})$. Expecting functions to typically have rather modest coefficients with small numbers of zero crossings, we place an independent Gaussian prior on the weights $p(\\mathbf{w}| \\sigma_{\\mathbf{w}}^2) = \\mathcal{N}(\\mathbf{w};\\mathbf{0},\\sigma_{\\mathbf{w}}^2 \\mathrm{I})$. Here the variance $\\sigma_{\\mathbf{w}}^2$ controls the magnitude of the polynomials we expect to see before observing the data. \n", "\n", "Here are some samples from the resulting model produced by first sampling weights from the prior, $\\mathbf{w}^{(m)} \\sim \\mathcal{N}(\\mathbf{0},\\sigma_{\\mathbf{w}}^2 \\mathrm{I})$, and then computing the resulting function $f^{(m)}(x)=\\boldsymbol{\\phi}(x)^\\top \\mathbf{w}^{(m)}$." ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "hide_input": true, "tags": [ "center-output", "hide-input" ] }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDUxMy4xNTE1NjI1IDIzMy4zMDgxMjUgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMSAwIFIgPj4Kc3RyZWFtCnicrZpNryW1EYb351ecJVnQ2C5/LoMISFmFMErWIxgmM5oPYEQI/z5PlbvPcde5MCyIhDT3TZ+2XV1+63F1x+vry2d/jdeXH67h+pr/frnG61fXz7548d9X377451efX7/9cAnoby8lyhZLLDXx55v1zySySegxFfRw/vM/l8u7C/fnN19x65eXS+5bjr2Va+Yyfl64dx5562fxzSKmULcsu3r8fBUZ5XvWkeY6XjIQa9k6q9FhUZ4e9dBkbH0f9PI5s/3l8vmz62dfxmsM12ffX6RtqZchOmjfml747LvLJ2EL4S/XZ6+vf3t2scEusaL5Ue7iR4aJzKc/OU4q53FSi1v149zFj4yTmFB+cpzi1iNdtuTHuYsfDVvawpPjNLeeJx7/mhMfGSczofrEOHF5Pj/ys3D9NJAZqdY9TpEVJ8vTb99yS83zbf72sy/TNVa9+Sf/01tcuMXXx2SfyqS8DZ+++1RPIzPPLHNFZU7YRl4W0zdpfQS9RbBV1GP8Hy+aR1fdjLd/8Gt++ukXL14//9fP3zx/9+HTt6/e/fzh+sV7pvsw4Z63mNyEb9rvT7iHTdIfm7D8eROOiRQYbsZ38TfSrmxhxjYmUmDYPy3vXNLdRqllq9WPchM/OkqNW6+3UeQ3Rrnb1TLK6mG/P0oKaatyG6XeE1sf1Kca3diPpxXj2EP0O4n96z2xf7w+JnXOTI1rhT18/enF9d/Xd9d0/fs1bgVbPReIJZaxkNmtNNGKkbaRWs/R5JwD204rR9M1D0kmS4ptDGTCOHINSUxOIcagV7NbcykxZpND60U0Wi1so+YUdEiiWbh5VZlxGpk65Z4axqDy2EZP0qvJlZyoOiTJn0fMrZlc6ghDr+4D5wildpOZaxe99xAMctQyTJaQtPYh993eVY2kTEaMPK5yrD1vgX/aPWJoWzgWL9h9kxhVj8T1WL1sjeinZnolA/blM/4Y/F+qJ7zrWL8QoTRji5kRoz0AWoaFFDMdxzkiYIU7V11qFCZ3hIDHNZqUZDrJdMQgbT2Hnmye5Ec8gkBGYtBpmN4OU4268iD2lCIpEI8wJMusYdMpOMsRBi4hWZtdTxLEIwxUjxTyvI/uxiMMbDfKV7bpkAbxCAPlr8moYnrWP2YYIg+51mbhJBHiEYa4CY+rWNi0hBxhYAoifVgYeicT9zCQcZSaavcZaWtHGKhnhKrZfUbba6PKmlBJZZJ5a0cYcNJWkj1d3f7pCAMrafiILivpPfcwBFK3jZyH6WyMPQzorbUi2XT+2MOAXloaw3SsMu1hQBcW26dOmdzDgK4QZ9MB3NIehdBZbU6STG9b36OAzt27PsSUeaJ7EJABwmTBTDhH3wss4Ru5ZLtLIeB7ENDJUszK9LL1PQjsjs59gk0STJEjCGRXamFOvrKxjyBgJSGSVqbrCvcgNIKZmZzqjX12BKGqxwBaplO6jiDUrZFLdnkn3Y8g4D38tNn0u461B4HHRpBtJxKwbRxRYLMyjWbLGvXwxFC2Nmq3jJUQbqaIXmoj300vN1fknyRgMnXcTBGVJBGxu6gvHTEgHF16n3q/2SI6k5rbX5LcfDGYpVWbJM/jZozoif1gtiCSbs4YBFsYxYIgZMJhjeiYcRuaOZLjzRvRtQBYjHl+hzkis7xuMZMSbu6IHjHpOU1S4XBHDHQ0JmTDlnFzR/QGs4ndnlQ43FENd1QMxPR+c0f0HHE1WxapcLgjesI2xabf2s0dg0JrSNHmSS4c7sgWHinVPvV6c0fd2k2LgerkwuGO6FViGTbPUW7IqW7Um96FbLqZI7K03Gz/ZH1yRxQimUa5MnncvBE5FjxBF5vx88Mb0UPp1RZF8t28UZOusQHt+pRu3oiunBDsYEUuHN6oeqLUmC7x5o3outg+TK83b1SdCmQlgjp/80b02pmPLSuXwxtVrkEs8VnFzRtVp9JY6uAaN29Unf00l1X63RtVr8HcKEMqN2+067NMvd29EV1663YfTifpHoZaWJaFudW7N6I3Nt08d8a7OQaLSLGn2MvdHFk5FdjkEe7miIw31vOh1uQ8StQMv3xz+fr6Z2JY5t4aPodhoNKRZmcMS6ElK7VnDMsy5k4+Y1htlJbqMKzgg7l2R2GVatpCEEdhVd0gtugoTPcVgyRHYRQUwYC7ozDMqGaxKrJQGCV4lGJmfsIw9niqkpvHMPKj7Ih3wjC1xjpq9BjW1SdnsThhGMHnp+0BwyCphr91j2FYNXOwvXTCMLa5xGE17YRhlDfsZHLJimF6S0AgOQzjCBCJq8cwZhyYsFnHimHolK65hVcMQwefQvMYlnTLw5fZYRj6KGFGZ6EwjmIR68jDURg6fjdSdRSW1D75paewpIetWDyEJTXqMGScIQyZ80WymrlCGHqxZHAQlrS4cMAQB2GUzxCJgoewZJ5TJ98sEKbllrOaHWBWCEMnBuEBwtBHa8EcYqUw/WnC+ZqjMHT8Kxp6rxiGDpaMOc87hiGPHmeqrRhGlvLrbs92xTD0Enq30r5iGHrT2ugoLOnmgAqzozB0FjsJe6UwbSuo2zZHYegjdbEz2IphFCaCMlJxGIZecg8lOgxD72DVZNo7hiXNomyld6UwZO4y9/NKYUn3J8+qOQ5jC6em5zzHYegVrrfVrhwmuuHYBtlxGDo/Hb07DkPvnYOoOA4TfW48jeI4DB0gGsVzGD/lagzyzGHI+P98iCuHadY17G44DkMni/YwLByGY/Hcqhn7ymGiJYaTTHYcJuQUdjqxcOEwpQg9I1THYeiDU5X518phrJwJxInHC4fhlJw5zcVXDBOsJkQHYYhYbwziKEzJBcyaGLNQWNbOap1mtGIYOw/fHbk5DMu6/UMzpl0xLGulS6F7DIOMCifOSYULhmXdKPyvOwzD5Tn7xVYdhlH/FfOsIK4YxmNO2NbEuTuG4QRqguIxjGrB5seUHIbxT8GArSKuGFb0fMNZNDsMgzAk952HFgwrujlqqOIwrBqGVasFK4ZVRXjKfHMYRtYl9r9VvhXDYBXAe0w6XjiMbGTNY97nzmFNo1NnAf2zOcxO3QOPPXOYyokSHc8cFtRacOR65jCVqWG2BRcO00Rss1YtGKYqJBddNwxZd5xVpIXDVGbxlnkLh+npv8V5zF84TGWqThxnDkPGP3M5d8NUpRQ3h2HaQKjU++QwTPU4pi+tGIZeLe8dhqken+iGaSei7Xt4xTDVU2rDd8PQAeL8gGGqk2izu7VgWNB6Jb14DFMdWB6uG2YdkLZj1YJhquP+E6sWDAtGl2Xy0IJh6JjqtM8Vw1QHga02rhimOsY3r184zDovQ6LnMNW5Y3EYpjIlbfYKFwxTXZ9cdRwWtI7twbxjmKp1tNnWWTBM9TL7IiuFmUzkPYWpnplLdBRm17Oq7ihs3n6nqrUVht6O69dWmE5+iPmb64URySiewlRnM84W37kZFhRj2pnC9JGkYbvY9cKIeJOHXhjRZuGGtKdeGInGvjcOOPXC0EcP7aEXRiJj55ZRp14YGwLAKQ+9MAISRB6bYezn1KOHMPWEWmab/dQLI7e0nXRmMLUywlEdgyHniiEmx2BBmWp2mE+tsKGAJ5MNFgTTHioOUh2Bacu1xG55sxJY1DZB3IlwITDt9LJj947XncBwJfguFk9g6laVzZccgUVtGcnewroTmDbC1TV8Jwy9pVpn52khMO3XH93JlcAUODJ7OTsC09cQWOBs8C0Epj2BgCc4ANON1OMMwsJfdpwn56PjLyyYghjF85ceu2uVyX0LgEXdPKxqnBEsanUoeSLMgmBs31rL3nlaEEwdnrS0crIiGOdBKvdjJ8xOeEBVcQiWNGggSXcIpien0WrtDsH0bMM2zI7AuIKMKd0TmLoJW8K2/kpgot39ESYK3QlM2Z1johRHYPoFRR9j6guBAdG4/czMlcAsGVOe+kJgonVcevONMJIU/GqWmSuBZT0E9O4BjAlDZcnzV9YaA+FFx19FLRsDTI6/CsggrGOc+Yvql/FvO3n82fwl2i6gTNUzf6kM9RjhLvxlR6lmbcAFv1QtIXXXBlM5Wu1a6EtTpDGKoy+V825kC32pzLNr7l0kcum4WzzTl8rEr7l3kWIUP6vlnb5UDVSt4vALHdwpvTj8Ur3sILjil+pY7OxqLfilOpuyVYdfekrlNGdUuuKX6mwAq1orfqnOnrXauuKX6gUSzg6/VM9FzCRX/FJdep8cdMcvk8k13wUzXWYLYsUv08sehgW/bFjO0tHhl+l97zst+GXL2nuBC31ZFDBJT18WNT2sO/yyKI9uGbnil+qcrHp3+KU6tmoev/CXJkPUDpDjL9WrlAcAs7xMYwwHYKpT6ZM4ADMbC3mC0wJgqnMMax7AdPuxpezguQIYuvLgbGstAEYNDlkb1w7A0GMm+M0BmLY+CtsqnQEMWdrR3VsIDJ39PbFyATCxLy7qQxdMOyh9P3itAIZOorFcB2AshGraJ+AtAGYLbHv0FwATmU7QHYCJ5kWfndiVwNAZtE3cXAhMlAPCEIdg2rjhNjtr3RlM7H3hfL+9Mpg2dLSxkh2EUYuStlD9+0ht9Eiq/nUkci+UPg9hFE8yeH7rcWqDwRm5TDM+tcEqqy2zdXtqg1GRQ2yzf7W2wah6Ie/vC5c2mMK4hImWaxsMXtIzgzgIE/XhEc03T20wyLXVWPzrSG0x6bmpOQjL2vWre5dwobCsNaTPrx1WDMv6vhAEqA7DyPagL4mbwzBtSfXyiGFqlViDa4ShNmI/K/vaCJvft0yeWBth+ukVzvDQCCv6ReEk6VMjrOpJKM33jmsjTD8sY4EPjTD90nK2LE99MG3o6ic+jsK0zityRkdhkBSPvFtbfqUw9Bqg2nGmMB4/D7DO139rH0zfXrZZU059sDhTVhyFFbXH2s0ATn0wbCrUbn3tUx9Mfac+vI3U9pu+1IyOwoq+KU/Fv4xEzhzd6wOFgQidJyyPFKZvu0J0FCZ27LPtv1OY8lW4fUl7Zqzf/Z729FnbNx/7vNNf/Ue/D16vPX1d91t3Drae+XVwtG+DX94+t+v2BS61iJPy/C3Hsflb/d7uH+/f/Pru/dtXz998uH54/vaHNy++u37/0/u31x9+evX+p+NjvK8v/wfBqgoFCmVuZHN0cmVhbQplbmRvYmoKMTEgMCBvYmoKMzgzOQplbmRvYmoKMTYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA5MiA+PgpzdHJlYW0KeJw9jLENwDAIBHum+AUiYYxt2CdK5ezf5i0naeD0D9fSoDiscXZVNB84i3x4S/WEjcSUppVHU5zd2hYOK4MUu9gWFl5hEaTyapjxeVPVwJJSlOXN+n93PcerG7oKZW5kc3RyZWFtCmVuZG9iagoxNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzOSA+PgpzdHJlYW0KeJw9j7ENxTAIRHumuAWQABsbz5PoV/77t8FxkgLxdKA78GEQsNUs6WhS4LXjVLIaYBf8yaSB1QTaLaEVaF1KKA5aOusIRNsW9ekHfa6TeORSsaRqL7W+KWK5O/SO0W1awKNnTvau0Obgck9GQSZOylPWoZM0fTaZB9QiyWU82vvQ/P6Z9LsAu7wt2wplbmRzdHJlYW0KZW5kb2JqCjE0IDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2Fucy1PYmxpcXVlIC9DaGFyUHJvY3MgMTUgMCBSCi9FbmNvZGluZyA8PCAvRGlmZmVyZW5jZXMgWyAxMjAgL3ggL3kgXSAvVHlwZSAvRW5jb2RpbmcgPj4gL0ZpcnN0Q2hhciAwCi9Gb250QkJveCBbIC0xMDE2IC0zNTEgMTY2MCAxMDY4IF0gL0ZvbnREZXNjcmlwdG9yIDEzIDAgUgovRm9udE1hdHJpeCBbIDAuMDAxIDAgMCAwLjAwMSAwIDAgXSAvTGFzdENoYXIgMjU1IC9OYW1lIC9EZWphVnVTYW5zLU9ibGlxdWUKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMTIgMCBSID4+CmVuZG9iagoxMyAwIG9iago8PCAvQXNjZW50IDkyOSAvQ2FwSGVpZ2h0IDAgL0Rlc2NlbnQgLTIzNiAvRmxhZ3MgOTYKL0ZvbnRCQm94IFsgLTEwMTYgLTM1MSAxNjYwIDEwNjggXSAvRm9udE5hbWUgL0RlamFWdVNhbnMtT2JsaXF1ZQovSXRhbGljQW5nbGUgMCAvTWF4V2lkdGggMTM1MCAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTIgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM1MCA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDI4IDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxNyA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjE3IDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDgKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk5NSA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMTUgMCBvYmoKPDwgL3ggMTYgMCBSIC95IDE3IDAgUiA+PgplbmRvYmoKMjIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjUgPj4Kc3RyZWFtCnicRY87EgMhDEN7TqEjgH/AeTaTir1/G8s7SRosjCU/ois69srDY2PKxmu0sSfCFu5SOg2nqYyviqdnXaDLYTJTb1zNXGCqsMhuTrH6GHyh8uzmhK9VnhjCl0wJDTCVO7mH9fpRnJZ8JLsLguqUjcrCMEfS90BMTZunhYH8jy95akFQmeaNa5aVR2sVUzRnmCpbC4L1gaA6pfoD0/9Mp70/3PQ9gAplbmRzdHJlYW0KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzA0ID4+CnN0cmVhbQp4nD2SO5LDMAxDe52CF8iM+JPk82Qnlff+7T4yyVaASYkAKC91mbKmPCBpJgn/0eHhYjvld9iezczAtUQvE8spz6ErxNxF+bKZjbqyOsWqwzCdW/SonIuGTZOa5ypLGbcLnsO1ieeWfcQPNzSoB3WNS8IN3dVoWQrNcHX/O71H2Xc1PBebVOrUF48XURXm+SFPoofpSuJ8PCghXHswRhYS5FPRQI6zXK3yXkL2DrcassJBaknnsyc82HV6Ty5uF80QD2S5VPhOUezt0DO+7EoJPRK24VjufTuasekamzjsfu9G1sqMrmghfshXJ+slYNxTJkUSZE62WG6L1Z7uoSimc4ZzGSDq2YqGUuZiV6t/DDtvLC/ZLMiUzAsyRqdNnjh4yH6NmvR5led4/QFs83M7CmVuZHN0cmVhbQplbmRvYmoKMjQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMjcgPj4Kc3RyZWFtCnicNU87sgMhDOs5hS6QGYxtYM+zmVQv92+fZLINEv5I8vRERyZe5sgIrNnxthYZiBn4FlPxrz3tw4TqPbiHCOXiQphhJJw167ibp+PFv13lM9bBuw2+YpYXBLYwk/WVxZnLdsFYGidxTrIbY9dEbGNd6+kU1hFMKAMhne0wJcgcFSl9sqOMOTpO5InnYqrFLr/vYX3BpjGiwhxXBU/QZFCWPe8moB0X9N/Vjd9JNIteAjKRYGGdJObOWU741WtHx1GLIjEnpBnkMhHSnK5iCqEJxTo7CioVBZfqc8rdPv9oXVtNCmVuZHN0cmVhbQplbmRvYmoKMjUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDUgPj4Kc3RyZWFtCnicRVC7jUMxDOs9BRcIYP0se553SJXbvz1KRnCFIVo/kloSmIjASwyxlG/iR0ZBPQu/F4XiM8TPF4VBzoSkQJz1GRCZeIbaRm7odnDOvMMzjDkCF8VacKbTmfZc2OScBycQzm2U8YxCuklUFXFUn3FM8aqyz43XgaW1bLPTkewhjYRLSSUml35TKv+0KVsq6NpFE7BI5IGTTTThLD9DkmLMoJRR9zC1jvRxspFHddDJ2Zw5LZnZ7qftTHwPWCaZUeUpnecyPiep81xOfe6zHdHkoqVV+5z93pGW8iK126HV6VclUZmN1aeQuDz/jJ/x/gOOoFk+CmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMzMgPj4Kc3RyZWFtCnicTY9BEsMwCAPvfoWegLEB8550ekr+fy2QNu4F7YyAkYYwCDxiDOswJbx6++FVpEtwNo75JRlFPAhqC9wXVAVHY4qd+Njdoeyl4ukUTYvrEXPTtKR0N1Eqbb2dyPjAfZ/eH1W2JJ2CHlvqhC7RJPJFAnPYVDDP6sZLS4+n7dneH2Y+M9cKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0NyA+PgpzdHJlYW0KeJxNUbttRDEM698UXOAA62t5ngtSXfZvQ8kIkMIgoS8ppyUW9sZLDOEHWw++5JFVQ38ePzHsMyw9yeTUP+a5yVQUvhWqm5hQF2Lh/WgEvBZ0LyIrygffj2UMc8734KMQl2AmNGCsb0kmF9W8M2TCiaGOw0GbVBh3TRQsrhXNM8jtVjeyOrMgbHglE+LGAEQE2ReQzWCjjLGVkMVyHqgKkgVaYNfpG1GLgiuU1gl0otbEuszgq+f2djdDL/LgqLp4fQzrS7DC6KV7LHyuQh/M9Ew7d0kjvfCmExFmDwVSmZ2RlTo9Yn23QP+fZSv4+8nP8/0LFShcKgplbmRzdHJlYW0KZW5kb2JqCjI4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNjggPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZcQL6piblCLhdIDMTKAbMMgLQlnIKIW0I0QZSCWBClZiZmEEk4AyKXBgDJtBXlCmVuZHN0cmVhbQplbmRvYmoKMjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA0NSA+PgpzdHJlYW0KeJwzMrdQMFCwNAEShhYmCuZmBgophlyWEFYuF0wsB8wC0ZZwCiKeBgCffQy1CmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNTUgPj4Kc3RyZWFtCnicRZFLkgMgCET3noIjgPzkPJmaVXL/7TSYTDZ2l6j9hEojphIs5xR5MP3I8s1ktum1HKudjQKKIhTM5Cr0WIHVnSnizLVEtfWxMnLc6R2D4g3nrpxUsrhRxjqqOhU4pufK+qru/Lgsyr4jhzIFbNY5DjZw5bZhjBOjzVZ3h/tEkKeTqaPidpBs+IOTxr7K1RW4Tjb76iUYB4J+oQlM8k2gdYZA4+YpenIJ9vFxu/NAsLe8CaRsCOTIEIwOQbtOrn9x6/ze/zrDnefaDFeOd/E7TGu74y8xyYq5gEXuFNTzPRet6wwd78mZY3LTfUPnXLDL3UGmz/wf6/cPUIpmiAplbmRzdHJlYW0KZW5kb2JqCjMxIDAgb2JqCjw8IC9CQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM3Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nOMyNDBTMDY1VcjlMjc2ArNywCwjcyMgCySLYEFk0wABXwoKCmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNjEgPj4Kc3RyZWFtCnicRZBLEsMgDEP3nEJH8EcGfJ50ukrvv60hTbOAp7FABncnBKm1BRPRBS9tS7oLPlsJzsZ46DZuNRLkBHWAVqTjaJRSfbnFaZV08Wg2cysLrRMdZg56lKMZoBA6Fd7touRypu7O+Udw9V/1R7HunM3EwGTlDoRm9SnufJsdUV3dZH/SY27Wa38V9qqwtKyl5YTbzl0zoATuqRzt/QWpczqECmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTQgPj4Kc3RyZWFtCnicPVC7EUMxCOs9BQvkznztN8/Lpcv+bSScpEI2QhKUmkzJlIc6ypKsKU8dPktih7yH5W5kNiUqRS+TsCX30ArxfYnmFPfd1ZazQzSXaDl+CzMqqhsd00s2mnAqE7qg3MMz+g1tdANWhx6xWyDQpGDXtiByxw8YDMGZE4siDEpNBv+tcvdS3O89HG+iiJR08K755fTLzy28Tj2ORLq9+YprcaY6CkRwRmryinRhxbLIQ6TVBDU9A2u1AK7eevk3aEd0GYDsE4njNKUcQ//WuMfrA4eKUvQKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDgwID4+CnN0cmVhbQp4nEWMuw3AMAhEe6ZgBH4mZp8olbN/GyBK3HBPunu4OhIyU95hhocEngwshlPxBpmjYDW4RlKNneyjsG5fdYHmelOr9fcHKk92dnE9zcsZ9AplbmRzdHJlYW0KZW5kb2JqCjM1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjM2ID4+CnN0cmVhbQp4nE1QS25EIQzbc4pc4EkkIQHOQ9VV5/7bscNU7SqGGH9ID+myVR7rU2J1iezypU2XyjJ5FajlT9v/UQwCbv/QyEG0t4ydYuYS1sXCJDzlNCMbJ9csH487TxtmhcbEjeOdLhlgnxYBNVuVzYE5bTo3QLqQGreqs95kUAwi6kLNB5MunKfRl4g5nqhgSncmtZAbXD7VoQNxWr0KuWOLk2/EHFmhwGHQTHHWXwHWqMmyWcggSYYhzn2je5QKjajKeSsVwg+ToRH1htWgBpW5haKp5ZL8HdoCMAW2jHXpDEqBqgDB3yqnfb8BJI1dUwplbmRzdHJlYW0KZW5kb2JqCjM2IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDkgPj4Kc3RyZWFtCnicMza0UDBQMDQwB5JGhkCWkYlCiiEXSADEzOWCCeaAWQZAGqI4B64mhysNAMboDSYKZW5kc3RyZWFtCmVuZG9iagozNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE1NyA+PgpzdHJlYW0KeJxFkLkRQzEIRHNVQQkSsAjqscfRd/+pF/lKtG8ALYevJVOqHyciptzXaPQweQ6fTSVWLNgmtpMachsWQUoxmHhOMaujt6GZh9TruKiquHVmldNpy8rFf/NoVzOTPcI16ifwTej4nzy0qehboK8LlH1AtTidSVAxfa9igaOcdn8inBjgPhlHmSkjcWJuCuz3GQBmvle4xuMF3QE3eQplbmRzdHJlYW0KZW5kb2JqCjM4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzMyID4+CnN0cmVhbQp4nC1SOY4kMQzL/Qp+YADr8vGeHkzU+/90SVUFBapsyzzkcsNEJX4skNtRa+LXRmagwvCvq8yF70jbyDqIa8hFXMmWwmdELOQxxDzEgu/b+Bke+azMybMHxi/Z9xlW7KkJy0LGizO0wyqOwyrIsWDrIqp7eFOkw6kk2OOL/z7FcxeCFr4jaMAv+eerI3i+pEXaPWbbtFsPlmlHlRSWg+1pzsvkS+ssV8fj+SDZ3hU7QmpXgKIwd8Z5Lo4ybWVEa2Fng6TGxfbm2I+lBF3oxmWkOAL5mSrCA0qazGyiIP7I6SGnMhCmrulKJ7dRFXfqyVyzubydSTJb90WKzRTO68KZ9XeYMqvNO3mWE6VORfgZe7YEDZ3j6tlrmYVGtznBKyV8NnZ6cvK9mlkPyalISBXTugpOo8gUS9iW+JqKmtLUy/Dfl/cZf/8BM+J8AQplbmRzdHJlYW0KZW5kb2JqCjM5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNjggPj4Kc3RyZWFtCnicMzM2UzBQsDACEqamhgrmRpYKKYZcQD6IlcsFE8sBs8wszIEsIwuQlhwuQwtjMG1ibKRgZmIGZFkgMSC60gBy+BKRCmVuZHN0cmVhbQplbmRvYmoKNDAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMTcgPj4Kc3RyZWFtCnicNVJLckMxCNu/U3CBzpi/fZ50smruv62EJyuwLUBCLi9Z0kt+1CXbpcPkVx/3JbFCPo/tmsxSxfcWsxTPLa9HzxG3LQoEURM9+DInFSLUz9ToOnhhlz4DrxBOKRZ4B5MABq/hX3iUToPAOxsy3hGTkRoQJMGaS4tNSJQ9Sfwr5fWklTR0fiYrc/l7cqkUaqPJCBUgWLnYB6QrKR4kEz2JSLJyvTdWiN6QV5LHZyUmGRDdJrFNtMDj3JW0hJmYQgXmWIDVdLO6+hxMWOOwhPEqYRbVg02eNamEZrSOY2TDePfCTImFhsMSUJt9lQmql4/T3AkjpkdNdu3Csls27yFEo/kzLJTBxygkAYdOYyQK0rCAEYE5vbCKveYLORbAiGWdmiwMbWglu3qOhcDQnLOlYcbXntfz/gdFW3ujCmVuZHN0cmVhbQplbmRvYmoKNDEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNyA+PgpzdHJlYW0KeJwzNrRQMIDDFEMuABqUAuwKZW5kc3RyZWFtCmVuZG9iago0MiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMzOCA+PgpzdHJlYW0KeJw1Ujmu3UAM630KXSCAds2c5wWpfu7fhpRfCkO0VoqajhaVafllIVUtky6/7UltiRvy98kKiROSVyXapQyRUPk8hVS/Z8u8vtacESBLlQqTk5LHJQv+DJfeLhznY2s/jyN3PXpgVYyEEgHLFBOja1k6u8Oajfw8pgE/4hFyrli3HGMVSA26cdoV70PzecgaIGaYlooKXVaJFn5B8aBHrX33WFRYINHtHElwjI1QkYB2gdpIDDmzFruoL/pZlJgJdO2LIu6iwBJJzJxiXTr6Dz50LKi/NuPLr45K+kgra0zad6NJacwik66XRW83b309uEDzLsp/Xs0gQVPWKGl80KqdYyiaGWWFdxyaDDTHHIfMEzyHMxKU9H0ofl9LJrookT8ODaF/Xx6jjJwGbwFz0Z+2igMX8dlhrxxghdLFmuR9QCoTemD6/9f4ef78Axy2gFQKZW5kc3RyZWFtCmVuZG9iago0MyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0OCA+PgpzdHJlYW0KeJwtUTmSA0EIy+cVekJz0++xy5H3/+kKygGDhkMgOi1xUMZPEJYr3vLIVbTh75kYwXfBod/KdRsWORAVSNIYVE2oXbwevQd2HGYC86Q1LIMZ6wM/Ywo3enF4TMbZ7XUZNQR712tPZlAyKxdxycQFU3XYyJnDT6aMC+1czw3IuRHWZRikm5XGjIQjTSFSSKHqJqkzQZAEo6tRo40cxX7pyyOdYVUjagz7XEvb13MTzho0OxarPDmlR1ecy8nFCysH/bzNwEVUGqs8EBJwv9tD/Zzs5Dfe0rmzxfT4XnOyvDAVWPHmtRuQTbX4Ny/i+D3j6/n8A6ilWxYKZW5kc3RyZWFtCmVuZG9iago0NCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzOCA+PgpzdHJlYW0KeJw9j0EOAzEIA+95hT8QKXZCWN6zVU/b/19Lmt1e0AiMMRZCQ2+oag6bgg3Hi6VLqNbwKYqJSg7ImWAOpaTSHWeRemI4GNwetBvO4rHp+hG7klZ90OZGuiVogkfsU2nclnETxAM1Beop6lyjvBC5n6lX2DSS3bSykms4pt+956nr/9NV3l9f3y6MCmVuZHN0cmVhbQplbmRvYmoKNDUgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTAgPj4Kc3RyZWFtCnicNVDLDUMxCLtnChaoFAKBZJ5WvXX/a23QO2ER/0JYyJQIeanJzinpSz46TA+2Lr+xIgutdSXsypognivvoZmysdHY4mBwGiZegBY3YOhpjRo1dOGCpi6VQoHFJfCZfHV76L5PGXhqGXJ2BBFDyWAJaroWTVi0PJ+QTgHi/37D7i3koZLzyp4b+Ruc7fA7s27hJ2p2ItFyFTLUszTHGAgTRR48eUWmcOKz1nfVNBLUZgtOlgGuTj+MDgBgIl5ZgOyuRDlL0o6ln2+8x/cPQABTtAplbmRzdHJlYW0KZW5kb2JqCjIwIDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2FucyAvQ2hhclByb2NzIDIxIDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDQ2IC9wZXJpb2QgNDggL3plcm8gL29uZSAvdHdvIC90aHJlZSA1MyAvZml2ZSAvc2l4IC9zZXZlbiA4MAovUCA5NyAvYSAxMDAgL2QgL2UgL2YgMTA1IC9pIDEwOCAvbCAvbSAvbiAvbyAvcCAxMTQgL3IgL3MgMTIxIC95IF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAxOSAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxOCAwIFIgPj4KZW5kb2JqCjE5IDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvRGVqYVZ1U2FucyAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTM0MiAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTggMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMjEgMCBvYmoKPDwgL1AgMjIgMCBSIC9hIDIzIDAgUiAvZCAyNCAwIFIgL2UgMjUgMCBSIC9mIDI2IDAgUiAvZml2ZSAyNyAwIFIKL2kgMjggMCBSIC9sIDI5IDAgUiAvbSAzMCAwIFIgL24gMzIgMCBSIC9vIDMzIDAgUiAvb25lIDM0IDAgUiAvcCAzNSAwIFIKL3BlcmlvZCAzNiAwIFIgL3IgMzcgMCBSIC9zIDM4IDAgUiAvc2V2ZW4gMzkgMCBSIC9zaXggNDAgMCBSIC9zcGFjZSA0MSAwIFIKL3RocmVlIDQyIDAgUiAvdHdvIDQzIDAgUiAveSA0NCAwIFIgL3plcm8gNDUgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAyMCAwIFIgL0YyIDE0IDAgUiA+PgplbmRvYmoKNCAwIG9iago8PCAvQTEgPDwgL0NBIDAgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PgovQTIgPDwgL0NBIDEgL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMSA+PiA+PgplbmRvYmoKNSAwIG9iago8PCA+PgplbmRvYmoKNiAwIG9iago8PCA+PgplbmRvYmoKNyAwIG9iago8PCAvRjEtRGVqYVZ1U2Fucy1taW51cyAzMSAwIFIgPj4KZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMCAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjQ2IDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMTAyMTYwMDQzMzUrMDMnMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My4zLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My4zLjMpID4+CmVuZG9iagp4cmVmCjAgNDcKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMTQ2MjYgMDAwMDAgbiAKMDAwMDAxNDM5MyAwMDAwMCBuIAowMDAwMDE0NDM2IDAwMDAwIG4gCjAwMDAwMTQ1MzUgMDAwMDAgbiAKMDAwMDAxNDU1NiAwMDAwMCBuIAowMDAwMDE0NTc3IDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDQwMCAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDQzMTQgMDAwMDAgbiAKMDAwMDAwNTIzOCAwMDAwMCBuIAowMDAwMDA1MDMwIDAwMDAwIG4gCjAwMDAwMDQ3MTEgMDAwMDAgbiAKMDAwMDAwNjI5MSAwMDAwMCBuIAowMDAwMDA0MzM1IDAwMDAwIG4gCjAwMDAwMDQ0OTkgMDAwMDAgbiAKMDAwMDAxMzA1OSAwMDAwMCBuIAowMDAwMDEyODU5IDAwMDAwIG4gCjAwMDAwMTI0MzAgMDAwMDAgbiAKMDAwMDAxNDExMiAwMDAwMCBuIAowMDAwMDA2MzMzIDAwMDAwIG4gCjAwMDAwMDY1NzEgMDAwMDAgbiAKMDAwMDAwNjk0OCAwMDAwMCBuIAowMDAwMDA3MjQ4IDAwMDAwIG4gCjAwMDAwMDc1NjYgMDAwMDAgbiAKMDAwMDAwNzc3MiAwMDAwMCBuIAowMDAwMDA4MDkyIDAwMDAwIG4gCjAwMDAwMDgyMzIgMDAwMDAgbiAKMDAwMDAwODM0OSAwMDAwMCBuIAowMDAwMDA4Njc3IDAwMDAwIG4gCjAwMDAwMDg4NDcgMDAwMDAgbiAKMDAwMDAwOTA4MSAwMDAwMCBuIAowMDAwMDA5MzY4IDAwMDAwIG4gCjAwMDAwMDk1MjAgMDAwMDAgbiAKMDAwMDAwOTgyOSAwMDAwMCBuIAowMDAwMDA5OTUwIDAwMDAwIG4gCjAwMDAwMTAxODAgMDAwMDAgbiAKMDAwMDAxMDU4NSAwMDAwMCBuIAowMDAwMDEwNzI1IDAwMDAwIG4gCjAwMDAwMTExMTUgMDAwMDAgbiAKMDAwMDAxMTIwNCAwMDAwMCBuIAowMDAwMDExNjE1IDAwMDAwIG4gCjAwMDAwMTE5MzYgMDAwMDAgbiAKMDAwMDAxMjE0NyAwMDAwMCBuIAowMDAwMDE0Njg2IDAwMDAwIG4gCnRyYWlsZXIKPDwgL0luZm8gNDYgMCBSIC9Sb290IDEgMCBSIC9TaXplIDQ3ID4+CnN0YXJ0eHJlZgoxNDg0MwolJUVPRgo=\n", "image/svg+xml": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-02-16T00:43:35.074582\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.3.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Order of polynomial\n", "D = 9\n", "\n", "# Prior weight variance\n", "var_w = 1\n", "\n", "# Number of samples\n", "M = 5\n", "\n", "# 100 points equispaced between 0 and 1\n", "x_plot = np.linspace(0, 1, 100) \n", "\n", "# Design matrix at training points\n", "phi_pred = np.array([[x ** d for d in range(D + 1)] for x in x_plot]) \n", "\n", "# Figure on which to plot sampled functions\n", "plt.figure(figsize=(8, 3))\n", "\n", "# Sample M different models\n", "for m in range(1, M):\n", " \n", " # Sample weights from prior\n", " w = np.random.normal(0, var_w ** 0.5, D + 1)\n", "\n", " # Plot function at input locations\n", " y_plot = phi_pred.dot(w)\n", "\n", " # Plot sampled model\n", " plt.plot(x_plot, y_plot, color='k')\n", "\n", "# Format plot\n", "plt.xticks(np.linspace(0, 1, 5))\n", "plt.yticks(np.linspace(-6, 6, 5))\n", "plt.xlim([0, 1])\n", "plt.xlabel('$x$', fontsize=16)\n", "plt.ylabel('$y$', fontsize=16)\n", "plt.title('Polynomials sampled from prior', fontsize=18)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "hide_input": true }, "source": [ "We can now apply probabilistic inference for the weights using Bayes' rule to find the maximum a posteriori setting of the weights given the observed data:\n", "\n", "$$\\begin{align}\n", "\\mathbf{w}^{\\text{MAP}} & = \\underset{\\mathbf{w}}{\\mathrm{arg\\,max}} \\; p(\\mathbf{w} | \\{x_n,y_n\\}_{n=1}^N,\\sigma_y^2,\\sigma_{\\mathbf{w}}^2)\\\\\n", "& = \\underset{\\mathbf{w}}{\\mathrm{arg\\,min}} \\; (\\mathbf{y} - \\mathbf{X}\\mathbf{w})^\\top (\\mathbf{y} - \\mathbf{X}\\mathbf{w}) - \\alpha \\mathbf{w}^\\top \\mathbf{w} \\;\\; \\text{where} \\;\\;\\alpha = \\frac{\\sigma_y^2}{\\sigma_\\mathbf{w}^2}\n", "\\end{align}$$\n", "\n", "
\n", "Details: Deriving the MAP weights\n", "
\n", "\n", "First we apply Bayes' rule\n", "\n", "$$\\begin{align}\n", "p(\\mathbf{w} | \\{x_n,y_n\\}_{n=1}^N,\\sigma_y^2,\\sigma_{\\mathbf{w}}^2) & = \\frac{ p(\\mathbf{w}| \\sigma_{\\mathbf{w}}^2) p(\\mathbf{y}\\mid\\mathbf{X}, \\mathbf{w}, \\sigma_y^2) }{p(\\mathbf{y}| \\sigma_y^2,\\sigma_{\\mathbf{w}}^2)} \\propto p(\\mathbf{w}| \\sigma_{\\mathbf{w}}^2) p(\\mathbf{y}\\mid\\mathbf{X}, \\mathbf{w}, \\sigma_y^2).\n", "\\end{align}$$\n", "\n", "Next we substitute in for the likelihood and prior,\n", "\n", "$$\\begin{align}\n", "p(\\mathbf{w} | \\{x_n,y_n\\}_{n=1}^N,\\sigma_y^2,\\sigma_{\\mathbf{w}}^2) = \\frac{1}{(2\\pi \\sigma_\\mathbf{w}^2)}\\text{exp}\\big(-\\frac{1}{2\\sigma_\\mathbf{w}^2}\\mathbf{w}^\\top \\mathbf{w} \\big) \\times \\frac{1}{(2\\pi \\sigma_y^2)^{N/2}}\\text{exp}\\big(-\\frac{1}{2\\sigma_y^2}(\\mathbf{y} - \\mathbf{X}\\mathbf{w})^\\top (\\mathbf{y} - \\mathbf{X}\\mathbf{w})\\big).\n", " \\end{align}$$\n", "\n", "Now pulling the prior and likelihood terms into a single exponential we have,\n", "\n", "$$\\begin{align}\n", "p(\\mathbf{w} | \\{x_n,y_n\\}_{n=1}^N,\\sigma_y^2,\\sigma_{\\mathbf{w}}^2) & = \\frac{1}{(2\\pi \\sigma_\\mathbf{w}^2) (2\\pi \\sigma_y^2)^{N/2}}\\text{exp}\\big(-\\frac{1}{2\\sigma_\\mathbf{w}^2}\\mathbf{w}^\\top \\mathbf{w} -\\frac{1}{2\\sigma_y^2}(\\mathbf{y} - \\mathbf{X}\\mathbf{w})^\\top (\\mathbf{y} - \\mathbf{X}\\mathbf{w})\\big)\n", "\\end{align}$$\n", "\n", "Taking logs and combining terms that do not depend on $\\mathbf{w}$ into a constant,\n", "\n", "$$\\begin{align}\n", "\\log p(\\mathbf{w} | \\{x_n,y_n\\}_{n=1}^N,\\sigma_y^2,\\sigma_{\\mathbf{w}}^2) & = -\\frac{1}{2\\sigma_\\mathbf{w}^2}\\mathbf{w}^\\top \\mathbf{w} -\\frac{1}{2\\sigma_y^2}(\\mathbf{y} - \\mathbf{X}\\mathbf{w})^\\top (\\mathbf{y} - \\mathbf{X}\\mathbf{w}) + \\text{const.}.\n", "\\end{align}$$\n", "\n", "Now we see that maximising $p(\\mathbf{w} | \\{x_n,y_n\\}_{n=1}^N,\\sigma_y^2,\\sigma_{\\mathbf{w}}^2)$ is the same as minimising $\\frac{\\sigma_y^2}{\\sigma_\\mathbf{w}^2}\\mathbf{w}^\\top \\mathbf{w} +(\\mathbf{y} - \\mathbf{X}\\mathbf{w})^\\top (\\mathbf{y} - \\mathbf{X}\\mathbf{w})$.\n", "\n", "
\n", "
\n", "
\n", "\n", "So, the simple quadratic regularisation can be interpreted as arising from a Gaussian prior on the weights and $\\alpha$ is the ratio of the observation noise variance to the prior weight variance. Again the probabilistic approach reveals the hidden assumptions and makes it simple to assess their suitability. \n", "\n", "Let's now optimise the new cost. The use of a quadratic penalty, or equivalently a Gaussian prior on the weights, leads to an analytic solution.\n", "\n", "$$\\begin{align}\\frac{\\partial\\mathcal{L}}{\\partial \\mathbf{w}} = - \\boldsymbol{\\Phi}^\\top(\\mathbf{y} - \\boldsymbol{\\Phi}\\mathbf{w}) + \\alpha\\mathbf{w} = 0\\\\\n", "~\\\\\n", "- \\boldsymbol{\\Phi}^\\top\\mathbf{y} + \\boldsymbol{\\Phi}^\\top\\boldsymbol{\\Phi}\\mathbf{w} + \\alpha \\mathbf{I}\\mathbf{w} = 0\\\\\n", "~\\\\\n", "\\implies \\boxed{\\mathbf{w} = (\\boldsymbol{\\Phi}^\\top\\boldsymbol{\\Phi} + \\alpha\\mathbf{I})^{-1}\\boldsymbol{\\Phi}^\\top\\mathbf{y}}\\\\\n", "\\end{align}$$\n", "\n", "Comparing this with the unregularized expression for $\\mathbf{w}$:\n", "\n", "$$\\begin{align}\n", "\\mathbf{w} &= (\\boldsymbol{\\Phi}^\\top\\boldsymbol{\\Phi})^{-1}\\boldsymbol{\\Phi}^\\top\\mathbf{y},~\\text{unregularized},\\\\\n", "~\\\\\n", "\\mathbf{w} &= (\\boldsymbol{\\Phi}^\\top\\boldsymbol{\\Phi} + \\alpha \\mathbf{I})^{-1}\\boldsymbol{\\Phi}^\\top\\mathbf{y},~\\text{regularized},\\\\\n", "\\end{align}$$\n", "\n", "we see that the only difference is the added $\\alpha\\mathbf{I}$ term. Since this is inside the $(\\cdot)^{-1}$ matrix inverse we can intuitively see that its effect is to reduce the magnitude of the matrix elements of $(\\boldsymbol{\\Phi}^\\top\\boldsymbol{\\Phi} + \\alpha\\mathbf{I})^{-1}$ and hence those of $\\mathbf{w}$. Consider for example the limiting case $\\boldsymbol{\\Phi}^\\top\\boldsymbol{\\Phi} << \\alpha\\mathbf{I}$. Then $(\\boldsymbol{\\Phi}^\\top\\boldsymbol{\\Phi} + \\alpha\\mathbf{I})^{-1} \\approx \\alpha^{-1}\\mathbf{I}$ and increasing $\\alpha$ results in smaller $\\mathbf{w}$. Note that the value of $\\alpha$ is arbitrarily chosen (we will later address this point). \n", "\n", "Let's have a look on how regularization affects the model. We'll assume small observations noise $\\sigma_{y} = 0.01$ and a prior weight standard deviation of $\\sigma_{\\mathbf{w}} = 1$ so that $\\alpha = 10^{-4}$:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "hide_input": true, "tags": [ "center-output", "hide-input" ] }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDU2OC43ODQzNzUgNTY4LjI2ODc1IF0gL1BhcmVudCAyIDAgUiAvUmVzb3VyY2VzIDggMCBSCi9UeXBlIC9QYWdlID4+CmVuZG9iago5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTEgMCBSID4+CnN0cmVhbQp4nL2dza8st3HF9/ev6GWy0IjfH4ssbCgRYCAL20KyNhRZsaCnwBYc//s551QPye47c+/rpxc5CHxfeabJbpLFOlW/5vjth5cvf+O373/e3PYD/v8fm9++3r786rv//cu33/3h699u3/784mD/8JJLu9WWYs3454/rP/lnKA1//YjPrv/675eXn15wdXzja1z4+5eX0G496luxx1uMLaeAi/uebq0m5/1q/3G151xvoXq78LzMwYz2/vzy1+1BI76UUjdfMv6XXmKGKeKfEf+8/2f723fbf24/bV/+JvB5+M3hbv72Pf74YfO3rEfDR/GHr/c//vb9y19f7E+/+ZBvtSTfgvcxbCn1W42ut1xyTdu3H7Yv/91tX/3P+PwXqd3Q65azz2Er4ZZDd7HX1MOjT/t0K64753rMFf+st5RKCD7lUB99Hhd3DrebUsLHg7u5GHNJ3edHnfEO12shtOhbK/x2CKH3WrKLD3uDz0cfeo69t7D5js/jNlKv/lHvY7r17mqsoYbCT8foPW6+pIdXj+4WEv7XEkPDc7p5XNuF0sujO2VXXAutlxBr0Ze796Hh4709unpJN+fwYV9ji+yMy652F1Ipy8d///L77a/3qeA2mwXu1jALOJdheTaVMQ/SK/OPLy+/RRf+sU+YL9gTzNsttn6zyan5+e2Hl99+s335b7yt7Zs/v7RbrK2jt1xz3/zXyz+Ff96++eHlX7/BdTCkdk/jD3wbX/3iq+9++NN//P2Pf/rp5y8+/OWnv/+8386Ler/0Gr28xaCVs/Z6MT/rdcJDzujTu931n7O7Nd+iy8G1Y3eneXT30Cmfb1mfwxp27b7W2TvH3m3o3bmp7MsttFfjuZjfbCr7cMNKuw/qeBAPm5ru69DU6tXeairHG57AnBtsgzP3s7jA8PK7xfu5VxvDeh8uYN25BEf94QXTo3iXcT2aU+4ZTujHlxjw+ErpXmbnWsEChBk9KKGlDjNayrX5TnO+lQiPV2X2rvTMhxHpeTASbBLNZHhFTzN60uF4o8wBEyKwyYSPlJqa31KHQ8YdFzaZwi3yzy5zdBH/RTNcg++1F5jbreWAp0QzetJ5EzJjM6qVTSb0pMLbYz30eusZ7llNwqPAl1ZZ0RmX2GJ2txpSgkOj2WF6VLaY8Xicy40tllvucMBsMaMjFf1OMvtUU2aLGVsMtwm2yD6V3Ngi5kqCtcgaUq5BLXY8tIyFDHO61Z5aZosFHWmerpJmPIXeE80cp5ojW4y31qPnNeArfYylB1nR/QAnCXO+NW5tsAY+9uS4Igr3kIJhlTknTACuyYJ+lIDJJ7ODo65ssGIlwAGjq3gWGOnWeIcVX/TJxstzf+yJDVb0Azeo8UL/Y4XHpxn9wPD2KjPmFu8WZjx3PBBMlwT32mKOWS1il/LYG6LMsWGLhrWhHx2bKFpsnM2xBLbY0JHqcRWZOZ6FLbaEB1z52Gl22C47W2xYU9h+EltE481jRtGMvzt2viizj5g5tKIftbjCFssN3Y+OLeIO0A/0UGbsxUl+Bs8MiyNWtojhrxXrg2aMnUu4M5ljKOgszehIw1Jhi4ycsMhprVwoHC8YE7bppga5D0eNV8MSrMkHhVxYbQgzNGCw5xBDbrKjJy3HkGR3mGWaOVj0t1g8HhHs7KzPUeaC54MtXlZfXdHU4XZdMf06W/X4E921VjkkveciOwLIrtmDOOGGr3rP62CofHOaPlh2t4r5WIPssWAlyowb9AhL4e87HFKJmj8entv1lFuTPSEu0gTyiKNqxRWL7M6n4q3VzviqO7Ra2y0XuDi1GjApA8YryM44o8uMzvQeaMWjLsFpDvmQ6Jw1drDDVXhNIh+wphH2cfBqQdTogmaRD/XGicbRgx3RY4rWKoIEBxfutds1DCrnkY9cIQwJZU4Od6Jm4WcRdvrCZhF5YeQtlsbGjuHDcMqeXe6aSj5iurmGQZXdISLVXMIo3VpLiGlhjwg/U8gyozcFIVOXGbcRNZ18wrDBV2Q2i+fhQrb5lLgzIODLsgf6bjULb5tbDzVqb8aNVJtPCd3BAEUvO12z5hPcbU8e34CZaxb+wJrt2GxqTVV2+I5gEyqjO9joOIRwQQhmo00ouNxenYawMPgqySYUfG5hSC2zw1PTfMoYh4CljYkDXVMQTduEgs9F6Jtjkd1jstqEgteFfyudrWJuIVKyCVVwJwg/cpA99OBtQmH7daEiIkL4eWtYgppP8LsVzqU0mXkT9uj5D/iLwFa1je+9RBcwbFjdssM1la4ZUihCcLUgO5Yj4h/a8VwxXXh1jgFDZVnRl4Zrmh3TAvuH7IlzFJMX9kgfEJJaxRzFJhpqkj0wEFCrtjI0gNhW0Gb21ip6g/CJIwh7xG3IA3usd40CRhCPCRtBldfy8DLYWjo6QTseZbc1Ts9Gh4KJg90MDsTZeoDbxDYFFyo7NkpvMweeGk8CD0523He0p8zNISGCQrscNtxIUbvYebBvYibLjtvEzcuO/mDJwL3Bk2A7z9iIjhqVdlymVbb78sdVQ4Ttd4t+OMZRz5TEY5GL6z6Wyx+eymV85aLwPnzjEKS+0YbjPe5ayetOvx9Ra2DUCkeG3S4i7uucNdg5XGToPkLyr7Z/2UasvCcmLH9gyYmZR/AMJrHH1+P9YIpj79TIH+5nsR8TCfM6rzMJCqTHB7DjtU+Npl8sobBdSChgumMZ9cicQnmYT/CX8gn+Yj7BX8sn+Gv5BH8xn+Av5RP8pXzCq66/k0/w1/IJfuYTPt9kojt5W5rNic3gGNMIgSKdiqfcQmybZS+MAKsWAhQFphq3Y9gxbCmYnUPTXNxDcoS/EjQItTEI+EKXPSJ+kj4LmBsFmxe9OR0pQoVidmouTIAsO+alk4oKCJIcdYUJgY5xkfcMgQoHu6q/64a62ym7MDO67Iy6S5OdTx7RvemMyC1e149UXjVwR4Yd88C6Gam8YuKGTLWCeCIW2Sm9erFWS61BYjRESq9ci4kbtB8VsqF5bEy+B2sVobdFE4iNsWCaa9YqgqskCRwS5RdCcGsWncH2LDu6U9z+6bbrGwhGbsM5t90eKQlpRxSIaNHvrZYMLSA7lx2+vl8Hc1KaDwsJOyDWnrWKDZCqFXaoUsQc3h5lpbPRI8tUYTUma7YkzmjZKcMQxpRd4RV7lJkyrNsI4jbQSUXF2B/xmXwfQSjKGOzzWt73EUR4Yft3KAzrqE1txH0vCiwDnVmloNMMaVi71nusPNxByruEhMSRXA9cRoGaQXb0sivWgkaCz2N8LTsCUW/NUo8xEJWZF7eHXynIGLOZ6IQAVKYBcwmrhvGN7AhnvYIt7PsI2bkxmhpN9mEKsphzl+jseHbKNATqBMTfvspeMA8V/IZKZ5a5B0mkYkQl8QNiJOcDnK7J1NYRlshOUda8yVSo6CSFAfHIFa8RbAxJ8Kx0eYRIjsNZpEh5FeUKQqMqgxNKsifEldZN9qGjkSA7pnO2RcsHVZpEHCRsbtHuVksD0Z7Z4TqcFAnuDbIT/r2YiC0uK+KEAZE2vpFkR2suaFV1yjJMxmAy1oWoTBG2EGxTXVqYDxshIC8fmR/CrAqmZDG9LYCMjisS4YUpWQw4Jr7s1GUtJNOm1SVzLdB1kDOJ2Qxp1hSKJhpGFWvJ7dIUGzCfK82UZcXGEHOlmorD2sJGHGwI6Xgxh5TP8pRlXUPISBgTWtM1esqyjKhTduZstBwis43wZ6lLa2KGJA05FjU9BJckVSW8Ce5cduqyiGUgO2R8kVOIgboM26qpSjg9C8cRKGNny72ZqoTYjZZ3g0PGFT2TLZSPVLN6aoG6rIVushL+X1MtBuqylJhtgYBIFAq6rcCQGj44SvZhKWelaLFVQ/4UptAk+zCaWj5wDnqEzLjA7uhgdf1IXdZdM32HBWX7SYxUZlmDWDgtvCXasJ9iP1ZKisosI+Sz24pUZpWDSGFGN6C7ShRmsfq2K7BmUypRl/VWd6WFwCSYncqsML0jRZUShJ/s6A08HTNl1E5wCZbyTFRmjUofu7nUum5K0VziE6EUwl7nLbkJ0YobYaqEkgeeNdgYZkozPPwkCUPBmCxHSmkW+KQoVeDKut1VpjTrTJpQkeRk8ztTmGVO08RYFy5MEjriqxUf5rPBFyG/LKsVqbpKLbpMxlchnHRTvBWKPQoebJdYJ1l3VSjMuoaQKhvrTZshWoFLKRpC3CAnrIRZpHKu6AaFFnfp7GyllK4dk3kzPKiM7tgMZAofn+bmjOfKHugymHYBQQydYFIIn2xhVQqV0uguEh2Nc9JrCOsQlkCGoFkMZ8MytDGfEgHDihmLv/w1HfdElDwTGbjyY73y4aleoS67KH2OXzlouTdaeV/LIZAMkBvYgSsX22MxFz5GzEVmszF2cDmHO8JIY/Rf19IW+6Hby3Uei7n5Acz2+iuKOQVG3OhKUBLygZoLl9RcuKjmwjU1F66puXBRzYVLai5cUnOvuv6OmgvX1FyYau7zzaZ31dy6QuByES3gcXEBI1KKgeJoo5etcN22YXSqs+qwq3CNI9boSsYm1swqQoKo0jOiLgtOsF0y0woPi9tgnNqs8sVFzV0IrhXRTkM8pAANISS2QV/o0Tm8iGq1PSaOfGVIsqXAfRizxi7PB4nG4VqZbcaeL4+OvY7rwMujsylM+SQ71VlhQMIdIyAWdKrbYaLjmgyEuLE5JpN1W57yrBVty4nxRUh2HcozhFVBwYPrGA67DnMrrlv5mPUmb2VBqrPqVRJB6Ib1UVTSC1RnMagkkqirFABCxkGe9aSKSGM0bRFdCpRnUG27tkQs7M3O0NpLnfETyQo/GByos6poGsOG51qln1hlh2xk3MkKaEAQrKtEegPEZAlmzLgeg5UjI9VZ4byAyOYSbdrusBliGwyZldHCTTZG+3hVz+CNMMOZobdyHcIGRgYQErA3puWDniSEa3eeI5hZqsHCtgmVKM4qR5D2ytKHWk0UZ5EjKHtlqCk7xVlHhCp75gZt16E4k4Kj3WeLLukooX0qNyj0hntN1jPLFGcVIYZ6z/KhQg3GHZgFGFrdLTO01iwWTfTqGx8Ok8QKmhnXdKY6sh4m/ETfP686DpaBatX4alN0ySgLa7wmrzpzhhuyBVGozyDZ+j6G1XWzoz/dO2Vh8PThqhRrcJaqZpVVhkWUHGwBFQq0GOuupX1KCq0Us/qeTXN6TMD98hyKzEGkPqsYCIWdiqCx+yUTXMoh6vGwMIOp7Uz5dAcXbaVs1QW5mDjxEyaDingUDBxzK2xh/NNuZlslsYJA/KLt+6xqa6yaF8X8DbGjLdtGgcZirpZhZGFVzTYKNCwGq4p0qAVbhyxTYk36vZyBZWgfpz7DJtEsmsYQZrNTn8WksgjWTMBKVXcI12CRNwuDsUOGZhV6KjSmFBTvJs5rPWTlUuRzJBCTolSOD5aGhpCJEawlmwqd+ixpCJlHQRhl661TWDsNIWcvCwkaq059BvHoFabWEhXCa2kn1jsVpeK2szJCBlS0xjxa4jOmvJCd8gyTz6LXxE3ZrkO/6n2x6BUuz6ryWtyJLl92qDvG27RTn9E3yu6wPlXlyJ76jH5E9sFdEGrplUvOroMHbu0y9sg2NZO2uLzbqc9sECmaKkIMwQqe+wJuuOu+MObJmx39wWRgGoZkBUIlzYUcKNCKBpFpLOwd0pE5oNNwfRpEKlwoBrNTofFpQjMVlv2SrNRnWPpNQ44PBXnBzF0KuxozaVRB3Frs8wzQ8H9W28J6s5ooRk+rI9qMjUylqPcxCm0w8YodJMm150h9hr43aW9SCcZ3YBOE53TOUgoYcyudYZ5RZvk9D0OGYrdTocHzG4hRGV3JDvWKFZWavBH3VznlzIGDK4nm23OBS1F3KHdCIOCCuJJuPu2f5ypoTJPxsSpBpLtlFFozxxB3TO3ok10HsQAmBG6XPYAjMy+FaBWPtnIM+Sd2uv1pZio0BkKbQncCCBoremUECxiKzCxcGerjIwXXM+HwTAjgyo81xYenmoIa7aI8OX7lILjeaOVdwYVYkrlYLGXGk0/0Vvyo4tmo7OHr3HYNqlsqe4v9UOCMBHTu97mUFFfzLJ09aEadvYThvlwSWicMlzliKRXsCdizFcHHS0IrXhRa8ZrQiteEVrwotOIloRUvCa1XXX9HaMVrQitexnAPk3mylIe5/BhoDY5Z5gGH/qoYbrAU1xk2XcxPe439s/aPh4c/J40bWPp4za0u5je5VRYi6s6tPsVwmSrO+dV4LuY322BmGqJwJX6fYriLAzs0tfq1t5riyxDpQPw+oXF/gSe8QOMGp7VLIHWlcYMEnSppK43LEq/LAmRWGpcVWPj8pNrfpHFZOS2IA5vlTO80LiuezDgocTxgXFYkxTcpYh4wLpQhnldJqgJNGJcVQBZg2hHGhe5WoW9XeHcYl4U4F0VHrDAuS2sI9KoqUgPGVaGsZMP7JozLwheDGgtxB4zLQhbzFXu0c4dxWa8KLRtTOGHcyKeNkO/A4kbKSSJcCoYHi8taEh+YEgWTxYXARLik3MQkcVkx4l6jSHWiuHjgrBHC6W8LisuyEMSAmpskLotC0BkI4baVxI3YiZjs0aObJC5LQiEHK5xPFBf/k0kCXnuiuCwUuUZce1tRXJoJH8o6SFxayeyJKJskLs2c/zJPFJdmxJRNs2miuDSz5lOPKC77V3C/KtZNFJd30yB93RHFZT0M+71XtyeLy0eCYNYC5cniRmWKsMfz05PF5dPG0zGufbK4UXkob9XKyeJGz7mQjXscMC4LhbW2cGRxVVXse8C+sLicOGxb5NpEcVnLbNWFZojuQHFZ+4yITIwyHCwuK6i1hljs44PFZcWV7IShkJPFZeUWwYVVfRcWlwVghA1G6i8sbmAeDKOp6T1hXNWdEZUZ5jph3MCyLEIECeoFxg1MQvSwE54TxuXario3HFhcVtkRPhjlOlhc+gc0b8WdhcVlwd/0xwLiEg4gXbpfeoC4hAkQlxgwODlcMgkIjbwKUAuIK4YB+7JBnxPEDRkuEbrIANrB4RKRwEZfbJQmh0ukgmFZPXG4JDCozbzxvAPEpT9l0V6eYgFxSXLASRk8OjlcmuHLmuGXk8OlXRkqYY2Tw6U9MqtmPO8AcWknaursOgPElXvHQ3MG1g4Ql7eFG+xGQU8QN0CSw4tVaeMJ4uqhIfg1unOSuAEiqrg7ID5JXAb7FRPCxnaSuNCVLMN3b0zsIHGDdvduNMkkcTnJMFKG3y8oLreQzKYMuR0oLuc8E0TJEN2B4sLvUTJ3W5oTxbWlSb++LSgud4C9Ln5AcSM5q6xw4IDiIijHfeyPfpK4LH9jPqvJyeGy3pyh5M08MFwWeOF17L2VBcNl5dS5Ows/MdxILwonb5j5guHydR7iRYbtTgxXe5I3hGXFcCNTrRh1+/yC4XpiQpalWzFcPjxIMnvCC4ZLPMW1Zvz5wqXCgfVUmgb8UzDco5J4LHOPGO4zIX3+ykXpfQR31yD1jTYuYriBMYJTzH7PIqSLCO7hXpai8uFe1vr0IY2w1LMf5xGO1OQnhtC/GMF9lEtIl3IJ6WIuIV3LJaRruYR0MZeQLuUS0qVcwquuv5NLSNdyCekpgvsLJtMVBFeCDPFxOCG4UmQBMY9wuIngSpLVnRmdBK4kWdizsAuBK02GbUo1rYXAJW6KoNKKPwuBK1mG9W9c5CRwtXGzJHwicCXMClyOkbaDwDVlVpqRwpPANWnWrYCyELjSZt7t5O9EcE2ddYNYB4Fr4qwGsw4A18QZX1vZDgCuqbNib3YtAO4uz+w9xwXAvSu0egBwd4G2U7mDvzWBFowsWvhbUX14SPZkJn9rIo0s13bgb02mhWAs8+BvTafVHXGe/K0JtWyQ2cLfmlJzRgouAK5JtWovUy4Armm1tN/WBHBNrFm4uvC3ptawmM0++FuTa82qgAt/a3ptx34nfmt6DY/V7AO/NcGWTVQt/O2u2MiJHvjbXbJFY1snf7trNotfFv7WRFuOer1tQXBNteVs5kHgmmpD/8w+CFyTbdVbdyaBK+GGeN/w4UHgmnK7o6qTwDXlhnjLDq4YBK5JN67m7UDgWnyY7NWuhcA17ZYsSp4Erkk3BIW6zCRwBd1iW88nAtekmxkHfmu6LextTvzWhFuxl9gmfivdho3BGWY78FvTbdgKjI8d+O2u27qBpxO/3YVb368z8Nu7dJMQmfitKTflRg74rYm31FTAXPhbE2/VXg1d+FtTb8lefFj4W1Nv3kqGC38rIRK7xeALf2vyDTPL+N7B30q+ZQqs7cDfmnzzpqMW/lbyDestWF5u8LeSbylXQ0wnfyv9Rq9zAnCl31LoO5g7AFzpt1DstiZ/a/LN7yDs5G9NvnlnlPPkbyXf4IHdib81+earjfrkb/XYsDb6kb81+YYh0mUmfyv5hmVtjPYEcE2+uZ3FngCuybccdjB3ALim36gLthXA3eXbDlFPAFfyDXNBHMcC4Eq+cWs2oHYAuLY8a7frTACXO0QkIa9uDgBX+q30YA9zErjSbxVuxLKrg8CVgCt+B2QHgCsFx7VngOwAcCXhsEtXA3YHgCsJx8qtej8BXEq4wlTNGcB1rO+maEtlAXDpMuEx6xnAZZ4YS8uW3ELg2gPMu30SuLwkXLix2AuBSwqNcWw/KgRECXjg1dDzTyJwjxruicY4ErjPZMwrEXdR+Zyg3VXGvdHKVQL3gY7LF+nbw90sterD3axl77XLa5n8oY478ZK/no470bePhFy+JOTyRSGXrwm5fE3I5YtCLl8ScvmSkHvV9XeEXL4m5PJT+vb/U8gtM9vrBJjCFxgO9G3grfpcdLzMQt8G6QDsWobTDvo2IATGbCkSJQt9i3tnUcVObFjw28BtwWcjzyZ9SyWCnnkrdk36lls7XG6LdplB32pLRrgl3bDQt4yUgkt3ynbQtwjyCMpajLrQt4HnVsDhG9c66VtGyf3+QsVC3zKO5C5oiOmgbxWLxuSsm5O+DaqdtJ2hnPQtpRw+rM1loW+p5Gop9nLkQt9Ky+GxOrPf6Vvt1B7DaZcZ9C3jfY/R8cbBDvo2SpcjPNbDmfitIkEWNLcDfctAUPid2e/0LeNA9KF4Y3sHfcs4sLIOZPZB31IkYRMOxrtO/FaSDQ4mGpY78FsEYywhBuO5J35LO75ub+0u+K3skIc2hBO/pR0exiKBhb+VxoMfsZrt5G8lLam19PnJ37L/5Eh3++Bveb+IFfaZP/lbijkCgDZFJn8rMQfBYv2c/K2UtImhhb7VWMHF6GYnfMuhLfcTRhb4lmIhUTic4Ftl/ood+jLZWyoUqIlmq2Gyt1Q0iDrLDsEO9pbKCL7O8uILe0shxYasKj7ZWwoyOIt9SAZ7G3q8UcU7Mw/2lrAFyw6qyy7sLSNa7i87YzvY28B3LBGH28OZ7C0jY6wGt8O0d/aWShgyx9DYid4yvHbNR6nfBb2l/k6NSYztgN7ShfQa74jtQG8p5uAv7ZXMgd5SyuEB26uUC3pLB+XxxNwJvQ0MhTEhDPid6C1FBta4vUG8oLdBZaHqDaUd6C39Iv50RltO9JYKDx7YNP2C3lLy8FCWcEJvKZEihYJdf6C3VH4ttjt+MNDbYMWu/fMTvaW9YuIFQ2MHesvrYzO3M5AW9FYpINasdP2J3rKfEOY9mH2gt7xfLNAmmbGgt0EHpDiD4Bf0lorZs6x6Qm8ZSjGLYmjyHb3lIPJlW4NLJ3rLlA7UUvVmH+itUjf097rbid4GvrqCVWK9nOgtJ2zja+nqzUBvmUPBzuytkxO9ZVKk80UAu8xAbzHJ4DwxX+zzA73lcuYxVtUQ3oHeRtW/fS1H9FbbQIcn0zOY6K0cMtaJHOaC3lJp4xG7ZojtQG8j+Rg4GmefH+gtJSw+EY3UHeRtpIvkHm8nww3ylmKPAI8N4SRvI/NgJHjt81RmnZsCxoa0iL3SkglH59yZRMMeHKnS9dS4PuCh+R47359oe1ICH8W+xJzJxiigYA+xdqdGSCyye2++9JPQ3qPGeqIyjmjvMyHzSvxd1D4nGnhVcm+0chHtfSDkykWsF6HDg/N6pvlQLvV04vc7nBc5mB9BveNqF4/V3dzLp/K8jWWUxFceMOtMEpRLyq1cVG7lmnIr15RbuajcyiXlVi4pt1ddf0e5lWvKrXwkzpt14Ajf1F4n8WLV+0PPAVP4WWz62AR0uphKC5hDBpreTqgpgabukvI164pZzO+0hniE77m7goAfof+puXxqjmIKAQ8zymtzi/m95njOGqZE4Es26dicP9/dY3ewsLqLN3iGRSMY+Ihjfj8nDs0U5mvmd1qf9ZUSIH88wf05WWgE5gzuzvDwYn4THibN547w8NORJJTVXp/ZvJjfbkpZ9fw2C734/UMb63bwVhuNruCAXT9joT9pA1G25uWjMGjEwTwEJbUjBU0qsfRwhKBZjPbByLrJQCMkY1LcSgkDgZZYLvbm13IeMU/ZRVgg6yCgC18HS/bZCUCzpJGjM2E4+GfSXdjm9hdi7/hzY1qEm9620s+Nb7dC1OizA35maRzxqSm8O/vcKamq0cUTfea0C8443Uk+ExVtOv57W8lnukQo1B4tqLyTzx5jzhNcpb4G+uxZ8og79D3RZ5KZETek7M1En7nDQyMYPj3hZ6/6SLGTQib87LkTYdNVjD/gZ/yPZMKCYc6DfoZa4MmzR/aZmCVmmZXaJ/uMx86DUlTfn+gzGcvIGu3xFGIMKPxwtKU90WcvuCBEd2SfCWQi2pdWnOgzrbj3ko7oM69BTFNh90SfaU6ZYnVb0Wf2D8rQTrKa6DNvpu2CfpLPBEnJfClNNslnr0petSr9JJ+JqRIVFrM8wGc+adblJO0m+CwEFhGFJNMEn0nMQina+csDfCZ22wiriKke5LMQ3eTtlbMJPpPz7ZiaClon+EwqGCPjogHRA3wmXVy6kjsr+OzJ3Owvay7cs+Z16vZm/cI9oxfMV9XTEcR06UyylBP2zH5BXu7A68CedeQ48ZvtQD0j2McY2Ik/C/TcOb2CEfIL80wMkq+Hi/+czDNPAUNH0vH8YbKX2Vs6a0GeMWFYzjWQdkLPTNa6kgxEncwz5mir0c4MWpjnygMXotHUk3jmQcWIQA0hncAzT46iiD8Bz5UvryJKFOQ6eWemxjAidmjtxJ1JEZUdsR6sc2XK2FvubEGdydFWAw8W0pkfRi+DHSw8QGdeGL5vP4d4cM7sBh3x6bxhdjrtufSFcia7AGcmvbtAzngglP87yzwYZx4jjfVreO9AnOnbsWrkrxfCuelcZW+86wScW9bZE3a49OSbMRVqzCe4mSflJWe5isk2C0kudrbCgjYzEbIfR7aAzQpMYks2mybYzFWQe7PK+AI2ex3csGPWd66ZqwS7VMynI4a1HqCj7GYm18yZHyI3ge0ANnOO47np1Y+FbOYkv+dMFrJZLibur8wvZLN2FfL2RjAPspnTPHBjNVJ5kM20JyJt9vlBNvM6PAzFAOYBNrNZuGNzG5Nr9uSderFD8hauWfdKNajpOblmr8IuD+c7CXU+ePQsh0uplCdC4LHQP2HNjxMJ529cyjwcPn8INt9o4SLTTI9TxlE89yxKvYo1Lzez1vaXm1nMxzTKvMqTPMqJQ/20ZMq1l6OPPPODXEq9lEupF3Mp9VoupV7LpdSLuZR6KZdSL+VSXnX9nVxKvZZLqR/7ajTiX4QzjOYOK38xv5NwCEzSp8BfImAc8XY6hVXbwPdIT45mMb/XXOOZGlTpb2dSdAYWbiMePdRqfqcl8mJYeqzvsq77LJPy2Vbq+y/vrkwRJiui2XBixbkBI+C005cHKk5ZmhHimHke1oydFv9dDSwfpDj0n8eENPMExVkmqiqcHThx3FGG9jSud2Li/D0FS1UfKHFs1nwR0YDTCYnzgBPWuOwU6MGI85zZaL9ashDi4lejEZULIE4iFSFVOB7RrLCbv2tjOPkgxLXiUjAYYyHEGcnjwdq5WAshThmL2H3HmichTiHLXKLRr3dC3Iu9tvB4IcSpY2O5H5U8CXGGCt706IEQp5JF4OStk5MQZxDNIqA7ntDMILozx3M6oZlRNHTCDiNPQpwSkG7J2+cHIc5IGnPd3plaCHEKTE/3awT6IMR3TWuicUHEJV+jVfEXQtwicp+dncQ8CHFpYARqdvmJiHvlRFjX2w6IOO0s6RvkPxFxdp/q2EDliYjzdh1PNLVTlAcibo+nGI+4IOJKDWDl2hsTExGnvmXCa7/+YMSpeLCFx/06gxFXgAjfmO1AlMGIU07x54vs84MRV7iKbaKdTmnWnCKDbqcuD0acESKTTsb5T0ac0TPERsknRtwL0enG0E5GXBOfMKidujwYccrc0PZIfmHEGWgiWjX+esHEueAindTplGaC4Zk787ZS4l2U537G7ITEeYARPICE+8KIM/PAtXxCxHl8dSs7Gj0JcR4jfdd7ExBvhKrg8+yU58GHC4HwwajuQYfzB39quR+2POBw0sOx2E97LGw4j0hm/siOYB5oOE9CxoK2FicZzh/GwbhahnCC4fz9G2yldl7z5MJ5ZBQcnR1+O7FwSijEO/10KjMhXIIQJyichynjP4YrTyacuaxMYb8dkHB0EMpkp5InEl4FECe7y0mEU+T3PbM5eXCGvojxjCofNDgPTkzdXsJdYHAtAdf3a9xZcBIp3HaMyR4oOCPmakfQLSA4f64gZXvbY+HA+ctkOdvPYi0YOBmRaC+uTwicYB7ckB2ePBlwppuYmvTGdA8GnOWdbHr6wIBrfWBR2nHIkwFnvompFXdkwLUfwV8kO8t5MOD8OIKKHaifDLgun9N+ePJgwLn6KrcAY70HA84DBxGp9tMZzNSTaLXakc2DAMcjxwy1FNYCgFcyhd2Oqlv478LjwPZs2op/81B5Buzbgf6mCEuI0e11zwF/85fSup1YRgohFU9c44tPR7vX6PKJ0DqR3Y+V3Csde0n7Hb9wULJvtHEV634sZdtVsnstpi5ww3I7i/nQ55V+fShlzyTuryFlT0j3Ay3bLmnZdlHLtmtatl3Tsu2ilm2XtGy7pGVfdf0dLduuadn2kVqW0DACW8Zu61xeze+RAdj6MeSBOAlCq7e1LH9yAT737AlW83vNVUa2rB5KNr4taDOT3oSfj65qNb/THLG5HvimTsUG9qag/Szr9V1Be3Q+hWelnE+s5hGR2Ebs6PZ5YDV/7KIXg2wnMQ9FhUAs2PnNE5jnq21pPxdn4eV13kq00zYXXr5w38YMsIsMXJ6RVtizyAstz1CwI6Iy+H3A8jw5N3Cf2w6sfGPJKNgZNwsqrx9OTfaDPgsp3wlghrwfXz1Aee7xnSUmOx77DsrLBXB7tuOoByjPwB5tmV5eQHkp2lzth+4WUl6KFjHCDtYPUt6y627HZQcpT02LG6p2q5OUp7rpLHbr45OUp6bFsNgPiS6kPDUtHECxevdE5VVj5I902ufvqDxFG+R7zaeDqlUaYrHsdFA1RaFL+w/LLqg87Z0/12InUg9UXqIz4DP2+YHKS6R6RXgHVF5aN2PKnlB5fV7n8WwHVF7XR8MGJU9UXhobD8TK8xOVN00erR61oPKqSeOK4XRUNZ8PTxOy48MnKk9Rm/iTAvb5gcpT1CIS6wp7F1ieorbhWdkknLQ8RS0LTflEy3PYy/5u3wLLe70xELutwUHLqyoJT7+z+IOW99wfc94nw6TlWQrFNrifujxpedVT+ZukdrL1oOU59zO2Hn88qZqaljF7smPhBi2v4kmsVqlcaHklkVy3Q20WWp4ytjU7c2aB5TsD6erSkZUnoZ+HdbDyVCxwWTtCP1B5/NVDyWdSXj+BFO1QhgWU508UMXlgRPzOyfNF68ID6rcDJo/tiW8b2QF6k5JnGS34/TWbCcmz6obh2cn8wchXnmVbgp1nNxl5ilcHrydseSLyVS+g9p1sH4Q8yYoksuIAyHNT5O/CnI6mJhje9sP5Fzy+Ml9Xdi580vEs3/K3Uk9wPHUIX9c3ln6w8aJqsh2xsaDxfAcVwtI6OMl41mlriqcjqcVnWE5goeJbJPHS7OzAwcRz+H23QvlA4lmud95OJFqIeP4SnuPbM9sBiFcmhmz3duDhCQ4UkhXbAYfv/AWGaim3ScMbQuHjfj71oOE53xkKn2B4LQPMhX46h5qX6fz1cDv+esDwWGzUn/F0DjW6qHdD1PPJwne9CGPveiwoPNdP2YmghYTniBF7MeJ9gPDFJqAdxD1BeEp+RvOnE6iD6vAunTB4HpRS9vc5+Nq9L8zRXdSiT4TVM6V0YtMfS7FX6vWSeDt+4aBF32jjIpj+WIr2EVu+/B8CkPOCCmVuZHN0cmVhbQplbmRvYmoKMTEgMCBvYmoKMTA3MzUKZW5kb2JqCjI1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYwID4+CnN0cmVhbQp4nD2QSxLDIAxD95xCR8D4f550ukruv60NnWywGIT0wH1iIqMWJ4KvxIcGpcMk8GwVhHsw2RFyhOQETYaygyRxDQ0GpcKmY5nvyaZ10kqsvNF+2Xc0Zp10ii1Fh3Yh/+f1Ityj4BrrGTwPYDHI2kql7JkwWmWmrvKo6LUf0+E9D0SrxeX1Bq7w2kc3qSgqS+r+RHc02P6Rt/se3x+ZZTgkCmVuZHN0cmVhbQplbmRvYmoKMjYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzOTIgPj4Kc3RyZWFtCnicPVJLbgUxCNvPKbhApfBNcp6p3u7df1ubzFSqCi8DtjGUlwypJT/qkogzTH71cl3iUfK9bGpn5iHuLjam+FhyX7qG2HLRmmKxTxzJL8i0VFihVt2jQ/GFKBMPAC3ggQXhvhz/8ReowdewhXLDe2QCYErUbkDGQ9EZSFlBEWH7kRXopFCvbOHvKCBX1KyFoXRiiA2WACm+qw2JmKjZoIeElZKqHdLxjKTwW8FdiWFQW1vbBHhm0BDZ3pGNETPt0RlxWRFrPz3po1EytVEZD01nfPHdMlLz0RXopNLI3cpDZ89CJ2Ak5kmY53Aj4Z7bQQsx9HGvlk9s95gpVpHwBTvKAQO9/d6Sjc974CyMXNvsTCfw0WmnHBOtvh5i/YM/bEubXMcrh0UUqLwoCH7XQRNxfFjF92SjRHe0AdYjE9VoJRAMEsLO7TDyeMZ52d4VtOb0RGijRB7UjhE9KLLF5ZwVsKf8rM2xHJ4PJntvtI+UzMyohBXUdnqots9jHdR3nvv6/AEuAKEZCmVuZHN0cmVhbQplbmRvYmoKMjcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MSA+PgpzdHJlYW0KeJyzMLZQMFAwNDBTMDQ3UjA3NlIwMTVRSDHkAgmBmLlcMMEcMMsYqCwHLItgQWRBLCNTU6gOEAuiwxCuDsGCyKYBAOvnGDIKZW5kc3RyZWFtCmVuZG9iagoyOCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDI0NyA+PgpzdHJlYW0KeJxNUbttRDEM698UXOAA62t5ngtSXfZvQ8kIkMIgoS8ppyUW9sZLDOEHWw++5JFVQ38ePzHsMyw9yeTUP+a5yVQUvhWqm5hQF2Lh/WgEvBZ0LyIrygffj2UMc8734KMQl2AmNGCsb0kmF9W8M2TCiaGOw0GbVBh3TRQsrhXNM8jtVjeyOrMgbHglE+LGAEQE2ReQzWCjjLGVkMVyHqgKkgVaYNfpG1GLgiuU1gl0otbEuszgq+f2djdDL/LgqLp4fQzrS7DC6KV7LHyuQh/M9Ew7d0kjvfCmExFmDwVSmZ2RlTo9Yn23QP+fZSv4+8nP8/0LFShcKgplbmRzdHJlYW0KZW5kb2JqCjI5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggOTAgPj4Kc3RyZWFtCnicTY1BEsAgCAPvvCJPUETQ/3R60v9fq9QOvcBOAokWRYL0NWpLMO64MhVrUCmYlJfAVTBcC9ruosr+MklMnYbTe7cDg7LxcYPSSfv2cXoAq/16Bt0P0hwiWAplbmRzdHJlYW0KZW5kb2JqCjMwIDAgb2JqCjw8IC9CQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM3Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nOMyNDBTMDY1VcjlMjc2ArNywCwjcyMgCySLYEFk0wABXwoKCmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzMjAgPj4Kc3RyZWFtCnicNVG7ccUwDOs1BRfwnfiVNI9zr8rbvw1AOxVhGgRAqrxkSrlc6pJVssLkR4fqFE35PmCm/A71kOPoHtkhulPWlnsYCMvEPKWOWE2We7gFgS8MTYm5hfP3COgrBqMwE4G6xd8/QLMkMGlw8FOQa61aYokOPCwWWLMrzK0aKVTIVXw7NrkHBXJxs9CnHJoUt9yC8GWIZEdqsa/LZSnyu/UJGIQV5ohPFImF54EOZiLxJwNie/bZYldXL6oRGdZJhwdSBNJsbhIwNEWy6oMb2FfHNT9PR9nByUG/isH4NjiZL0l5XwWhEI8X/g7P2cixkkMkFPJ9tcCII2yAEaFP7SMQZSA0RffumVI+JlWK7wBGIRx9qlcyvBeR2WqGzf8ZXdkqCgZVWR+fRnAmg0k482SjCtNStdO/+9zj8wdjY3qACmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MCA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JmafKJWzfxsgStxwT7p7uDoSMlPeYYaHBJ4MLIZT8QaZo2A1uEZSjZ3so7BuX3WB5npTq/X3BypPdnZxPc3LGfQKZW5kc3RyZWFtCmVuZG9iagozMyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDQ5ID4+CnN0cmVhbQp4nDM2tFAwUDA0MAeSRoZAlpGJQoohF0gAxMzlggnmgFkGQBqiOAeuJocrDQDG6A0mCmVuZHN0cmVhbQplbmRvYmoKMzQgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA2OCA+PgpzdHJlYW0KeJwzMzZTMFCwMAISpqaGCuZGlgophlxAPoiVywUTywGzzCzMgSwjC5CWHC5DC2MwbWJspGBmYgZkWSAxILrSAHL4EpEKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMxNyA+PgpzdHJlYW0KeJw1UktyQzEI279TcIHOmL99nnSyau6/rYQnK7AtQEIuL1nSS37UJdulw+RXH/clsUI+j+2azFLF9xazFM8tr0fPEbctCgRREz34MicVItTP1Og6eGGXPgOvEE4pFngHkwAGr+FfeJROg8A7GzLeEZORGhAkwZpLi01IlD1J/Cvl9aSVNHR+Jitz+XtyqRRqo8kIFSBYudgHpCspHiQTPYlIsnK9N1aI3pBXksdnJSYZEN0msU20wOPclbSEmZhCBeZYgNV0s7r6HExY47CE8SphFtWDTZ41qYRmtI5jZMN498JMiYWGwxJQm32VCaqXj9PcCSOmR0127cKyWzbvIUSj+TMslMHHKCQBh05jJArSsIARgTm9sIq95gs5FsCIZZ2aLAxtaCW7eo6FwNCcs6Vhxtee1/P+B0Vbe6MKZW5kc3RyZWFtCmVuZG9iagozNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDE3ID4+CnN0cmVhbQp4nDM2tFAwgMMUQy4AGpQC7AplbmRzdHJlYW0KZW5kb2JqCjM3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzM4ID4+CnN0cmVhbQp4nDVSOa7dQAzrfQpdIIB2zZznBal+7t+GlF8KQ7RWipqOFpVp+WUhVS2TLr/tSW2JG/L3yQqJE5JXJdqlDJFQ+TyFVL9ny7y+1pwRIEuVCpOTksclC/4Ml94uHOdjaz+PI3c9emBVjIQSAcsUE6NrWTq7w5qN/DymAT/iEXKuWLccYxVIDbpx2hXvQ/N5yBogZpiWigpdVokWfkHxoEetffdYVFgg0e0cSXCMjVCRgHaB2kgMObMWu6gv+lmUmAl07Ysi7qLAEknMnGJdOvoPPnQsqL8248uvjkr6SCtrTNp3o0lpzCKTrpdFbzdvfT24QPMuyn9ezSBBU9YoaXzQqp1jKJoZZYV3HJoMNMcch8wTPIczEpT0fSh+X0smuiiRPw4NoX9fHqOMnAZvAXPRn7aKAxfx2WGvHGCF0sWa5H1AKhN6YPr/1/h5/vwDHLaAVAplbmRzdHJlYW0KZW5kb2JqCjM4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjQ4ID4+CnN0cmVhbQp4nC1ROZIDQQjL5xV6QnPT77HLkff/6QrKAYOGQyA6LXFQxk8Qlive8shVtOHvmRjBd8Gh38p1GxY5EBVI0hhUTahdvB69B3YcZgLzpDUsgxnrAz9jCjd6cXhMxtntdRk1BHvXa09mUDIrF3HJxAVTddjImcNPpowL7VzPDci5EdZlGKSblcaMhCNNIVJIoeomqTNBkASjq1GjjRzFfunLI51hVSNqDPtcS9vXcxPOGjQ7Fqs8OaVHV5zLycULKwf9vM3ARVQaqzwQEnC/20P9nOzkN97SubPF9Phec7K8MBVY8ea1G5BNtfg3L+L4PePr+fwDqKVbFgplbmRzdHJlYW0KZW5kb2JqCjM5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjEwID4+CnN0cmVhbQp4nDVQyw1DMQi7ZwoWqBQCgWSeVr11/2tt0DthEf9CWMiUCHmpyc4p6Us+OkwPti6/sSILrXUl7MqaIJ4r76GZsrHR2OJgcBomXoAWN2DoaY0aNXThgqYulUKBxSXwmXx1e+i+Txl4ahlydgQRQ8lgCWq6Fk1YtDyfkE4B4v9+w+4t5KGS88qeG/kbnO3wO7Nu4SdqdiLRchUy1LM0xxgIE0UePHlFpnDis9Z31TQS1GYLTpYBrk4/jA4AYCJeWYDsrkQ5S9KOpZ9vvMf3D0AAU7QKZW5kc3RyZWFtCmVuZG9iagoyMyAwIG9iago8PCAvQmFzZUZvbnQgL0RlamFWdVNhbnMgL0NoYXJQcm9jcyAyNCAwIFIKL0VuY29kaW5nIDw8Ci9EaWZmZXJlbmNlcyBbIDMyIC9zcGFjZSA0NiAvcGVyaW9kIDQ4IC96ZXJvIC9vbmUgL3R3byAvdGhyZWUgL2ZvdXIgL2ZpdmUgL3NpeCAvc2V2ZW4KL2VpZ2h0IC9uaW5lIDYxIC9lcXVhbCA2OCAvRCBdCi9UeXBlIC9FbmNvZGluZyA+PgovRmlyc3RDaGFyIDAgL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udERlc2NyaXB0b3IgMjIgMCBSCi9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdIC9MYXN0Q2hhciAyNTUgL05hbWUgL0RlamFWdVNhbnMKL1N1YnR5cGUgL1R5cGUzIC9UeXBlIC9Gb250IC9XaWR0aHMgMjEgMCBSID4+CmVuZG9iagoyMiAwIG9iago8PCAvQXNjZW50IDkyOSAvQ2FwSGVpZ2h0IDAgL0Rlc2NlbnQgLTIzNiAvRmxhZ3MgMzIKL0ZvbnRCQm94IFsgLTEwMjEgLTQ2MyAxNzk0IDEyMzMgXSAvRm9udE5hbWUgL0RlamFWdVNhbnMgL0l0YWxpY0FuZ2xlIDAKL01heFdpZHRoIDEzNDIgL1N0ZW1WIDAgL1R5cGUgL0ZvbnREZXNjcmlwdG9yIC9YSGVpZ2h0IDAgPj4KZW5kb2JqCjIxIDAgb2JqClsgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAKNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCAzMTggNDAxIDQ2MCA4MzggNjM2Cjk1MCA3ODAgMjc1IDM5MCAzOTAgNTAwIDgzOCAzMTggMzYxIDMxOCAzMzcgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNgo2MzYgNjM2IDMzNyAzMzcgODM4IDgzOCA4MzggNTMxIDEwMDAgNjg0IDY4NiA2OTggNzcwIDYzMiA1NzUgNzc1IDc1MiAyOTUKMjk1IDY1NiA1NTcgODYzIDc0OCA3ODcgNjAzIDc4NyA2OTUgNjM1IDYxMSA3MzIgNjg0IDk4OSA2ODUgNjExIDY4NSAzOTAgMzM3CjM5MCA4MzggNTAwIDUwMCA2MTMgNjM1IDU1MCA2MzUgNjE1IDM1MiA2MzUgNjM0IDI3OCAyNzggNTc5IDI3OCA5NzQgNjM0IDYxMgo2MzUgNjM1IDQxMSA1MjEgMzkyIDYzNCA1OTIgODE4IDU5MiA1OTIgNTI1IDYzNiAzMzcgNjM2IDgzOCA2MDAgNjM2IDYwMCAzMTgKMzUyIDUxOCAxMDAwIDUwMCA1MDAgNTAwIDEzNDIgNjM1IDQwMCAxMDcwIDYwMCA2ODUgNjAwIDYwMCAzMTggMzE4IDUxOCA1MTgKNTkwIDUwMCAxMDAwIDUwMCAxMDAwIDUyMSA0MDAgMTAyMyA2MDAgNTI1IDYxMSAzMTggNDAxIDYzNiA2MzYgNjM2IDYzNiAzMzcKNTAwIDUwMCAxMDAwIDQ3MSA2MTIgODM4IDM2MSAxMDAwIDUwMCA1MDAgODM4IDQwMSA0MDEgNTAwIDYzNiA2MzYgMzE4IDUwMAo0MDEgNDcxIDYxMiA5NjkgOTY5IDk2OSA1MzEgNjg0IDY4NCA2ODQgNjg0IDY4NCA2ODQgOTc0IDY5OCA2MzIgNjMyIDYzMiA2MzIKMjk1IDI5NSAyOTUgMjk1IDc3NSA3NDggNzg3IDc4NyA3ODcgNzg3IDc4NyA4MzggNzg3IDczMiA3MzIgNzMyIDczMiA2MTEgNjA1CjYzMCA2MTMgNjEzIDYxMyA2MTMgNjEzIDYxMyA5ODIgNTUwIDYxNSA2MTUgNjE1IDYxNSAyNzggMjc4IDI3OCAyNzggNjEyIDYzNAo2MTIgNjEyIDYxMiA2MTIgNjEyIDgzOCA2MTIgNjM0IDYzNCA2MzQgNjM0IDU5MiA2MzUgNTkyIF0KZW5kb2JqCjI0IDAgb2JqCjw8IC9EIDI1IDAgUiAvZWlnaHQgMjYgMCBSIC9lcXVhbCAyNyAwIFIgL2ZpdmUgMjggMCBSIC9mb3VyIDI5IDAgUgovbmluZSAzMSAwIFIgL29uZSAzMiAwIFIgL3BlcmlvZCAzMyAwIFIgL3NldmVuIDM0IDAgUiAvc2l4IDM1IDAgUgovc3BhY2UgMzYgMCBSIC90aHJlZSAzNyAwIFIgL3R3byAzOCAwIFIgL3plcm8gMzkgMCBSID4+CmVuZG9iagozIDAgb2JqCjw8IC9GMSAyMyAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4gPj4KZW5kb2JqCjUgMCBvYmoKPDwgPj4KZW5kb2JqCjYgMCBvYmoKPDwgPj4KZW5kb2JqCjcgMCBvYmoKPDwgL0YxLURlamFWdVNhbnMtbWludXMgMzAgMCBSIC9NMCAxMiAwIFIgL00xIDEzIDAgUiAvTTIgMTQgMCBSIC9NMyAxNSAwIFIKL000IDE2IDAgUiAvTTUgMTcgMCBSIC9NNiAxOCAwIFIgL003IDE5IDAgUiAvTTggMjAgMCBSID4+CmVuZG9iagoxMiAwIG9iago8PCAvQkJveCBbIC0xMC41IC0xMC41IDEwLjUgMTAuNSBdIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzYKL1N1YnR5cGUgL0Zvcm0gL1R5cGUgL1hPYmplY3QgPj4Kc3RyZWFtCnicM1TI4jJQ8OLSNVYAolwuYwVjhRwQD8LRBfG4nLgAjTkG5gplbmRzdHJlYW0KZW5kb2JqCjEzIDAgb2JqCjw8IC9CQm94IFsgLTEwLjUgLTEwLjUgMTAuNSAxMC41IF0gL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNgovU3VidHlwZSAvRm9ybSAvVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJwzVMjiMlDw4tI1VgCiXC5jBWOFHBAPwtEF8bicuACNOQbmCmVuZHN0cmVhbQplbmRvYmoKMTQgMCBvYmoKPDwgL0JCb3ggWyAtMTAuNSAtMTAuNSAxMC41IDEwLjUgXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM2Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nDNUyOIyUPDi0jVWAKJcLmMFY4UcEA/C0QXxuJy4AI05BuYKZW5kc3RyZWFtCmVuZG9iagoxNSAwIG9iago8PCAvQkJveCBbIC0xMC41IC0xMC41IDEwLjUgMTAuNSBdIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzYKL1N1YnR5cGUgL0Zvcm0gL1R5cGUgL1hPYmplY3QgPj4Kc3RyZWFtCnicM1TI4jJQ8OLSNVYAolwuYwVjhRwQD8LRBfG4nLgAjTkG5gplbmRzdHJlYW0KZW5kb2JqCjE2IDAgb2JqCjw8IC9CQm94IFsgLTEwLjUgLTEwLjUgMTAuNSAxMC41IF0gL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNgovU3VidHlwZSAvRm9ybSAvVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJwzVMjiMlDw4tI1VgCiXC5jBWOFHBAPwtEF8bicuACNOQbmCmVuZHN0cmVhbQplbmRvYmoKMTcgMCBvYmoKPDwgL0JCb3ggWyAtMTAuNSAtMTAuNSAxMC41IDEwLjUgXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM2Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nDNUyOIyUPDi0jVWAKJcLmMFY4UcEA/C0QXxuJy4AI05BuYKZW5kc3RyZWFtCmVuZG9iagoxOCAwIG9iago8PCAvQkJveCBbIC0xMC41IC0xMC41IDEwLjUgMTAuNSBdIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzYKL1N1YnR5cGUgL0Zvcm0gL1R5cGUgL1hPYmplY3QgPj4Kc3RyZWFtCnicM1TI4jJQ8OLSNVYAolwuYwVjhRwQD8LRBfG4nLgAjTkG5gplbmRzdHJlYW0KZW5kb2JqCjE5IDAgb2JqCjw8IC9CQm94IFsgLTEwLjUgLTEwLjUgMTAuNSAxMC41IF0gL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzNgovU3VidHlwZSAvRm9ybSAvVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJwzVMjiMlDw4tI1VgCiXC5jBWOFHBAPwtEF8bicuACNOQbmCmVuZHN0cmVhbQplbmRvYmoKMjAgMCBvYmoKPDwgL0JCb3ggWyAtMTAuNSAtMTAuNSAxMC41IDEwLjUgXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDM2Ci9TdWJ0eXBlIC9Gb3JtIC9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nDNUyOIyUPDi0jVWAKJcLmMFY4UcEA/C0QXxuJy4AI05BuYKZW5kc3RyZWFtCmVuZG9iagoyIDAgb2JqCjw8IC9Db3VudCAxIC9LaWRzIFsgMTAgMCBSIF0gL1R5cGUgL1BhZ2VzID4+CmVuZG9iago0MCAwIG9iago8PCAvQ3JlYXRpb25EYXRlIChEOjIwMjEwMjE2MDA0MzQ0KzAzJzAwJykKL0NyZWF0b3IgKE1hdHBsb3RsaWIgdjMuMy4zLCBodHRwczovL21hdHBsb3RsaWIub3JnKQovUHJvZHVjZXIgKE1hdHBsb3RsaWIgcGRmIGJhY2tlbmQgdjMuMy4zKSA+PgplbmRvYmoKeHJlZgowIDQxCjAwMDAwMDAwMDAgNjU1MzUgZiAKMDAwMDAwMDAxNiAwMDAwMCBuIAowMDAwMDE4NzI1IDAwMDAwIG4gCjAwMDAwMTY4NzQgMDAwMDAgbiAKMDAwMDAxNjkwNiAwMDAwMCBuIAowMDAwMDE3MDA1IDAwMDAwIG4gCjAwMDAwMTcwMjYgMDAwMDAgbiAKMDAwMDAxNzA0NyAwMDAwMCBuIAowMDAwMDAwMDY1IDAwMDAwIG4gCjAwMDAwMDAzOTggMDAwMDAgbiAKMDAwMDAwMDIwOCAwMDAwMCBuIAowMDAwMDExMjA4IDAwMDAwIG4gCjAwMDAwMTcxOTUgMDAwMDAgbiAKMDAwMDAxNzM2NSAwMDAwMCBuIAowMDAwMDE3NTM1IDAwMDAwIG4gCjAwMDAwMTc3MDUgMDAwMDAgbiAKMDAwMDAxNzg3NSAwMDAwMCBuIAowMDAwMDE4MDQ1IDAwMDAwIG4gCjAwMDAwMTgyMTUgMDAwMDAgbiAKMDAwMDAxODM4NSAwMDAwMCBuIAowMDAwMDE4NTU1IDAwMDAwIG4gCjAwMDAwMTU2MTYgMDAwMDAgbiAKMDAwMDAxNTQxNiAwMDAwMCBuIAowMDAwMDE1MDIzIDAwMDAwIG4gCjAwMDAwMTY2NjkgMDAwMDAgbiAKMDAwMDAxMTIzMCAwMDAwMCBuIAowMDAwMDExNDYzIDAwMDAwIG4gCjAwMDAwMTE5MjggMDAwMDAgbiAKMDAwMDAxMjA3MSAwMDAwMCBuIAowMDAwMDEyMzkxIDAwMDAwIG4gCjAwMDAwMTI1NTMgMDAwMDAgbiAKMDAwMDAxMjcyMyAwMDAwMCBuIAowMDAwMDEzMTE2IDAwMDAwIG4gCjAwMDAwMTMyNjggMDAwMDAgbiAKMDAwMDAxMzM4OSAwMDAwMCBuIAowMDAwMDEzNTI5IDAwMDAwIG4gCjAwMDAwMTM5MTkgMDAwMDAgbiAKMDAwMDAxNDAwOCAwMDAwMCBuIAowMDAwMDE0NDE5IDAwMDAwIG4gCjAwMDAwMTQ3NDAgMDAwMDAgbiAKMDAwMDAxODc4NSAwMDAwMCBuIAp0cmFpbGVyCjw8IC9JbmZvIDQwIDAgUiAvUm9vdCAxIDAgUiAvU2l6ZSA0MSA+PgpzdGFydHhyZWYKMTg5NDIKJSVFT0YK\n", "image/svg+xml": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-02-16T00:43:44.406162\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.3.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Regularisation alpha\n", "alpha = 10 ** -4 \n", "\n", "# 100 points equispaced between 0 and 1\n", "x_plot = np.linspace(-0.2, 1.2, 100)\n", "\n", "# Figure to plot maximum a posteriori models and training data\n", "fig = plt.figure(figsize = (8, 8))\n", "\n", "for D in range(1, 10):\n", " \n", " # Design matrix at training data\n", " phi = np.array([[x ** d for d in range(D + 1)] for x in x_data])\n", " \n", " # regularization term = lambda * (indentity matrix)\n", " reg_term = alpha * np.eye(D + 1) \n", " \n", " # Solve for the maximum a posteriori weights\n", " w = np.linalg.solve((phi.T).dot(phi) + reg_term, (phi.T).dot(y_data))\n", " \n", " # Design matrix for plotting\n", " phi_pred = np.array([[x ** d for d in range(D + 1)] for x in x_plot])\n", " \n", " # model predictions as before\n", " y_plot = phi_pred.dot(w)\n", " \n", " # Plot data and predictions separately for each model\n", " plt.subplot(3, 3, D)\n", " \n", " # Plot data and maximum a posteriori polynomial models\n", " plt.plot(x_plot, y_plot, color = 'black')\n", " plt.scatter(x_data, y_data, marker = 'x', color = 'red')\n", " \n", " # Format plots\n", " plt.text(0.1, -1.8, s = 'D = ' + str(D), color = 'black', fontsize = 12,\n", " horizontalalignment = 'right', verticalalignment = 'bottom')\n", " plt.xlim([-0.2, 1.2])\n", " plt.ylim([-2, 2])\n", " \n", " if D % 3 is not 1:\n", " plt.yticks([])\n", " else:\n", " plt.yticks(np.linspace(-2, 2, 5))\n", " \n", " if D < 7:\n", " plt.xticks([])\n", " else:\n", " plt.xticks(np.linspace(0, 1, 3))\n", " \n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "hide_input": true }, "source": [ "The plots show that overfitting is greatly diminished. This did involve introducing another free-parameter $\\alpha$ which has to be set. **Try altering the value of \\\\(\\boldsymbol{\\alpha}\\\\) to see how the fits change**. One principled way to choose \\\\(\\alpha\\\\) would be to try different values for it using training/test datasets, and pick the \\\\(\\alpha\\\\) which results in the best test performance. Let's also have a look at how regularisation affects the train/test errors." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "# Load larger dataset for splitting into training and validation sets\n", "x_extended = np.load('reg_nonlin_x_extended.npy')\n", "y_extended = np.load('reg_nonlin_y_extended.npy')\n", "\n", "# Split into training and validation sets\n", "x_train = x_extended[:10]\n", "x_valid = x_extended[10:]\n", "y_train = y_extended[:10]\n", "y_valid = y_extended[10:]" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "hide_input": true, "tags": [ "center-output", "hide-input" ] }, "outputs": [ { "data": { "application/pdf": "JVBERi0xLjQKJazcIKu6CjEgMCBvYmoKPDwgL1BhZ2VzIDIgMCBSIC9UeXBlIC9DYXRhbG9nID4+CmVuZG9iago4IDAgb2JqCjw8IC9FeHRHU3RhdGUgNCAwIFIgL0ZvbnQgMyAwIFIgL1BhdHRlcm4gNSAwIFIKL1Byb2NTZXQgWyAvUERGIC9UZXh0IC9JbWFnZUIgL0ltYWdlQyAvSW1hZ2VJIF0gL1NoYWRpbmcgNiAwIFIKL1hPYmplY3QgNyAwIFIgPj4KZW5kb2JqCjEwIDAgb2JqCjw8IC9Bbm5vdHMgWyBdIC9Db250ZW50cyA5IDAgUgovR3JvdXAgPDwgL0NTIC9EZXZpY2VSR0IgL1MgL1RyYW5zcGFyZW5jeSAvVHlwZSAvR3JvdXAgPj4KL01lZGlhQm94IFsgMCAwIDUxNC4wMTg3NSAyMzUuMjYxMjUgXSAvUGFyZW50IDIgMCBSIC9SZXNvdXJjZXMgOCAwIFIKL1R5cGUgL1BhZ2UgPj4KZW5kb2JqCjkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMSAwIFIgPj4Kc3RyZWFtCnicnVjLbhVHEN3PV/QyWdDuevRricVDQmJhsJK1ZYxjhI1iIvj9nJ65M1M9ufZcAkJmjqrreaq62uS+DGcvyd1+d8F9wb+fjtxbd/bq5sfd9c2Ht+fu+vsQgN8PkdQHKjni66v5YomeE3EEGrqvv4bhYYBuHHgLtbfDkILX8ZBGL6EK/gfFIfmyQb8alEPxUg7woqFDYekz4uApjlsYQyy+IJpmGshQgs+hplg70wYlX2bTwzm8/jmcX7qzN+SI3eXnIcMbWFPIhuq4eirapC8/Db+F393lF/f6chhNDpTxs5Lm3Nmy8I4xStmHTJQgzHljjXtrSDZkhWufVAvvWOOYPUeuBcIaN9a0tyaRIBuJtLNm4R1rotmraAlEknRjLfXWFMpUUhXurFl4xxo0+0QpURUqvLFWZmt/42BwLwJ4wzl7GglGwUcujcnX91DausDLBODw2Rt2lJqJ3141NQPUXBzcPs7y5Ot/SH60HMmnyUcFOzWVmWU+bIi22CkJtd7aWcE9O4V8JGOH4xN2CGeKbg0ZdM8SMXsuxlR8KqTWAq1qG1MrumsqiQ/RmMpPRWUmiTHVzZfnTXEQX3gxRaZQ84mRKY1fLxrTqHqJiRvH1LPmiVT346BtGj68//h64aY7QifV5HUKsbjHG/ene3Ds3jkUcpx5zdKH7Rg30w4M97XEhA7CYOKIJmq5Gv2BN7VWBG0GFiFCiZBQ4EwRh1lKcol9TBmd2OB14jQ419oyJxjjmGMcgJJHzKWB67hAcDkjuKZBavKYsIFljFkwI5oK0+6AMT9gu8EgeORclEaYcFk0r4ePSOO7Qw4e21Xwc/k4NHr7WHIR2jxf8kG15IKGd2fvg3v1bRFH6kOd/6KIKFuQJMJS6QRxrwzdCJjrvnRr+4L0UUBmd8UxFtBVEUmhuK87eMx4iiFSPCFM1F9z0JQyan2CtCgIUBSVPUUaPwm9JnpCTiDNNYkSBav7Yrhw/6NBJiNdgwybBsnsBTSN2waR4jVxqbxpEFWQW0ikbxBIgChBqG8QwJQRfN8gmXzmFEn6FoErRZlz6lsEcFbNo7RpEcDo1rH3TIdAtcZMyXTIlIRDh8wfT3fIkhAEOVWAnqmXYCJm5hS0atkVbxkIrFgIMGB2dWdFOwUQoYS8r7tRR9B4uDpK3pUGPyLSixjnvn5WmChmNDYIcZIfNWPqV8X9cYJ0QjM13fEk3UkRHz6S1T21B4/jcF6I+zvh6KpyfMcGbY5t6vdPbeqQ/4V130r3F/KT2qcxPy3705S/Xe7bMq7RuMynvYQxsA9H2/16+Xh193D3cOuuHj65H1df7z5d/XP37cHdPD5+e5wv3rOX0p4T0/tlyt7hFQMvPRpxXEjQWEHHuxx7aa0HVw3cXjIgSoT3YvGGrsIZMxwzxYIyS14PKwzerzmwMIYBRgSGnbHW9pKD8OqaAa9tJAbH5CA8QJJE5U4JpVV6MdihZHWnNUuypqPDl8iNRZOk45m+bu+98/m9N471RoHndh8MjEVRxQu1ziXTY/jOFoFXhTnFRpsg95UVU8zgDa1YnmiSToRZineSYEnC0lIbKrgOoHRMm7Q7K2SM+4ZnzMTSdMhqUbE2h4CdskMX7ycdC45nQCUWdZ1mNFXEjRF7P/A0wx2hkw7jdfY118jJdRGWNUKTj2L8wHsrYT1JqZPm4osUkl4zUM4RfzZ+cG4cqVk7r4EeidCiNh8WX7NnNa+Ztn7Yuliv1yraCPloPnj1Y6RtOPxCxI6rML5VdSV7RWNMp5Z5Nc+lZ9eYLddxiWMtyVuqL/DONtAxHS/jRZehwQJbxgCsUvEq7PiVC3LPKhua54rWx9zq+Ayw4LWtPbo43pMcL1hFtdrvcYzeglWo8MaH0tamGPKG4gXKcN9G6igOdI7NJE9WH0z5V1nDFKPW8Mr6YEi4+mv4akKzqMmDgU3SjN41v8YHWwvjsCmcic3U+AgffoXY7dcrPN/DfyxX70zui+Ffm2cpqwplbmRzdHJlYW0KZW5kb2JqCjExIDAgb2JqCjE0OTcKZW5kb2JqCjE4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTg4ID4+CnN0cmVhbQp4nD2Pyw3DMAxD756CI+hjSfY8KXpK97+WcooCcURIJvVcJRBY8VdmPBsvHZr7yM/wpUfdY24/KmqhJJCeyChcIzlJ2Scg/KlzBietfNGnCjfeWhuWPcksmBRTChpM3RtqwkmkYAmClhDMoLPbvhUqTNHmnU19jcN9/3HvYWHIuQiu00B2tkof0U/hN7lZlaQ0a2VvXGQhd+pi/lMf/lahQTrWTMSPlB6ypjONnV7oTOp6/RHu8f4Cwp5BxgplbmRzdHJlYW0KZW5kb2JqCjE2IDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2Fucy1PYmxpcXVlIC9DaGFyUHJvY3MgMTcgMCBSCi9FbmNvZGluZyA8PCAvRGlmZmVyZW5jZXMgWyA2OCAvRCBdIC9UeXBlIC9FbmNvZGluZyA+PiAvRmlyc3RDaGFyIDAKL0ZvbnRCQm94IFsgLTEwMTYgLTM1MSAxNjYwIDEwNjggXSAvRm9udERlc2NyaXB0b3IgMTUgMCBSCi9Gb250TWF0cml4IFsgMC4wMDEgMCAwIDAuMDAxIDAgMCBdIC9MYXN0Q2hhciAyNTUgL05hbWUgL0RlamFWdVNhbnMtT2JsaXF1ZQovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxNCAwIFIgPj4KZW5kb2JqCjE1IDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyA5NgovRm9udEJCb3ggWyAtMTAxNiAtMzUxIDE2NjAgMTA2OCBdIC9Gb250TmFtZSAvRGVqYVZ1U2Fucy1PYmxpcXVlCi9JdGFsaWNBbmdsZSAwIC9NYXhXaWR0aCAxMzUwIC9TdGVtViAwIC9UeXBlIC9Gb250RGVzY3JpcHRvciAvWEhlaWdodCAwID4+CmVuZG9iagoxNCAwIG9iagpbIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwCjYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgMzE4IDQwMSA0NjAgODM4IDYzNgo5NTAgNzgwIDI3NSAzOTAgMzkwIDUwMCA4MzggMzE4IDM2MSAzMTggMzM3IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYKNjM2IDYzNiAzMzcgMzM3IDgzOCA4MzggODM4IDUzMSAxMDAwIDY4NCA2ODYgNjk4IDc3MCA2MzIgNTc1IDc3NSA3NTIgMjk1CjI5NSA2NTYgNTU3IDg2MyA3NDggNzg3IDYwMyA3ODcgNjk1IDYzNSA2MTEgNzMyIDY4NCA5ODkgNjg1IDYxMSA2ODUgMzkwIDMzNwozOTAgODM4IDUwMCA1MDAgNjEzIDYzNSA1NTAgNjM1IDYxNSAzNTIgNjM1IDYzNCAyNzggMjc4IDU3OSAyNzggOTc0IDYzNCA2MTIKNjM1IDYzNSA0MTEgNTIxIDM5MiA2MzQgNTkyIDgxOCA1OTIgNTkyIDUyNSA2MzYgMzM3IDYzNiA4MzggNjAwIDYzNiA2MDAgMzE4CjM1MiA1MTggMTAwMCA1MDAgNTAwIDUwMCAxMzUwIDYzNSA0MDAgMTA3MCA2MDAgNjg1IDYwMCA2MDAgMzE4IDMxOCA1MTggNTE4CjU5MCA1MDAgMTAwMCA1MDAgMTAwMCA1MjEgNDAwIDEwMjggNjAwIDUyNSA2MTEgMzE4IDQwMSA2MzYgNjM2IDYzNiA2MzYgMzM3CjUwMCA1MDAgMTAwMCA0NzEgNjE3IDgzOCAzNjEgMTAwMCA1MDAgNTAwIDgzOCA0MDEgNDAxIDUwMCA2MzYgNjM2IDMxOCA1MDAKNDAxIDQ3MSA2MTcgOTY5IDk2OSA5NjkgNTMxIDY4NCA2ODQgNjg0IDY4NCA2ODQgNjg0IDk3NCA2OTggNjMyIDYzMiA2MzIgNjMyCjI5NSAyOTUgMjk1IDI5NSA3NzUgNzQ4IDc4NyA3ODcgNzg3IDc4NyA3ODcgODM4IDc4NyA3MzIgNzMyIDczMiA3MzIgNjExIDYwOAo2MzAgNjEzIDYxMyA2MTMgNjEzIDYxMyA2MTMgOTk1IDU1MCA2MTUgNjE1IDYxNSA2MTUgMjc4IDI3OCAyNzggMjc4IDYxMiA2MzQKNjEyIDYxMiA2MTIgNjEyIDYxMiA4MzggNjEyIDYzNCA2MzQgNjM0IDYzNCA1OTIgNjM1IDU5MiBdCmVuZG9iagoxNyAwIG9iago8PCAvRCAxOCAwIFIgPj4KZW5kb2JqCjIzIDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNzkgPj4Kc3RyZWFtCnicTc27DcAgDATQnik8AuD/PlGqsH8bGyJCYz/pTjrBDhXc4rAYaHe4WvGlUZh96pkSklBzPURYMyU6hKRf+ssww5jYyLbvt1buF94bHBkKZW5kc3RyZWFtCmVuZG9iagoyNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDg3ID4+CnN0cmVhbQp4nD2OuxHAMAhDe6ZgBPMJhn1yqZz924A/adBDOk64CTYMz9Htws6BN0HuRS+wLm+ACiGLJBn59s7FolaZxa+mReqENEmcN1HoTotK5wPj9A54Prp1HXoKZW5kc3RyZWFtCmVuZG9iagoyNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDIyNyA+PgpzdHJlYW0KeJxFkEuOAyEQQ/ecwkeg/nCejrLq3H87LjrRbLAlKNczuQMTe/HITJRuvGS4O8wVn+EZMHP4SphsxEzoTlwjlK4U4VSfCI7L3rzpoIl7RM6jngVZ1c4NagFnkuaC7YIu54wVN87JrUblzfSj1xC+aXcf13mH9kjj3sNUvs451c67ighpC1nVtL6QbBTJDms/Kk3bzssQseBsGlboHN4Iu1d3J0sYfr/yMCUTPw/d+lF8XTej6xRnJ1cma8956EnpX/XKow/FcSnoF7HtzCT3X6dTkqlTe2fvaf2nuMf7D5BuVjkKZW5kc3RyZWFtCmVuZG9iagoyNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMzOCA+PgpzdHJlYW0KeJw1UkuSW0EI279T6AKuav7NeZya1eT+2wjsrKCBFhKQFjjIxEsMUY1yxR95gvE6gb/r5Wn8Pt6F1IKnIv3AtWkb78eaNVGwNGIpzD72/Sghx1Pj3xDouUgTZmQyciAZiPu1Pn/Wm0w5/AakaXP6KEl6EC3Y3Rp2fFmQQdKTGpbs5Id1LbC6CE2YG2siGTm1MjXPx57hMp4YI0HVLCBJn7hPFYxIMx47Zy15kOF4qhcvfr2N1zKPqZdVBTK2CeZgO5kJpygiEL+gJLmJu2jqKI5mxprbhYaSIvfdPZyc9Lq/nEQFXgnhLNYSjhl6yjInOw1KoGrlBJhhvfaFcZo2SrhT0+1dsa/fZyZh3Oaws1IyDc5xcC+bzBEke90xYRMeh5j37hGMxLz5XWwRXLnMuSbTj/0o2kgfFNfnXE2ZrSjhH6rkiRXX+P/83s/PP5A3fbEKZW5kc3RyZWFtCmVuZG9iagoyNyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDY0ID4+CnN0cmVhbQp4nDMzNFQwUNA1AhJmhiYK5kaWCimGXEA+iJXLBRPLAbPMTMyALGNTUySWAZA2MjWD0xAZoAFwBkR/GgApTxROCmVuZHN0cmVhbQplbmRvYmoKMjggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MCA+PgpzdHJlYW0KeJwzszBRMFCwAGIzczMFcyNLhRRDLiMLM6BALpcFWCCHy9DQEMoyNjFSMDQ0BbJMzY2hYjCNQFlLkEE5UP05XGkAT1QSLwplbmRzdHJlYW0KZW5kb2JqCjI5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzA0ID4+CnN0cmVhbQp4nD2SO5LDMAxDe52CF8iM+JPk82Qnlff+7T4yyVaASYkAKC91mbKmPCBpJgn/0eHhYjvld9iezczAtUQvE8spz6ErxNxF+bKZjbqyOsWqwzCdW/SonIuGTZOa5ypLGbcLnsO1ieeWfcQPNzSoB3WNS8IN3dVoWQrNcHX/O71H2Xc1PBebVOrUF48XURXm+SFPoofpSuJ8PCghXHswRhYS5FPRQI6zXK3yXkL2DrcassJBaknnsyc82HV6Ty5uF80QD2S5VPhOUezt0DO+7EoJPRK24VjufTuasekamzjsfu9G1sqMrmghfshXJ+slYNxTJkUSZE62WG6L1Z7uoSimc4ZzGSDq2YqGUuZiV6t/DDtvLC/ZLMiUzAsyRqdNnjh4yH6NmvR5led4/QFs83M7CmVuZHN0cmVhbQplbmRvYmoKMzAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMjcgPj4Kc3RyZWFtCnicNU87sgMhDOs5hS6QGYxtYM+zmVQv92+fZLINEv5I8vRERyZe5sgIrNnxthYZiBn4FlPxrz3tw4TqPbiHCOXiQphhJJw167ibp+PFv13lM9bBuw2+YpYXBLYwk/WVxZnLdsFYGidxTrIbY9dEbGNd6+kU1hFMKAMhne0wJcgcFSl9sqOMOTpO5InnYqrFLr/vYX3BpjGiwhxXBU/QZFCWPe8moB0X9N/Vjd9JNIteAjKRYGGdJObOWU741WtHx1GLIjEnpBnkMhHSnK5iCqEJxTo7CioVBZfqc8rdPv9oXVtNCmVuZHN0cmVhbQplbmRvYmoKMzEgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDUgPj4Kc3RyZWFtCnicRVC7jUMxDOs9BRcIYP0se553SJXbvz1KRnCFIVo/kloSmIjASwyxlG/iR0ZBPQu/F4XiM8TPF4VBzoSkQJz1GRCZeIbaRm7odnDOvMMzjDkCF8VacKbTmfZc2OScBycQzm2U8YxCuklUFXFUn3FM8aqyz43XgaW1bLPTkewhjYRLSSUml35TKv+0KVsq6NpFE7BI5IGTTTThLD9DkmLMoJRR9zC1jvRxspFHddDJ2Zw5LZnZ7qftTHwPWCaZUeUpnecyPiep81xOfe6zHdHkoqVV+5z93pGW8iK126HV6VclUZmN1aeQuDz/jJ/x/gOOoFk+CmVuZHN0cmVhbQplbmRvYmoKMzIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAzOTIgPj4Kc3RyZWFtCnicPVJLbgUxCNvPKbhApfBNcp6p3u7df1ubzFSqCi8DtjGUlwypJT/qkogzTH71cl3iUfK9bGpn5iHuLjam+FhyX7qG2HLRmmKxTxzJL8i0VFihVt2jQ/GFKBMPAC3ggQXhvhz/8ReowdewhXLDe2QCYErUbkDGQ9EZSFlBEWH7kRXopFCvbOHvKCBX1KyFoXRiiA2WACm+qw2JmKjZoIeElZKqHdLxjKTwW8FdiWFQW1vbBHhm0BDZ3pGNETPt0RlxWRFrPz3po1EytVEZD01nfPHdMlLz0RXopNLI3cpDZ89CJ2Ak5kmY53Aj4Z7bQQsx9HGvlk9s95gpVpHwBTvKAQO9/d6Sjc974CyMXNvsTCfw0WmnHBOtvh5i/YM/bEubXMcrh0UUqLwoCH7XQRNxfFjF92SjRHe0AdYjE9VoJRAMEsLO7TDyeMZ52d4VtOb0RGijRB7UjhE9KLLF5ZwVsKf8rM2xHJ4PJntvtI+UzMyohBXUdnqots9jHdR3nvv6/AEuAKEZCmVuZHN0cmVhbQplbmRvYmoKMzMgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDcgPj4Kc3RyZWFtCnicTVG7bUQxDOvfFFzgAOtreZ4LUl32b0PJCJDCIKEvKaclFvbGSwzhB1sPvuSRVUN/Hj8x7DMsPcnk1D/muclUFL4VqpuYUBdi4f1oBLwWdC8iK8oH349lDHPO9+CjEJdgJjRgrG9JJhfVvDNkwomhjsNBm1QYd00ULK4VzTPI7VY3sjqzIGx4JRPixgBEBNkXkM1go4yxlZDFch6oCpIFWmDX6RtRi4IrlNYJdKLWxLrM4Kvn9nY3Qy/y4Ki6eH0M60uwwuileyx8rkIfzPRMO3dJI73wphMRZg8FUpmdkZU6PWJ9t0D/n2Ur+PvJz/P9CxUoXCoKZW5kc3RyZWFtCmVuZG9iagozNCAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDkwID4+CnN0cmVhbQp4nE2NQRLAIAgD77wiT1BE0P90etL/X6vUDr3ATgKJFkWC9DVqSzDuuDIVa1ApmJSXwFUwXAva7qLK/jJJTJ2G03u3A4Oy8XGD0kn79nF6AKv9egbdD9IcIlgKZW5kc3RyZWFtCmVuZG9iagozNSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDMzOCA+PgpzdHJlYW0KeJxFUktyxTAI2+cUXCAz5mfj87xOV+n9t5VwOt089AwICTI9ZUim3DaWZITkHPKlV2SI1ZCfRo5ExBDfKaHArvK5vJbEXMhuiUrxoR0/l6U3Ms2u0Kq3R6c2i0Y1KyPnIEOEelbozO5R22TD63Yh6TpTFodwLP9DBbKUdcoplARtQd/YI+hvFjwR3Aaz5nKzuUxu9b/uWwue1zpbsW0HQAmWc95gBgDEwwnaAMTc2t4WKSgfVbqKScKt8lwnO1C20Kp0vDeAGQcYOWDDkq0O12hvAMM+D/SiRsX2FaCoLCD+ztlmwd4xyUiwJ+YGTj1xOsWRcEk4xgJAiq3iFLrxHdjiLxeuiJrwCXU6ZU28wp7a4sdCkwjvUnEC8CIbbl0dRbVsT+cJtD8qkjNipB7E0QmR1JLOERSXBvXQGvu4iRmvjcTmnr7dP8I5n+v7Fxa4g+AKZW5kc3RyZWFtCmVuZG9iagozNiAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDY4ID4+CnN0cmVhbQp4nDMyt1AwULA0ARKGFiYK5mYGCimGXEC+qYm5Qi4XSAzEygGzDIC0JZyCiFtCNEGUglgQpWYmZhBJOAMilwYAybQV5QplbmRzdHJlYW0KZW5kb2JqCjM3IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggNDUgPj4Kc3RyZWFtCnicMzK3UDBQsDQBEoYWJgrmZgYKKYZclhBWLhdMLAfMAtGWcAoingYAn30MtQplbmRzdHJlYW0KZW5kb2JqCjM4IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTYxID4+CnN0cmVhbQp4nEWQSxLDIAxD95xCR/BHBnyedLpK77+tIU2zgKexQAZ3JwSptQUT0QUvbUu6Cz5bCc7GeOg2bjUS5AR1gFak42iUUn25xWmVdPFoNnMrC60THWYOepSjGaAQOhXe7aLkcqbuzvlHcPVf9Uex7pzNxMBk5Q6EZvUp7nybHVFd3WR/0mNu1mt/FfaqsLSspeWE285dM6AE7qkc7f0FqXM6hAplbmRzdHJlYW0KZW5kb2JqCjM5IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMjE0ID4+CnN0cmVhbQp4nD1QuxFDMQjrPQUL5M587TfPy6XL/m0knKRCNkISlJpMyZSHOsqSrClPHT5LYoe8h+VuZDYlKkUvk7Al99AK8X2J5hT33dWWs0M0l2g5fgszKqobHdNLNppwKhO6oNzDM/oNbXQDVocesVsg0KRg17YgcscPGAzBmROLIgxKTQb/rXL3UtzvPRxvooiUdPCu+eX0y88tvE49jkS6vfmKa3GmOgpEcEZq8op0YcWyyEOk1QQ1PQNrtQCu3nr5N2hHdBmA7BOJ4zSlHEP/1rjH6wOHilL0CmVuZHN0cmVhbQplbmRvYmoKNDAgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA4MCA+PgpzdHJlYW0KeJxFjLsNwDAIRHumYAR+JmafKJWzfxsgStxwT7p7uDoSMlPeYYaHBJ4MLIZT8QaZo2A1uEZSjZ3so7BuX3WB5npTq/X3BypPdnZxPc3LGfQKZW5kc3RyZWFtCmVuZG9iago0MSAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDQ5ID4+CnN0cmVhbQp4nDM2tFAwUDA0MAeSRoZAlpGJQoohF0gAxMzlggnmgFkGQBqiOAeuJocrDQDG6A0mCmVuZHN0cmVhbQplbmRvYmoKNDIgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxNTcgPj4Kc3RyZWFtCnicRZC5EUMxCERzVUEJErAI6rHH0Xf/qRf5SrRvAC2HryVTqh8nIqbc12j0MHkOn00lVizYJraTGnIbFkFKMZh4TjGro7ehmYfU67ioqrh1ZpXTacvKxX/zaFczkz3CNeon8E3o+J88tKnoW6CvC5R9QLU4nUlQMX2vYoGjnHZ/IpwY4D4ZR5kpI3Fibgrs9xkAZr5XuMbjBd0BN3kKZW5kc3RyZWFtCmVuZG9iago0MyAwIG9iago8PCAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDY4ID4+CnN0cmVhbQp4nDMzNlMwULAwAhKmpoYK5kaWCimGXEA+iJXLBRPLAbPMLMyBLCMLkJYcLkMLYzBtYmykYGZiBmRZIDEgutIAcvgSkQplbmRzdHJlYW0KZW5kb2JqCjQ0IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMzE3ID4+CnN0cmVhbQp4nDVSS3JDMQjbv1Nwgc6Yv32edLJq7r+thCcrsC1AQi4vWdJLftQl26XD5Fcf9yWxQj6P7ZrMUsX3FrMUzy2vR88Rty0KBFETPfgyJxUi1M/U6Dp4YZc+A68QTikWeAeTAAav4V94lE6DwDsbMt4Rk5EaECTBmkuLTUiUPUn8K+X1pJU0dH4mK3P5e3KpFGqjyQgVIFi52AekKykeJBM9iUiycr03VojekFeSx2clJhkQ3SaxTbTA49yVtISZmEIF5liA1XSzuvocTFjjsITxKmEW1YNNnjWphGa0jmNkw3j3wkyJhYbDElCbfZUJqpeP09wJI6ZHTXbtwrJbNu8hRKP5MyyUwccoJAGHTmMkCtKwgBGBOb2wir3mCzkWwIhlnZosDG1oJbt6joXA0JyzpWHG157X8/4HRVt7owplbmRzdHJlYW0KZW5kb2JqCjQ1IDAgb2JqCjw8IC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTcgPj4Kc3RyZWFtCnicMza0UDCAwxRDLgAalALsCmVuZHN0cmVhbQplbmRvYmoKNDYgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAxMzEgPj4Kc3RyZWFtCnicRY/LDQQhDEPvVOES8hk+qYfVntj+r+swmkFC+EEiO/EwCKzz8jbQxfDRosM3/jbVq2OVLB+6elJWD+mQh7zyFVBpMFHEhVlMHUNhzpjKyJYytxvhtk2DrGyVVK2DdjwGD7anZasIfqltYeos8QzCVV64xw0/kEutd71Vvn9CUzCXCmVuZHN0cmVhbQplbmRvYmoKNDcgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyNDggPj4Kc3RyZWFtCnicLVE5kgNBCMvnFXpCc9PvscuR9//pCsoBg4ZDIDotcVDGTxCWK97yyFW04e+ZGMF3waHfynUbFjkQFUjSGFRNqF28Hr0HdhxmAvOkNSyDGesDP2MKN3pxeEzG2e11GTUEe9drT2ZQMisXccnEBVN12MiZw0+mjAvtXM8NyLkR1mUYpJuVxoyEI00hUkih6iapM0GQBKOrUaONHMV+6csjnWFVI2oM+1xL29dzE84aNDsWqzw5pUdXnMvJxQsrB/28zcBFVBqrPBAScL/bQ/2c7OQ33tK5s8X0+F5zsrwwFVjx5rUbkE21+Dcv4vg94+v5/AOopVsWCmVuZHN0cmVhbQplbmRvYmoKNDggMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCA3MiA+PgpzdHJlYW0KeJw1jLERwDAIA3um0Ag2WGDvk0tF9m9DfE4DLx0Pl6LBWg26giNwdan80SNduSlFl2POguFxql9IMUY9qCPj3sdPuV9wFhJ9CmVuZHN0cmVhbQplbmRvYmoKNDkgMCBvYmoKPDwgL0ZpbHRlciAvRmxhdGVEZWNvZGUgL0xlbmd0aCAyMTAgPj4Kc3RyZWFtCnicNVDLDUMxCLtnChaoFAKBZJ5WvXX/a23QO2ER/0JYyJQIeanJzinpSz46TA+2Lr+xIgutdSXsypognivvoZmysdHY4mBwGiZegBY3YOhpjRo1dOGCpi6VQoHFJfCZfHV76L5PGXhqGXJ2BBFDyWAJaroWTVi0PJ+QTgHi/37D7i3koZLzyp4b+Ruc7fA7s27hJ2p2ItFyFTLUszTHGAgTRR48eUWmcOKz1nfVNBLUZgtOlgGuTj+MDgBgIl5ZgOyuRDlL0o6ln2+8x/cPQABTtAplbmRzdHJlYW0KZW5kb2JqCjIxIDAgb2JqCjw8IC9CYXNlRm9udCAvRGVqYVZ1U2FucyAvQ2hhclByb2NzIDIyIDAgUgovRW5jb2RpbmcgPDwKL0RpZmZlcmVuY2VzIFsgMzIgL3NwYWNlIDQ2IC9wZXJpb2QgNDggL3plcm8gL29uZSAvdHdvIDUyIC9mb3VyIC9maXZlIC9zaXggL3NldmVuCi9laWdodCA2OSAvRSA3NyAvTSA4MiAvUiAvUyAvVCA4NiAvViA5NyAvYSAxMDAgL2QgL2UgMTAzIC9nIDEwNSAvaSAxMDggL2wKMTEwIC9uIC9vIDExNCAvciAxMTYgL3QgMTE4IC92IF0KL1R5cGUgL0VuY29kaW5nID4+Ci9GaXJzdENoYXIgMCAvRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250RGVzY3JpcHRvciAyMCAwIFIKL0ZvbnRNYXRyaXggWyAwLjAwMSAwIDAgMC4wMDEgMCAwIF0gL0xhc3RDaGFyIDI1NSAvTmFtZSAvRGVqYVZ1U2FucwovU3VidHlwZSAvVHlwZTMgL1R5cGUgL0ZvbnQgL1dpZHRocyAxOSAwIFIgPj4KZW5kb2JqCjIwIDAgb2JqCjw8IC9Bc2NlbnQgOTI5IC9DYXBIZWlnaHQgMCAvRGVzY2VudCAtMjM2IC9GbGFncyAzMgovRm9udEJCb3ggWyAtMTAyMSAtNDYzIDE3OTQgMTIzMyBdIC9Gb250TmFtZSAvRGVqYVZ1U2FucyAvSXRhbGljQW5nbGUgMAovTWF4V2lkdGggMTM0MiAvU3RlbVYgMCAvVHlwZSAvRm9udERlc2NyaXB0b3IgL1hIZWlnaHQgMCA+PgplbmRvYmoKMTkgMCBvYmoKWyA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMAo2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDYwMCA2MDAgNjAwIDMxOCA0MDEgNDYwIDgzOCA2MzYKOTUwIDc4MCAyNzUgMzkwIDM5MCA1MDAgODM4IDMxOCAzNjEgMzE4IDMzNyA2MzYgNjM2IDYzNiA2MzYgNjM2IDYzNiA2MzYgNjM2CjYzNiA2MzYgMzM3IDMzNyA4MzggODM4IDgzOCA1MzEgMTAwMCA2ODQgNjg2IDY5OCA3NzAgNjMyIDU3NSA3NzUgNzUyIDI5NQoyOTUgNjU2IDU1NyA4NjMgNzQ4IDc4NyA2MDMgNzg3IDY5NSA2MzUgNjExIDczMiA2ODQgOTg5IDY4NSA2MTEgNjg1IDM5MCAzMzcKMzkwIDgzOCA1MDAgNTAwIDYxMyA2MzUgNTUwIDYzNSA2MTUgMzUyIDYzNSA2MzQgMjc4IDI3OCA1NzkgMjc4IDk3NCA2MzQgNjEyCjYzNSA2MzUgNDExIDUyMSAzOTIgNjM0IDU5MiA4MTggNTkyIDU5MiA1MjUgNjM2IDMzNyA2MzYgODM4IDYwMCA2MzYgNjAwIDMxOAozNTIgNTE4IDEwMDAgNTAwIDUwMCA1MDAgMTM0MiA2MzUgNDAwIDEwNzAgNjAwIDY4NSA2MDAgNjAwIDMxOCAzMTggNTE4IDUxOAo1OTAgNTAwIDEwMDAgNTAwIDEwMDAgNTIxIDQwMCAxMDIzIDYwMCA1MjUgNjExIDMxOCA0MDEgNjM2IDYzNiA2MzYgNjM2IDMzNwo1MDAgNTAwIDEwMDAgNDcxIDYxMiA4MzggMzYxIDEwMDAgNTAwIDUwMCA4MzggNDAxIDQwMSA1MDAgNjM2IDYzNiAzMTggNTAwCjQwMSA0NzEgNjEyIDk2OSA5NjkgOTY5IDUzMSA2ODQgNjg0IDY4NCA2ODQgNjg0IDY4NCA5NzQgNjk4IDYzMiA2MzIgNjMyIDYzMgoyOTUgMjk1IDI5NSAyOTUgNzc1IDc0OCA3ODcgNzg3IDc4NyA3ODcgNzg3IDgzOCA3ODcgNzMyIDczMiA3MzIgNzMyIDYxMSA2MDUKNjMwIDYxMyA2MTMgNjEzIDYxMyA2MTMgNjEzIDk4MiA1NTAgNjE1IDYxNSA2MTUgNjE1IDI3OCAyNzggMjc4IDI3OCA2MTIgNjM0CjYxMiA2MTIgNjEyIDYxMiA2MTIgODM4IDYxMiA2MzQgNjM0IDYzNCA2MzQgNTkyIDYzNSA1OTIgXQplbmRvYmoKMjIgMCBvYmoKPDwgL0UgMjMgMCBSIC9NIDI0IDAgUiAvUiAyNSAwIFIgL1MgMjYgMCBSIC9UIDI3IDAgUiAvViAyOCAwIFIgL2EgMjkgMCBSCi9kIDMwIDAgUiAvZSAzMSAwIFIgL2VpZ2h0IDMyIDAgUiAvZml2ZSAzMyAwIFIgL2ZvdXIgMzQgMCBSIC9nIDM1IDAgUgovaSAzNiAwIFIgL2wgMzcgMCBSIC9uIDM4IDAgUiAvbyAzOSAwIFIgL29uZSA0MCAwIFIgL3BlcmlvZCA0MSAwIFIKL3IgNDIgMCBSIC9zZXZlbiA0MyAwIFIgL3NpeCA0NCAwIFIgL3NwYWNlIDQ1IDAgUiAvdCA0NiAwIFIgL3R3byA0NyAwIFIKL3YgNDggMCBSIC96ZXJvIDQ5IDAgUiA+PgplbmRvYmoKMyAwIG9iago8PCAvRjEgMjEgMCBSIC9GMiAxNiAwIFIgPj4KZW5kb2JqCjQgMCBvYmoKPDwgL0ExIDw8IC9DQSAwIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EyIDw8IC9DQSAxIC9UeXBlIC9FeHRHU3RhdGUgL2NhIDEgPj4KL0EzIDw8IC9DQSAwLjggL1R5cGUgL0V4dEdTdGF0ZSAvY2EgMC44ID4+ID4+CmVuZG9iago1IDAgb2JqCjw8ID4+CmVuZG9iago2IDAgb2JqCjw8ID4+CmVuZG9iago3IDAgb2JqCjw8IC9NMCAxMiAwIFIgL00xIDEzIDAgUiA+PgplbmRvYmoKMTIgMCBvYmoKPDwgL0JCb3ggWyAtOCAtOCA4IDggXSAvRmlsdGVyIC9GbGF0ZURlY29kZSAvTGVuZ3RoIDEzMSAvU3VidHlwZSAvRm9ybQovVHlwZSAvWE9iamVjdCA+PgpzdHJlYW0KeJxtkEEOhCAMRfc9RS/wSUtFZevSa7iZTOL9twNxQEzdNNC+PH5R/pLwTqXA+CQJS06z5HrTkNK6TIwY5tWyKMegUS3WznU4qM/QcGN0i7EUptTW6Hijm+k23pM/+rBZIUY/HA6vhHsWQyZcKTEGh98LL9vD/xGeXtTAH6KNfmNaQ/0KZW5kc3RyZWFtCmVuZG9iagoxMyAwIG9iago8PCAvQkJveCBbIC04IC04IDggOCBdIC9GaWx0ZXIgL0ZsYXRlRGVjb2RlIC9MZW5ndGggMTMxIC9TdWJ0eXBlIC9Gb3JtCi9UeXBlIC9YT2JqZWN0ID4+CnN0cmVhbQp4nG2QQQ6EIAxF9z1FL/BJS0Vl69JruJlM4v23A3FATN000L48flH+kvBOpcD4JAlLTrPketOQ0rpMjBjm1bIox6BRLdbOdTioz9BwY3SLsRSm1NboeKOb6Tbekz/6sFkhRj8cDq+EexZDJlwpMQaH3wsv28P/EZ5e1MAfoo1+Y1pD/QplbmRzdHJlYW0KZW5kb2JqCjIgMCBvYmoKPDwgL0NvdW50IDEgL0tpZHMgWyAxMCAwIFIgXSAvVHlwZSAvUGFnZXMgPj4KZW5kb2JqCjUwIDAgb2JqCjw8IC9DcmVhdGlvbkRhdGUgKEQ6MjAyMTAyMTYwMDQzNDkrMDMnMDAnKQovQ3JlYXRvciAoTWF0cGxvdGxpYiB2My4zLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcpCi9Qcm9kdWNlciAoTWF0cGxvdGxpYiBwZGYgYmFja2VuZCB2My4zLjMpID4+CmVuZG9iagp4cmVmCjAgNTEKMDAwMDAwMDAwMCA2NTUzNSBmIAowMDAwMDAwMDE2IDAwMDAwIG4gCjAwMDAwMTMxODMgMDAwMDAgbiAKMDAwMDAxMjQwNSAwMDAwMCBuIAowMDAwMDEyNDQ4IDAwMDAwIG4gCjAwMDAwMTI1OTAgMDAwMDAgbiAKMDAwMDAxMjYxMSAwMDAwMCBuIAowMDAwMDEyNjMyIDAwMDAwIG4gCjAwMDAwMDAwNjUgMDAwMDAgbiAKMDAwMDAwMDM5NyAwMDAwMCBuIAowMDAwMDAwMjA4IDAwMDAwIG4gCjAwMDAwMDE5NjkgMDAwMDAgbiAKMDAwMDAxMjY3NSAwMDAwMCBuIAowMDAwMDEyOTI5IDAwMDAwIG4gCjAwMDAwMDI3NzQgMDAwMDAgbiAKMDAwMDAwMjU2NiAwMDAwMCBuIAowMDAwMDAyMjUxIDAwMDAwIG4gCjAwMDAwMDM4MjcgMDAwMDAgbiAKMDAwMDAwMTk5MCAwMDAwMCBuIAowMDAwMDExMDI4IDAwMDAwIG4gCjAwMDAwMTA4MjggMDAwMDAgbiAKMDAwMDAxMDM2MyAwMDAwMCBuIAowMDAwMDEyMDgxIDAwMDAwIG4gCjAwMDAwMDM4NTkgMDAwMDAgbiAKMDAwMDAwNDAxMCAwMDAwMCBuIAowMDAwMDA0MTY5IDAwMDAwIG4gCjAwMDAwMDQ0NjkgMDAwMDAgbiAKMDAwMDAwNDg4MCAwMDAwMCBuIAowMDAwMDA1MDE2IDAwMDAwIG4gCjAwMDAwMDUxNTggMDAwMDAgbiAKMDAwMDAwNTUzNSAwMDAwMCBuIAowMDAwMDA1ODM1IDAwMDAwIG4gCjAwMDAwMDYxNTMgMDAwMDAgbiAKMDAwMDAwNjYxOCAwMDAwMCBuIAowMDAwMDA2OTM4IDAwMDAwIG4gCjAwMDAwMDcxMDAgMDAwMDAgbiAKMDAwMDAwNzUxMSAwMDAwMCBuIAowMDAwMDA3NjUxIDAwMDAwIG4gCjAwMDAwMDc3NjggMDAwMDAgbiAKMDAwMDAwODAwMiAwMDAwMCBuIAowMDAwMDA4Mjg5IDAwMDAwIG4gCjAwMDAwMDg0NDEgMDAwMDAgbiAKMDAwMDAwODU2MiAwMDAwMCBuIAowMDAwMDA4NzkyIDAwMDAwIG4gCjAwMDAwMDg5MzIgMDAwMDAgbiAKMDAwMDAwOTMyMiAwMDAwMCBuIAowMDAwMDA5NDExIDAwMDAwIG4gCjAwMDAwMDk2MTUgMDAwMDAgbiAKMDAwMDAwOTkzNiAwMDAwMCBuIAowMDAwMDEwMDgwIDAwMDAwIG4gCjAwMDAwMTMyNDMgMDAwMDAgbiAKdHJhaWxlcgo8PCAvSW5mbyA1MCAwIFIgL1Jvb3QgMSAwIFIgL1NpemUgNTEgPj4Kc3RhcnR4cmVmCjEzNDAwCiUlRU9GCg==\n", "image/svg+xml": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-02-16T00:43:49.740006\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.3.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Regularisation alpha\n", "alpha = 10 ** -5\n", "\n", "# Arrays for storing train and validation errors\n", "train_errors = []\n", "valid_errors = []\n", "\n", "for D in range(10):\n", " \n", " # regularization term = lambda * (indentity matrix)\n", " reg_term = alpha * np.eye(D + 1)\n", " \n", " # Design matrix at training data\n", " phi = np.array([[x ** d for d in range(D + 1)] for x in x_train])\n", " \n", " # Solve for the maximum a posteriori weights\n", " w = np.linalg.solve((phi.T).dot(phi) + reg_term, (phi.T).dot(y_train))\n", " \n", " # Mean predictions at the training data\n", " y_pred_train = phi.dot(w)\n", " \n", " # Calculating RMS error of the training data\n", " train_rmse = np.mean((y_pred_train - y_train) ** 2) ** 0.5\n", " \n", " # Compute mean model predictions at the validation data\n", " phi_pred = np.array([[x ** d for d in range(D + 1)] for x in x_valid])\n", " y_pred = phi_pred.dot(w)\n", " \n", " # Calculating RMS error of the validation data\n", " valid_rmse = np.mean((y_pred - y_valid) ** 2) ** 0.5\n", " \n", " train_errors.append(train_rmse)\n", " valid_errors.append(valid_rmse)\n", " \n", "# Plot training and validation errors\n", "plt.figure(figsize=(8, 3))\n", "plt.plot(np.arange(0, 10),\n", " train_errors,\n", " color='blue',\n", " marker='o',\n", " label='Train')\n", "\n", "plt.plot(np.arange(0, 10),\n", " valid_errors,\n", " color='red',\n", " marker='o',\n", " label='Validation')\n", "\n", "# Format plots\n", "plt.title('Training and validation error', fontsize=18)\n", "plt.xlabel('$D$', fontsize=16)\n", "plt.ylabel('RMSE', fontsize=16)\n", "plt.xticks(fontsize=12)\n", "plt.yticks(np.linspace(0, 1, 5), fontsize=12)\n", "plt.gca().legend()\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "hide_input": true }, "source": [ "As expected, the training error steadily decreases with $D$. The test error again reaches a minimum for $D = 3$, but unlike in the unregularised case, it doesn't explode for large $D$, because the regularization term prevents the weights from becoming large. To verify this point, let's check the values of the weights:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "hide_input": true, "tags": [ "hide-input" ] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
D\\(w_0\\)\\(w_1\\)\\(w_2\\)\\(w_3\\)\\(w_4\\)\\(w_5\\)\\(w_6\\)\\(w_7\\)\\(w_8\\)
00.230.000.000.000.000.000.000.000.00
10.52-0.780.000.000.000.000.000.000.00
20.430.11-0.920.000.000.000.000.000.00
30.028.73-26.8618.380.000.000.000.000.00
40.057.49-18.782.579.130.000.000.000.00
50.057.49-18.782.579.110.010.000.000.00
60.047.70-18.950.629.986.50-5.560.000.00
70.047.65-17.99-1.478.328.662.44-7.500.00
80.047.45-16.79-2.526.318.065.13-0.43-7.24
\n", "
" ], "text/plain": [ "D \\(w_0\\) \\(w_1\\) \\(w_2\\) \\(w_3\\) \\(w_4\\) \\(w_5\\) \\(w_6\\) \\(w_7\\) \\\n", "0 0.23 0.00 0.00 0.00 0.00 0.00 0.00 0.00 \n", "1 0.52 -0.78 0.00 0.00 0.00 0.00 0.00 0.00 \n", "2 0.43 0.11 -0.92 0.00 0.00 0.00 0.00 0.00 \n", "3 0.02 8.73 -26.86 18.38 0.00 0.00 0.00 0.00 \n", "4 0.05 7.49 -18.78 2.57 9.13 0.00 0.00 0.00 \n", "5 0.05 7.49 -18.78 2.57 9.11 0.01 0.00 0.00 \n", "6 0.04 7.70 -18.95 0.62 9.98 6.50 -5.56 0.00 \n", "7 0.04 7.65 -17.99 -1.47 8.32 8.66 2.44 -7.50 \n", "8 0.04 7.45 -16.79 -2.52 6.31 8.06 5.13 -0.43 \n", "\n", "D \\(w_8\\) \n", "0 0.00 \n", "1 0.00 \n", "2 0.00 \n", "3 0.00 \n", "4 0.00 \n", "5 0.00 \n", "6 0.00 \n", "7 0.00 \n", "8 -7.24 " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Arrays to store the maximuum a posteriori weights\n", "weights = []\n", "\n", "# Regularisation alpha\n", "alpha = 10 ** -4 \n", "\n", "for D in range(9):\n", " \n", " # Design matrix at training data\n", " phi = np.array([[x ** d for d in range(D + 1)] for x in x_data])\n", " \n", " # regularization term = lambda * (indentity matrix)\n", " reg_term = alpha * np.eye(D + 1)\n", " \n", " # Solve for the maximum likelihood weights\n", " w = np.linalg.solve((phi.T).dot(phi) + reg_term, (phi.T).dot(y_data))\n", " \n", " # Pad with zeroes for unused weights - this is used for the table below\n", " w = np.pad(w, (0, 9 - w.shape[0]), 'constant', constant_values = (0, 0))\n", " weights.append(w)\n", "\n", "# Row and column names for table\n", "row_names = [d for d in range(9)]\n", "column_names = ['\\(w_{}\\)'.format(d) for d in range(9)]\n", "\n", "# Create table and show it\n", "table = pandas.DataFrame(weights, row_names, column_names).round(2)\n", "table.columns.name = \"D\"\n", "table" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The weights are significantly decreased by regularization. **You can change $\\boldsymbol{\\alpha}$ to see how the weights are affected by the size of the regularization term**." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary\n", "\n", "Overfitting causes the weight estimates in a linear-in-the-parameters regression model to become extreme. Extreme settings of the weights can be discouraged by penalising their magnitude. This can be interpreted as placing a prior distribution that encodes knowledge before observing the data that the weights are likely to be small in magnitude. In the next section we will look at [bayesian](regression_bayesian.ipynb) approach to regression." ] }, { "cell_type": "markdown", "metadata": { "hide_input": true }, "source": [ "## Questions\n", "\n", "
    \n", "
  1. (a) Look at the plot of the functions sampled from the probabilistic model, \\(\\mathbf{w}^{(m)} \\sim \\mathcal{N}(\\mathbf{0},\\sigma_{\\mathbf{w}}^2 \\mathrm{I})\\) and \\(f^{(m)}(x)=\\boldsymbol{\\phi}(x)^\\top \\mathbf{w}^{(m)}\\) in the case of i) polynomial and ii) Gaussian basis functions. Alter the code to draw a large number of samples. Averaging over samples, qualitatively, how does the mean and variance of the distribution over functions depend on \\(x\\)?

    (b) Derive an analytic expression for the mean and the variance of the function, that is \\(\\mathbb{E}_{\\mathbf{w}}(f_{\\mathbf{w}}(x))\\) and \\(\\mathbb{E}_{\\mathbf{w}}(f_{\\mathbf{w}}^2(x)) - \\mathbb{E}_{\\mathbf{w}}(f_{\\mathbf{w}}(x))^2\\). Compare what happens to these statistics for the two models in regions far away from the origin.

    (c) Do these results shed light on when it is appropriate to use i) polynomial basis functions and ii) Gaussian basis functions?\n", " \n", "
    \n", "Answer\n", "
    \n", "\n", "The mean of the distribution over functions is defined as:\n", "\n", "$$\\begin{align}\n", "\\mathbb{E}_{\\mathbf{w}}(f_{\\mathbf{w}}(x)) = \\int f_{\\mathbf{w}}(x) p(\\mathbf{w}) \\mathrm{d} \\mathbf{w}.\n", "\\end{align}$$\n", "\n", "Substituting in the definition of the function $f_{\\mathbf{w}}(x) = \\sum_{d=0}^D w_d \\phi_d(x)$:\n", "\n", "$$\\begin{align}\n", "\\mathbb{E}_{\\mathbf{w}}(f_{\\mathbf{w}}(x)) = \\int \\sum_{d=0}^D w_d \\phi_d(x) p(\\mathbf{w}) \\mathrm{d} \\mathbf{w} = \\sum_{d=0}^D \\mathbb{E}_{\\mathbf{w}} (w_d) \\phi_d(x) = 0\n", "\\end{align}$$\n", "\n", "So the average function is one which takes the value $0$ everywhere. The variance is defined as\n", "\n", "$$\\begin{align}\n", "\\mathbb{E}_{\\mathbf{w}}(f^2_{\\mathbf{w}}(x)) = \\int f^2_{\\mathbf{w}}(x) p(\\mathbf{w}) \\mathrm{d} \\mathbf{w}\n", "\\end{align}$$\n", "\n", "where we have used the fact that the mean is zero. Substituting in the definition of the function again yields\n", "\n", "$$\\begin{align}\n", "\\mathbb{E}_{\\mathbf{w}}(f^2_{\\mathbf{w}}(x)) = \\int \\sum_{d=0}^D \\sum_{d'=0}^D w_d \\phi_d(x) w_{d'} \\phi_{d'}(x) p(\\mathbf{w}) \\mathrm{d} \\mathbf{w} = \\sum_{d=0}^D \\mathbb{E}_{\\mathbf{w}}(w^2_d) \\phi^2_d(x)\n", "\\end{align}$$\n", "\n", "where we used the fact that the weights are independent in the prior. Therefore\n", "\n", "$$\\begin{align}\n", "\\mathbb{E}_{\\mathbf{w}}(f^2_{\\mathbf{w}}(x)) = \\sigma_{\\mathbf{w}}^2 \\sum_{d=0}^D \\phi^2_d(x).\n", "\\end{align}$$\n", "\n", "Notice that in the case of polynomial regression the variance grows with the magnitude of the inputs $|x|$. That is, the functions will typically blow up once the magnitude of $x$ is large enough. For Gaussian basis functions the variance shrinks to zero away from the basis functions. Both behaviours can be problematic for extrapolation. \n", "\n", "
    \n", "
    2. \n", "
\n", "\n", "
  1. The plot below shows a dataset which has been fit with a linear model using the MAP method. Is the fit satisfactory? How would you modify the model to improve the fit?
    2. \n", "
" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "hide_input": true, "tags": [ "center-output", "remove-input" ] }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "\n", "\n", " \n", " \n", " \n", " \n", " 2021-02-16T00:42:08.135463\n", " image/svg+xml\n", " \n", " \n", " Matplotlib v3.3.3, https://matplotlib.org/\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# For this example we sample the data using a different noise model,\n", "# Student's t-distribution. This distribution has heavy tails, meaning\n", "# that it produces outliers more frequently. Modelling heavy-tail data\n", "# with a Gaussian noise model will result in poor fits because the Gaussian\n", "# distribution has ligher tails and is therefore more sensitive to outliners.\n", "\n", "# Set random seed\n", "np.random.seed(1)\n", "\n", "# Order of polynomial\n", "D = 1 \n", "\n", "# Number of data points per model\n", "N = 10 \n", "\n", "# Regularisation alpha\n", "alpha = 10 ** -4\n", "\n", "# Prior variance of weights\n", "var_w = 1000 \n", "\n", "# Observation noise variance\n", "var_y = 0.01\n", "\n", "\n", "# =============================================================================\n", "# First we sample the data\n", "# =============================================================================\n", "\n", "# Input locations of true function / fitted function\n", "x_plot = np.linspace(0, 1, 100)\n", "\n", "# Polynomial basis functions\n", "phi_pred = np.array([[x ** d for d in range(D + 1)] for x in x_plot]) \n", " \n", "# Input locations of data\n", "x_samp = np.random.uniform(0, 1, N) \n", "\n", "# Polynomial basis functions\n", "phi_samp = np.array([[x_ ** d for d in range(D + 1)] for x_ in x_samp]) \n", " \n", "# Sample weights\n", "w = np.random.normal(0, var_w ** 0.5, D+1)\n", "\n", "# Plot function at input locations\n", "f_true = phi_pred.dot(w)\n", "\n", "# Generate data, corrupted with Student-t heavy tailed noise\n", "y_samp = phi_samp.dot(w) + var_y ** 0.5 * np.random.standard_t(0.5, N)\n", "\n", "\n", "# =============================================================================\n", "# Next we fit the data\n", "# =============================================================================\n", "\n", "# Design matrix at sampled data locations\n", "phi = np.array([[x ** d for d in range(D + 1)] for x in x_samp])\n", "\n", "# regularization term = alpha * (indentity matrix)\n", "reg_term = alpha * np.eye(D + 1)\n", "\n", "# Solve for maximum a posteriori weights\n", "w = np.linalg.solve((phi.T).dot(phi) + reg_term, (phi.T).dot(y_samp))\n", "y_plot = phi_pred.dot(w)\n", " \n", "# Figure on whicht to plot data and linear model\n", "plt.figure(figsize=(8, 3))\n", "\n", "# Plot sampled data\n", "plt.scatter(x_samp,\n", " y_samp,\n", " marker='x',\n", " color='red')\n", "\n", "# Plot fitted linear fit\n", "plt.plot(x_plot, y_plot, color='k')\n", "\n", "# Format plots\n", "plt.title('Training and validation error', fontsize=18)\n", "plt.xlabel('$D$', fontsize=16)\n", "plt.ylabel('RMSE', fontsize=16)\n", "plt.xticks(np.linspace(0, 1, 5), fontsize=12)\n", "plt.yticks(np.linspace(0, 200, 5), fontsize=12)\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "Answer\n", "
\n", "The dataset contains an outlier which is causing the best fit line to deviate from the bulk of the data. This will likely lead to bad predictions on test data.\n", " \n", "The assumption of Gaussian noise (or squared error cost) means that outliers will have significant effect. \n", "\n", "Instead, an alternative observation noise model should be used e.g. a [Laplace distribution](https://en.wikipedia.org/wiki/Laplace_distribution)\n", " \n", "$$\\begin{align}\n", "p(w_d | \\sigma_{w}) = \\frac{1}{2 \\sigma_{w}}\\exp(-|w_d|/\\sigma_{w}),\n", "\\end{align}$$\n", " \n", "which results in a cost on the absolute value when using MAP or maximum-likelihood fitting, or a [Student's t-distribution](https://en.wikipedia.org/wiki/Student%27s_t-distribution)\n", " \n", "$$\\begin{align}\n", "p(w_d | \\sigma_{w}) \\propto (1 + \\frac{v}{v-2} |w_d|^2/\\sigma^2_{w})^{- (v+1)/2},\n", "\\end{align}$$\n", " \n", "which was in fact used to generate the data above.\n", "\n", "These alternative observation noise models do not result in analytic fits when using maximum likelihood or MAP estimation. However optimisation based approaches can be used. This price is often worth paying as real datasets typically contain outliers and non-Gaussian noise models can make the predictions substantially more robust.\n", "\n", "
\n", "
\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 4 }