Basics of Regularization and Optimization

Hello, I am (name) from LearnVern.

In the last tutorial on machine learning we had seen LDA meaning Linear Discriminant Analysis and now we will see regularization and optimization. So let us understand what it is. Regularization is a technique for improvement on a model of machine learning. Now what does it do exactly, from its description or definition you can understand that it tries to reduce the coefficient estimate or shrink it down to zero but just by seeing it like this we will not be understand that what will actually happen by bringing the coefficient estimate to zero , so let us understand that. So, it basically tries to minimize the problem of overfitting , that we reduce it.

Now how we will minimize the overfitting problem is that when in a dataset we try to fit over a complex curve then it becomes overfitting. This I will explain to you diagrammatically also, so to reduce it normalization works. Now watch here, there are three terms under fitting, appropriate fitting and overfitting. What Is underfitting, too simple to explain the variance means in underfitting it is not able to differentiate in a correct manner that for the line of fit or the regression line if their is any data close to it then it will have difficulty in clearly identifying, that to which class the data belongs or what should be it’s value and on similar line is overfitting, overfitting is too good to be true. meaning that it will work well on your training data but won’t work well on testing data meaning it will give wrong predictions.

So we are trying here to do an optimized appropriate fitting. So let us see how it happens. Here you will see, we will start from the beginning , just see , this is our dataset and I have drawn a blue color line here and this blue color line is separating both of them and now you can see that this line will wrongly classify this circle and this circle , so something is wrong with it. so I will say it is underfitting, now you see this, here I have with caution drawn this curve such that there is no scope of mistake meaning too good to be good, meaning it will give a perfect answer, now see this is a triangle , this is a triangle but if you bring some new data then this won’t work with it.

 so this is the problem with overfitting. Now if in a manner like this I draw a curve in a general and optimized manner which might not give correct answers to 100 percent but in most cases it will be able to predict for training data also and testing data also. so to achieve this appropriate fitting we have to do regularization. Now, let us proceed ahead , so in regularization we have some techniques like Ridge regression, Lasso regression and dropout, so we will see how they exactly work. Now the next sub topic is optimization. optimization is also a similar concept and what happens here is, here we basically try that the objective function , the objective function is able to achieve minimum and maximum elevation , so we try to find some inputs like this so that this condition is fulfilled.Now when do we do it, this we do at the time of data preparation, at the time of hyperparameter tuning and at the time of model selection. Hyperparameter tuning may be new to you , but hyperparameter tuning we have seen in one or two algorithms like in reinforcement learning , so in that, what are the steps we should take ahead, and what weightage should we give , so that weightage that we give becomes a hyperparameter. Now how high or low we should keep it, so normally the hyperparameter is kept to a low value , and in low also how much we must keep that we tune. At the time of model selection also we execute again and again and in that also we test that which is the optimized model, OK.

The next thing is that when we do function optimization so it’s purpose it to minimize the error , now error can have another form , cost , because of so many iterations the cost increase, because of many iterations that take place or it takes a lot of time or loss , so cost and loss are similar things and so what we do is we minimize then, the error also, cost also and loss also is minimized. Now, let’s talk about hyperparameter tuning, so hyperparameter tuning is that we choose optimal hyperparameters, OK set of optimal hyperparameters. now this optimal hyperparameters can be of what type , these optional hyperparameters can be like let’s assume that we have an equation A is equal to B plus C, the answer from A is equal to B plus C is always 10 less than the real answer , it is always 10 less so what we will do is we will add a hyperparameter C or add D , so now A is equal to B plus C plus D, now we started getting the perfect answer , so this plus D that we did was a hyperparameter , so to set this value is called tuning so now see, this is the definition given here that, the hyperparameter is that value that we have to set , but how this value will come so we will set another parameter , a learning rate we will use such that every time that value is increased by 0.5, 0.5, 0.5 so when you increase it by 0.5 that value will come in 20 times isn’t it, in 20 times or it will increase till 10 , so this hyperparameter controls the complete learning process and everytime it calculates the error . So as and when we see the error that it is becoming less or more , it is becoming less or more, so wherever error is less or cost is less or loss is less there we stop or that many iterations we do , so this is how hyperparameters are run, OK. So we have seen that hyperparameters control the overall behavior , the overall behavior of the learning process. Although we have the learning function, the complete behavior is controlled by hyperparameters. So let's conclude hyperparameter here , that finally we want an optimal combination of hyperparameters so that we can minimize the loss function and we achieve better results. So friends let us conclude here today, we will end today’s session here and the next parts to this we will see in the next session , so keep learning, remain motivated, thank you.

If you have any questions or comments related to this course then you can post them beneath this video using the discussion button and by means of this you can discuss this course with other learners too.