Hyperparameter Tuning

Namaskar I am Kushal from learnvern.

You are welcome to the course on machine learning and this tutorial is the continuation of the previous session , so let’s proceed further.

In today’s tutorial we will study hyperparameter tuning. Till now we have explored a lot of algorithms and for all of them we did examples also with different datasets, and while doing so we encountered some parameters for which we had to keep default values or at times we had to tweak the values also, so these are those parameters that are called hyperparameters.Now these hyperparameters, hyper parameter , so these parameters are to be set before the algorithm or before starting of the model. Now because we have to set them before they are run so we should know their values beforehand but the reality is that till the time you do not run the model , you will not get outputs, you will not be able to observe the outputs till then you will not be experience the hyperparameter properly.So for this reason this hyperparameter tuning is an iterative process and this keeps on going and it may be so that for one particular dataset hyperparameter value may be different and for the other dataset hyperparameter value may be different. This is quite possible and it can happen. So for example we now have logistic regression , so LR, OK logistic regression, so in logistic regression we have L1 and L2 , here L1 and L2 for regularization, so what is this L1 and L2 regularization, this is hyperparameter tuning, now when I talk of LR meaning learning rate , so when we talk about neural networks, it's part of deep networks, neural networks, neural networks so there we adjust the learning rate and that also we have to set in advance and out of experience, gradually we standardize its value for some model. Then after this like there is C , then there is sigma , then we use in svm , so in support vector machines and in the same manner we have hyperparameter K that we use in KNN , so these are some examples of hyperparameters that we use in different algorithms. Now I will give you an example that how we can tune these hyperparameters, and as I said that to tune these hyperparameters what we do is we test it with different values and that if you do manually that is if with your hand you will change the value of LR or repeatedly regularize L1 and L2 or change C then it will be very time consuming, so because of this we can use grid search or random search, so for this grid search and random search we will see the implementations now. So grid, means you will get a kind of list or grid and in that grid we will insert values in advance for the hyperparameters. So in gridsearch it will first check accuracy for a value , then for the second one, then for the third one, then for fourth one , so the best value that it gets it will notify that for all the values this is the best output. So let us start with grid search, OK. Now here let’s start with grid search OK, so from sklearn dot linear model import logistic regression ,logistic regression and from sklearn, sklearn dot model selection import gridsearch, gridsearch OK. Now what will be our hyperparameters , so what happened here, gridsearch , gridsearch cv, ok it's gridsearch cv so grid S E A OK, gridsearch cv , OK done. So this is called gridsearch CV, Ok. Now here what we will do is, create hyperparameters for the grid , so create hyperparameter grid OK, OK so here C underscore S P A C E , C underscore space is equal to np dot L O G logspace OK ,so np dot logspace and here -5 comma 8 comma 15 , so here if you see C underscore S P A , ok we had not imported numpy here , so let me do that so import, import numpy as np, numpy np , let’s import it and create this and here press backspace and all this data we will put in brackets , and now let’s execute and yes it got executes, so now here C underscore space and let’s see what do we have in C underscore space , so these are the values that we have in C underscore space , these are the values. So, these values we will now use and just watch, lt;s create a dictionary , P A R A M param underscore G R I D grid is equal to and here , yes C and colon C underscore space , OK.So this we have created a parameter grid and this grid we will use ahead OK. So we will use it is, we will move further and create a model , so what is the logistic relation to create a model, so LR is equal to logistic regression and here M A X max iteration , w will now initialize , I give max iteration is equal to 1000 and after that now, we will initialize the grid search , so L R G R I D lrgrid is equal to ,here G R I D gridserach CV and inside this we will put, what shall we put, in this LR ,we will put the model that we had created and pass it and in this P A R A M paramgrid that we have created P A R A M param underscore G R I D grid and then next I will put is CV for cross validation, so cv is equal to 5 OK, alright. Now L R G R I D lrgrid dot F I T fit and here we have X and then Y and let’s execute this and here what is X here , so X is the input data we have , so input data till now I have not taken so let me take it , so we will take input data here , so here itself , from S K L E A R N learn dot datasets and import so here we will import L O A D load underscore I R I S iris OK, and after this I will create a dataset , so it will be something like this I R I S is equal to L O A D load iris and from that I want data so, X, X value would be I R I S iris dot data, this will be the value of X OK, so this is the X that I have created here , now this X we will pass here but this is logistic regression so we will require Y also, so this is our X and for Y , Y is equal to data , data we don’t have , so I R I S iris dot L A B E L labels so labels will come inside this , so they are not labels but targets actually, so iris dot T A R G E T S, target, target , OK , so targets will come in Y and here we will now execute it , we have X and Y and we are executing on iris dataset , so you can see here that data has fit here OK, so now we will print the results here OK, so let’s print the best parameters , so print B E S T best and P A R A M S params, we now print it , so what will come in best params, in best params it will be logs, L O G log, L R G R I D lrgrid dot B E S T best params , so we can see the best params , so these are the best parameters, which one C is equal to 31 so when C is 31 that is our best parameter and after this we will print one more, print and here let’s see the best score means let’s see what is the score B E S T Best S C O R E score , so best score that we have got is L R G R I D dot best score , so this is our best score OK, so 0.98, this is our best score.So how did we do this, we did it by parameter tuning, hyperparameter tuning ,so this was grid search and similar to gridsearch the other algorithm that we have is random search. So the basic difference between the two is that in grid there is a sequential approach to proceed one by one but in random search it takes some fixed number of hyperparameters and makes a grid by selecting values randomly , like it picks one value, then the other value , it will not go sequentially as seen earlier. So when it is done randomly then there can be a combination, means chances of getting a better combination is quick and that is why it is called random search or randomized search , S E A R C H search CV, so we will use this and see, let us use this , so in this everything will be as it is OK, everything will be as it is and something extra if I want then that I will take, from S C I P Y scipy dot stats import R A N D rand I N T int randint and here we will take, OK let us apply this on decision tree classifier so that it will be a little different so, from S K L E A R N sklearn T R E E tree OK, tree import, we will import D E C decision tree classifier OK, this we have imported , now after this next is from OK from S K L E A R N sklearn dot model selection import , from here we will import R A N D O M random randomized , till now, OK it has come searchCV, so this we have imported OK, so we have the dataset already with us so we do not have to bother about it , so we will just create a grid here and in this grid we will require many parameters because we are using decision tree, so P A R A M parameters underscore D I S T dist is equal to and here the first thing we will take is M A X max underscore D E P T H depth and we will give the value 3 comma none to max depth, OK.Now after this let’s move further, so next is M A X max underscore F E A T U R E S features right, now, for this also we have R A N D rand I N T int 1 to 9, OK, and next one we have is criteria so C R I T E R I A OK, so this cou;ld be either gini or entropy so we will pass a list here , so we have put ginni and after ginni here we have put entropy , entropy OK, so this we have created a parameter grid, parameter grid OK. Now let’s move further and now let’s make a model , so, here we have to make a tree model so here dt is equal to D E C I S I O N decision tree qualifier and this we have made a model and after this here we write decision tree and R A N D O M random and this is R A N D O M random yes randomized, here it is and this will work for us and in this we have to feed model and dt is the name of our model , and after model what we have to do is, feed parameters so P A R A M param and we have dist so paramdist we will use and after this we will keep the value of CV to , let us keep it 5, so now we will pass it and so let’s execute this , so here dt, I had to feed random here ,so R A N D O M random OK, now we will move further and dt underscore random so this is arranged in datafit so we will fit the data here and we will pass X and Y in it. So we have passed X and Y here. So here we have invalid parameters in X and Y OK, so max features, M A X max , OK there is a underscore feature with max and the one that I have written is, where is it yes here it is, so let’s make it underscore and let’s execute it again , so wherever some mistake is there, whenever you do a mistake just cross check it like this , so it is giving warning but it has created OK, but it is giving some value error also, OK, I have given it 1 to 9but we have 0,1,2,3 we have 4 only so that is why it was giving a problem , so here I have reduced the features, so now you can see that this is done OK. So you know that from highest area we have all the features so those features I have given here, so the random search CV is also initialized here and now what we have to do is we have to basically fit our data here and that data fit also we have done OK, so now we can check here that exactly what are the best parameters that we got this time. Sop here I do print, print and B E S T best, B E S T best P A R A M E T E R S parameters , so in best parameters we have dt R A N random ,OK dt underscore random , so dot here and params , so this we will get printed , so these are the best parameters we have gini, max depth is 3 and max features is 3 and these are the best parameters with us OK, as per the algorithm we have run just now and what is the best score here that also we can see , so by putting a comma here and doing a slash N , B E S T best and S C O R E score , we will give, and here dt underscore random , so dt underscore random dot , so this is the best parameter and that which will be with us will be the best score that we will just display , best score , so just watch here 0.97 , can you see this 0.97, this is the best score. So this is how we do hyper parameter tuning where we change the values of hyperparameters and to change them also we saw two packages, random , randomized search cv and before that we saw grid search cv. So you must apply this with the rest of the algorithms and try what kind of results you get to observe it. So friends, let's conclude here today. We end today’s session here and in the next session we will see the parts ahead. So keep learning, remain motivated, thank you.

If you have any questions or comments related to this course then you can post them beneath this video using the discussion button and by means of this you can discuss this course with other learners too.