model validation

Innocent Iguana answered on July 19, 2022 Popularity 9/10 Helpfulness 3/10

answer model validation

related model validation

model validation

Comment

- determines MODEL PERFORMANCE
- Way to determine that the model is performing in the as expected
- common way : Accuracy in unseen data (test data) is same as seen data (train data)
- Goal : Choose right model, right parameters, right accuracy metrics, high accuracy on new data
- Split dataset : split dataset into 80-20 split. train with 80% and test with 20%
- validation dataset : Split the train again into training and validation dataset with 75-25 split.
- tune hyperparameters : You need validation dataset for tuning hyperparameters
- Accuracy metrics for assessing model performance:
    - Regression :
        - Good rule of thumb : make your y into percentage, then the metric will also generate percentage value
        - Mean absolute error (MAE) : Treats all points equally
        - Mean Squared error (MSE) : Issues penalty for large difference
    - Classification: 
        - Measured from confusion matrix : `cm[<true_category_index>, <predicted_category_index>]`
        - precision : True positives out of all predicted positive values
            - Used when you do not want to over predic-positive values (you need assurity for cancer paitent test)
        - recall : True positives out of all real positive values
            - Used when you cannot afford to miss any positive values (you need assurity for cancer paitent test)
        - accuracy: overall ability of model to predict correct class
        - Specificity 
        - F1-score
- Cross validation : 
    - gold standard for model validation
    - Every sample produce a slightly different result.
    - gets rid of bias result that occurs due to sampling by taking the average of many samples without replacement
    - breaks training data further into training and validating set for producing compact training result
    - LOOCV : trains with n-1 data and validates with only 1 data point (generally used for less data. computationally expensive). During cross validation, put `cv=len(X.shape[0])`
- Overfitting : 
    - accuracy in test data is lower than accuracy in train data (High variance on train data)
    - model pays too much attention to the details of training data and also learns noise 
    - model becomes complex, more data may be needed
- Underfitting : 
    - accuracy in both train and test data is lower since the model is too simple to capture pattern (High bias, low variance)
    - model fails to find relationship between the data and the response target value
    - model is too simple
- BIAS VARIANCE TRADEOFF : 
    - variance : model pays too much attention to the details of training data and also learns noise
    - bias : models fails to learn the details
- Regression model : Target is a continuous value
- Classification model : Target is a discrete value / category
- hyper-parameter tuning : 
    - 2 types of parameters.
    - parameters that do not exist before training the model : you cannot change those (eg: co-efficient of linear regression)
    - parameters of the model that can be manually selected to see what parameters might produce optimal result. (eg: maximum depth of a tree)
    - GridSearch: Brute force on all available combination of hyper-parameters
    - Random Search: randomly selecting combination from available hyper-parameters
    - Bayesian Search : use pass test on each step for the next run

Popularity 9/10 Helpfulness 3/10 Language python

Source: Grepper

Tags: model-validation python

Link to this answer
Share Copy Link

Contributed on Jan 17 2024

Innocent Iguana

0 Answers Avg Quality 2/10

Closely Related Answers

model validation

Comment

const userSchema = new Schema({
  phone: {
    type: String,
    validate: {
      validator: function(v) {
        return /\d{3}-\d{3}-\d{4}/.test(v);
      },
      message: props => `${props.value} is not a valid phone number!`
    },
    required: [true, 'User phone number required']
  }
});

const User = db.model('user', userSchema);
const user = new User();
let error;

user.phone = '555.0123';
error = user.validateSync();
assert.equal(error.errors['phone'].message,
  '555.0123 is not a valid phone number!');

user.phone = '';
error = user.validateSync();
assert.equal(error.errors['phone'].message,
  'User phone number required');

user.phone = '201-555-0123';
// Validation succeeds! Phone number is defined
// and fits `DDD-DDD-DDDD`
error = user.validateSync();
assert.equal(error, null);

Popularity 9/10 Helpfulness 3/10 Language javascript

Source: mongoosejs.com

Tags: javascript j

Link to this answer
Share Copy Link

Contributed on Jul 19 2022

Light Leopard

0 Answers Avg Quality 2/10

model validation

Contents

More Related Answers

model validation

Closely Related Answers

model validation

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.