preprocessing dataset in python

_loganthx answered on May 6, 2023 Popularity 8/10 Helpfulness 1/10

answer preprocessing dataset in python

related preprocessing data python

preprocessing dataset in python

Comment

'''Preprocessing dataset: (Suppose your .py file is inside Dataset folder
and Labels are folder's names: Example => C:\Users\Dataset\Cats\cat1.png)'''
import os, cv2 as cv, numpy as np, random

DIR = os.getcwd()
Labels = [i for i in os.listdir(DIR) if not('.') in i]
data, Cut, X, y, New_W, New_H = [], [], [], [], 50, 50

for Root, dir, files in os.walk(DIR):
	if Root!=DIR: Cut.append(len(os.listdir(Root))) # Normalize samples
      
for R, d, f in os.walk(DIR):
	if R!= DIR: # Loads img as grayscale, resizing and appending => [img, label]
		lbl = [(Labels[id]) for id, _ in enumerate(Labels) if Labels[id] in R]
		Imgs = [(cv.imread(f'{R}\\{img}', 0)/255.0) for img in os.listdir(R)]
		for img in Imgs[:min(Cut)]: 
			data.append([cv.resize(img, (New_W, New_H)), lbl[0]])
		print(f'Num of imgs for {lbl} label ==> {len(Imgs[:min(Cut)])}')
random.shuffle(data)

for img, label in data: 
	X.append(img)
	y.append(label)
np.array(X).reshape(-1, New_W, New_H, 1)   #1 if gray; 3 if rgb
  
np.save(f"{DIR}\\X.npx", X) # Save file in DIR directory
np.save(f"{DIR}\\y.npx", y)

Popularity 8/10 Helpfulness 1/10 Language python

Source: Grepper

Tags: dataset python

Link to this answer
Share Copy Link

Contributed on May 06 2023

_loganthx

0 Answers Avg Quality 2/10

Closely Related Answers

preprocessing data python

Comment

# Inspect dataset
df.head()
df.info()
df.describe() # Summary stats

# DEAL WITH MISSING VALUES
df.drop([1, 2, 3]) # Drop specific rows
df.dropna(thresh=2) # keep at least 2 non-missing values in each row
df.dropna(subset=['C']) # Drop missing values of specified column

# Convert column types
df["C"] = df["C"].astype("float")

# Verify class imbalance
y.value_counts()

# Split into training and testing data (Consider class imbalance)
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(X, y, stratify=y, test_size=0.2, random_state=42)

# STANDARDIZE DATASET
df.var() # Detect high variance difference in columns are candidates of log normalization

# FEATURE ENGINEERING
# TEXT PROCESSING (cleaning / vectorizing / regular expression )

Popularity 6/10 Helpfulness 5/10 Language python

Source: Grepper

Tags: python

Link to this answer
Share Copy Link

Contributed on Jan 16 2024

Innocent Iguana

0 Answers Avg Quality 2/10

preprocessing dataset in python

Contents

More Related Answers

preprocessing dataset in python

Closely Related Answers

preprocessing data python

Grepper

Documentation

Social

Legal

Contact

Oops, You will need to install Grepper and log-in to perform this action.