Data Handling & Preprocessing

Anatomy of a Dataset: Features, Labels, and Splits

Before we start cleaning, we need to understand the basic terminology for our data's structure and purpose. Let's use our familiar house price prediction example.

Features (Inputs / Predictors)

A **feature** is an individual measurable property or characteristic of a data point. In our Pandas DataFrame, these are the columns. They are the **inputs** we use to make a prediction. For our house dataset, the features would be:

• Number of Bedrooms
• Square Footage
• Age of the House
• Location (e.g., ZIP code)

Label (Output / Target)

The **label** (or target) is the value we are trying to predict. It's the "answer" we want our model to learn to output. In our example, the label is the **Price** of the house. In a supervised learning problem, our dataset must contain this label column for the model to learn from.

Training Data vs. Testing Data 🧑‍🏫📝

This is one of the most important concepts in all of machine learning. You should **never** evaluate your model on the same data it was trained on. Why?

Imagine you're studying for a final exam. The professor gives you a set of practice questions and the answers. You study these questions intensely. This is your **training data**. If the final exam consists of the *exact same questions*, you might get a perfect score. But does that mean you truly understand the subject? Not necessarily. You might have just memorized the answers.

The true test of your knowledge is when the professor gives you a **testing set**—new questions you've never seen before. If you do well on these, it proves you've actually learned the underlying concepts (generalized).

We do the exact same thing with our models. We split our dataset into two parts:

• Training Set (usually 70-80% of the data): The model sees this data and learns the patterns.
• Testing Set (usually 20-30% of the data): This data is held back. We use it only at the very end to evaluate how well our trained model performs on unseen data.

The `scikit-learn` library, a cornerstone of Python ML, makes this easy.

from sklearn.model_selection import train_test_split

# X contains our features (all columns except the price)
# y contains our label (the price column)

# This single line splits our data into four pieces
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# X_train, y_train -> Used to train the model
# X_test, y_test   -> Used to test the model after training

Data Cleaning: Taking Out the Trash

Real-world data is notoriously messy. It's plagued with errors, missing entries, and inconsistencies. Our first job is to clean it up.

Handling Missing Values

Your dataset will often have blank spots or "NaN" (Not a Number) values. This can happen for many reasons: a sensor failed, a user skipped a field in a form, or there was a data entry error. Most ML algorithms cannot work with missing data, so we have to deal with it.

First, let's find them. Pandas makes this simple.

import pandas as pd
import numpy as np

# Sample messy data
data = {'Age': [25, 30, np.nan, 35, 40, 30],
        'Salary': [50000, 60000, 70000, 80000, np.nan, 60000],
        'Experience': [5, 10, 7, 12, 15, 10]}
df = pd.DataFrame(data)

# Find the count of missing values in each column
print(df.isnull().sum())

We have two main strategies to handle these:

1. Dropping:** The simplest approach is to remove the rows (or columns) that contain missing values. This is okay if you have a huge dataset and only a few missing entries. But be careful! If you drop too many rows, you lose valuable information.

df_dropped = df.dropna() # Drops rows with any missing values
print(df_dropped)

2. Imputation (Filling):** A better strategy is often to fill the missing values with a calculated value. Common choices are the **mean**, **median**, or **mode** of the column.

# Fill missing Age with the median age (robust to outliers)
median_age = df['Age'].median()
df['Age'].fillna(median_age, inplace=True)

# Fill missing Salary with the mean salary
mean_salary = df['Salary'].mean()
df['Salary'].fillna(mean_salary, inplace=True)

print(df.isnull().sum()) # Check if any missing values remain

# Check for duplicate rows
print(f"Number of duplicate rows: {df.duplicated().sum()}")

# Remove duplicate rows
df_no_duplicates = df.drop_duplicates()
print(f"Shape of DataFrame after dropping duplicates: {df_no_duplicates.shape}")