Data Fundamentals

Every AI model is only as good as the data it learns from. "Garbage in, garbage out" is the most important principle in machine learning. This module covers the practical fundamentals of working with data — from understanding data types and cleaning messy datasets to engineering useful features and building reliable data pipelines.

Why Data Is the Foundation

In the AI world, there's a common saying: "The team with the best data usually wins, not the team with the best algorithm." A simple model with excellent data will almost always outperform a sophisticated model with poor data. Here's why data quality matters so much:

• Models learn patterns from data. If the data contains errors, the model learns those errors as truth
• Bias in data becomes bias in predictions. If training data underrepresents certain groups, the model will perform poorly on those groups
• Missing context leads to wrong conclusions. A model predicting house prices without location data will give meaningless results
• Data quantity matters, but quality matters more. 10,000 clean, well-labeled examples often beat 1,000,000 noisy ones

Types of Data

Before you can work with data, you need to understand the different forms it takes. The type of data determines what tools and techniques you'll use.

Most real-world AI projects combine multiple data types. A customer sentiment analysis project might use structured data (purchase history from a database), unstructured data (support ticket text), and semi-structured data (JSON responses from a review API).

Data Cleaning and Preprocessing

Raw data is almost never ready for machine learning. It contains missing values, outliers, inconsistent formats, and errors. Cleaning and preprocessing is where you fix these issues.

Handling Missing Values

Missing data is ubiquitous. A customer might skip a survey question, a sensor might malfunction, or a form field might be optional. You need a strategy for dealing with it:

import pandas as pd
import numpy as np

df = pd.read_csv("customers.csv")

# Check for missing values
print(df.isnull().sum())

# Drop rows where target variable is missing
df = df.dropna(subset=["revenue"])

# Fill numerical columns with median
df["age"] = df["age"].fillna(df["age"].median())

# Fill categorical columns with mode
df["country"] = df["country"].fillna(df["country"].mode()[0])

# Create a flag for missing values before filling
df["income_missing"] = df["income"].isnull().astype(int)
df["income"] = df["income"].fillna(df["income"].median())

Handling Outliers

Outliers are data points that are dramatically different from the rest. An age of 150 is clearly an error, but a salary of $10 million might be real (just unusual). The key question is always: is this a data error or a legitimate extreme value?

Normalization and Scaling

Many ML algorithms expect features to be on similar scales. If "age" ranges from 0-100 and "income" ranges from 0-10,000,000, the model will be dominated by income simply because the numbers are larger — not because income is more important.

from sklearn.preprocessing import StandardScaler, MinMaxScaler

# Standard scaling (most common)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)  # use same scale as training!

# Min-Max scaling
minmax = MinMaxScaler()
X_train_normalized = minmax.fit_transform(X_train)
X_test_normalized = minmax.transform(X_test)

Feature Engineering

Feature engineering is the art of creating new, informative variables from your raw data. Good features can dramatically improve model performance — often more than switching to a better algorithm. It's where domain knowledge meets data science.

Common Feature Engineering Techniques

df["day_of_week"] = df["date"].dt.dayofweek
df["month"] = df["date"].dt.month
df["is_weekend"] = df["day_of_week"].isin([5, 6]).astype(int)
df["hour"] = df["timestamp"].dt.hour
df["is_business_hours"] = df["hour"].between(9, 17).astype(int)

# Customer-level features from transaction data
customer_features = transactions.groupby("customer_id").agg({
    "amount": ["mean", "sum", "count", "std"],
    "date": ["min", "max"],
}).reset_index()

# Time since last purchase
customer_features["days_since_last"] = (
    pd.Timestamp.now() - customer_features["date"]["max"]
).dt.days

# One-hot encoding: create binary columns for each category
df_encoded = pd.get_dummies(df, columns=["color", "size"])
# color_red, color_blue, color_green, size_S, size_M, size_L

# Label encoding: assign integers (for ordinal data)
size_map = {"S": 1, "M": 2, "L": 3, "XL": 4}
df["size_encoded"] = df["size"].map(size_map)

# Price per square foot (combines two features meaningfully)
df["price_per_sqft"] = df["price"] / df["square_feet"]

# Ratio features
df["debt_to_income"] = df["total_debt"] / df["annual_income"]

# Polynomial features for non-linear relationships
df["age_squared"] = df["age"] ** 2

Train/Test Splits: Why and How

The most fundamental concept in ML evaluation is the train/test split. You divide your data into two (or three) parts, train your model on one part, and evaluate it on data it has never seen. This is how you measure real-world performance rather than just memorization.

from sklearn.model_selection import train_test_split

# Simple train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,      # 20% for testing
    random_state=42,    # reproducible split
    stratify=y          # maintain class proportions
)

# Three-way split: train / validation / test
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42, stratify=y
)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.18, random_state=42, stratify=y_temp
)
# Results: ~70% train, ~15% validation, ~15% test

Working with Common Data Formats

In practice, data comes from many sources and formats. Here's how to work with the most common ones:

CSV (Comma-Separated Values)

The most common format for tabular data. Simple, readable, and supported everywhere.

import pandas as pd

# Read CSV
df = pd.read_csv("data.csv")

# Handle common issues
df = pd.read_csv("data.csv",
    encoding="utf-8",        # handle special characters
    na_values=["N/A", ""],   # treat these as missing
    parse_dates=["date"],    # auto-parse date columns
    dtype={"zip_code": str}  # prevent treating zip codes as numbers
)

# Write CSV
df.to_csv("cleaned_data.csv", index=False)

JSON (JavaScript Object Notation)

The standard format for API responses. Nested and flexible, but requires flattening for ML use.

import json
import pandas as pd

# Read JSON
df = pd.read_json("data.json")

# For nested JSON (common with APIs)
with open("nested_data.json") as f:
    data = json.load(f)

# Flatten nested structure
df = pd.json_normalize(data["results"], sep="_")

# Fetch from API
import requests
response = requests.get("https://api.example.com/data")
data = response.json()
df = pd.DataFrame(data["items"])

SQL Databases

For production data stored in databases like PostgreSQL, MySQL, or SQLite:

import pandas as pd
import sqlite3

# Connect and query
conn = sqlite3.connect("database.db")
df = pd.read_sql_query("SELECT * FROM customers WHERE active = 1", conn)
conn.close()

# For larger databases, use SQLAlchemy
from sqlalchemy import create_engine
engine = create_engine("postgresql://user:pass@host/dbname")
df = pd.read_sql("SELECT * FROM orders WHERE date > '2025-01-01'", engine)

Data Pipelines

A data pipeline is an automated sequence of steps that takes raw data and transforms it into a format ready for machine learning. Instead of running cleaning steps manually each time, you encode them in a reproducible pipeline.

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestClassifier

# Build a pipeline: impute missing values → scale → train model
pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
    ("model", RandomForestClassifier(n_estimators=100))
])

# The pipeline handles everything in one call
pipeline.fit(X_train, y_train)
predictions = pipeline.predict(X_test)
accuracy = pipeline.score(X_test, y_test)

Common Data Pitfalls

These mistakes are responsible for most ML failures. Learn to recognize and avoid them:

Recommended Resources