ML-fundamentals
An overview of machine learning
Terminology
- general intelligence: stuff like humans, biological intelligence at the level of humans
- artificial intelligence: when a machine can perform a task as good or better than a human can, like classifying images.
- data: tabular data or data that can be stored in a single file
- big data: data that is too large to be stored in a tabular format, so it's stored in a stream instead.
- AGI: when artificial intelligence achieves the capability of general intelligence, when AI becomes better at tasks than most humans are.
- Strong AI refers to machines that display full human-like intelligence, including emotions, creativity, and a sense of purpose—like the intelligent characters you see in science fiction. It's the idea of a machine that truly thinks and understands like a person, but this is still mostly theoretical today.
- Weak AI, on the other hand, is designed for specific tasks. It can process language or recognize patterns but doesn't actually understand or have consciousness. For example, Siri is a weak AI—it listens to your questions and matches them to programmed responses without real understanding.
- predictive AI: regression, classification, or clustering tasks, which is a type of AI that analyzes large amounts of data to find patterns and make predictions about future behavior
- generative AI: generates content
Big data
Bid data satisfies the three Vs:
- Volume: high volume of data, at least several terabytes
- Velocity: high influx rate of data, like 100 megabits per second
- Variety: high variety arising from a complex data structure storage like audio, stream, or image data.
When dealing with Big Data, you arive at the CAP theorem from system design, where there is a tradeoff between consistency, availability, and partitioning with trying to replicate data across multiple machines.
Weak AI vs Strong AI
Machine learning is weak AI, where we use mathematical techniques like statistics or hardcoded rules to learn patterns in data.
- early weak AI: hardcoded rules designated by experts, much like a hardcoded decision tree
- modern weak AI: learning patterns via computational statistic inference and gradient descent, where a function hypothesis is approximated computationally instead of cognitively.
Alignment
The mismatch between what the AI does and what humans actually want is called the alignment problem, which is what happens when an AI might do exactly what it's programmed to do but in a way that humans don't expect or want.
The alignment problem arises when you program an AI to do a task and you assume that it will think and reason like a human on that same task without implementing guardrails or thinking it through better.
It's a key ethical issue because AI systems can make many decisions quickly, and if they're not aligned with human values, they might cause harm without anyone noticing right away.
There are 4 alignment problems in AI:
- value alignment: Value alignment in AI refers to the challenge of ensuring that an AI system's actions match human values and intentions
- traceability: Decision traceability in AI refers to the challenge of understanding how an AI system arrives at its decisions.
- Many AI systems, especially those analyzing massive datasets, act like a "black box"—they produce results based on complex patterns that are nearly impossible for humans to fully trace or explain.
- copyright: Copyright law protects original human-created content, but there is ongoing debate about whether using such content to train AI models qualifies as fair use.
- privacy: Privacy in the U.S. is often governed by flexible guidelines rather than strict rules, allowing companies to collect and use personal data extensively.
Value alignment
For example, if an AI system is designed to reduce health insurance costs by targeting high-cost customers, it might cancel policies for older people to save money, which is not the intended ethical outcome.
Types of machine learning
Supervised learning
Supervised learning is based on known feature data and known labels , where is the unknown, true function that provides a 100% deterministic, correlated relationship between and .
The goal of supervised learning is to estimate this true relationship using an approximation function to make predictions , and then based on the error between prediction and expected known label , we tune the hypothesis function to grow closer to the true by trying to minimize error as much as possible
The closer is to , the better the model performs, which means that hypothesis that minimizes the error (the discrepancy between the hypothesis function and the true unknown function) the most, is the best hypothesis possible, accounting for initial assumptions and bias.
Basically, the goal is to minimize something like this:
Where is some loss function calculating the prediction error for the th training sample , and is some cost function that aggregates the losses.
- loss: the prediction error for one training sample and its expected known value
- cost: the total prediction error over all training samples, found via aggregating losses.
Semantic gap in machine learning
The semantic gap refers to the difference between the data we have and the actual information or meaning we want to extract from it.
In machine learning, this often means that the features or data available (like photographs) do not fully capture the true characteristics that distinguish what we want to classify (such as the difference between cats and dogs).
NOTE
Basically a semantic gap occurs when the features you have don't explain the data 100%, and there are some other hidden variables you can't account for, and we just approximate through machine learning.
For example, a photo contains visual information, but it doesn’t include deeper properties like DNA, bone structure, or behavior, which are essential to truly distinguish a cat from a dog. The machine learning model tries to bridge this gap by finding patterns in the available data, but it is limited to what is present in the data. Sometimes, the model may pick up on irrelevant features (like snow in the background of dog photos) rather than the actual distinguishing features.
The semantic gap highlights the challenge that the data used for training may not be sufficient or relevant enough to solve the real-world problem accurately.
Features

Univariate data and bivariate data
In statistics, the terms univariate and bivariate refer to the number of variables being analyzed.
-
Univariate data involves the analysis of a single variable. The goal is to describe the distribution of this variable, often using measures of central tendency (like mean or median) and measures of spread (like variance or standard deviation).
-
Bivariate data involves the analysis of two variables simultaneously. The goal is to understand the relationship or correlation between these two variables.
Univariate data and feature types
There are 4 types of univariate data, meaning when you're looking at just one feature:
- nominal data : categorical data with no order to the data
- ordinal data : categorical data with order to the data and labels
- continuous data : numerical data that includes decimals
- discrete data : numerical data that includes only integers or a finite collection of integers.
There are two types of feature types:
- qualitative: either nominal or ordinal data, where the feature is based on categories and uses one-hot encoding to encode the category numerically.
- quantitative: either continuous or discrete data, using numerical data.
Measures of central tendency
Based on the skewness of the distribution, these facts are guaranteed:
- Symmetric : mean = median
- Right skewed : mean > median
- Left Skewed : mean < median
Best measures of central tendency for each situation
- Median : when dealing with ordinal data or with skewed data
- Mean : When dealing with evenly distributed data like in a normal distribution
Measures of spread
- variance : we can use the
np.var(arr)method to get the variance of the data within a vector. - standard deviation : we can use the
np.std(arr)method to get the standard deviation of the data within a vector. - coefficient of variation : the coefficient of variation is the standard deviation divided by the mean.
x = np.random.randn(20)
print("standard deviation:", np.std(x))
print("variance:", np.var(x))
print("coefficient of variation", np.std(x) / np.mean(x))
Feature scaling
Most machine learning algorithms use distance metrics (like Euclidean distance) or gradient descent for optimization.
If one feature has a range of 0-1 and another has a range of 0-1,000,000, the algorithm will be dominated by the larger magnitude feature, even if the smaller feature is more predictive. That is why we must scale feature ranges into a more suitable, standardized and smaller range.
So why do we use feature scaling? Three key principles:
-
Comparability: It brings all features to a similar scale, making them directly comparable.
-
Convergence: Algorithms that use Gradient Descent (like Logistic Regression or Neural Networks) converge much faster when features are scaled.
-
Distance Sensitivity: Algorithms like KNN, K-Means, and PCA are highly sensitive to the magnitude of features.
There are three types of scaling:
- Standardization (Z-score normalization): Transforms data to have a mean of 0 and a standard deviation of 1. It is robust to outliers compared to Min-Max scaling.
- Normalization (Min-Max Scaling): Rescales the data to a fixed range, usually 0 to 1. It is very sensitive to outliers (a single outlier can compress all other values into a tiny range).
- log 10 scaling: Simple rescaling of data on large magnitudes, but not actually used to improve the performance of a machine learning model in the sense of how feature scaling is supposed to be. It's only used for human-readable values so we can see smaller values instead of all values being in the thousands.
from sklearn.preprocessing import StandardScaler, MinMaxScaler
import numpy as np
# Sample data: [Feature A, Feature B]
data = np.array([[10, 0.001], [20, 0.002], [30, 0.005], [1000, 0.01]])
# Standard Scaler
std_scaler = StandardScaler()
std_data = std_scaler.fit_transform(data)
# Min-Max Scaler
mm_scaler = MinMaxScaler()
mm_data = mm_scaler.fit_transform(data)
print("Original Data:\n", data)
print("\nStandard Scaled (Mean=0, Std=1):\n", std_data)
print("\nMin-Max Scaled (Range 0-1):\n", mm_data)
Log transformation
We use log transformation to get better resolution on a plot where our data varies widely between magnitudes, like on a range from 1,000 - 350,000.
np.log10(arr): applies base 10 log to all elements in the array. Returns new arraynp.log(arr): applies natural log to all elements in the array. Returns new array
We want to apply the log 10 transformation on our features that vary widely in magnitude.
Min-max scaling
Rescales the data to a fixed range, usually between 0 to 1.
- pro (fixed range): a fixed range offers predictability and standardization in mathematical outputs
- con (sensitive to outliers): very sensitive to outliers since a single outlier can compress all other values into a tiny range.
- For example if the max is 1,000,000 and the average is maybe 29, then that single million value outlier basically ruins the rest of the range because it makes every single other value extremely small while the largest value is equal to 1.
Standard scaling
Standard scaling fits features to a normal distribution, making each feature have a mean and a standard deviation
The formula for standard scaling is this, where you subtract the mean from each feature value, and then divide that by the standard deviation.
This results in each scaled feature having a mean = 0 and standard deviation = 1.
NOTE
This is the exact same thing as the Z-score. It returns the z-score of each feature value, about how many standard deviations the observation is from the mean.
WARNING
If you try to calculate the coefficient of variation on standard scaled data, then you will get an error because standard scaled data always has a mean = 0 and variance = 1, thus 1 / 0 nets you undefined.
We can get access to a standard scaler through the sklearn library, like so:
from sklearn.preprocessing import StandardScaler as SS
standard_scaler = SS()
ss.fit_transform(df): takes in a dataframe or another 2D array, and then feature scales all the feature columns. It returns the scaled dataframe or array.ss.fit(df): calculates mean and standard deviation of data and stores it in thessobject. ReturnsNoness.transfom(df): feature scales the data after you callss.fit(). Only does this for standard scaler, but applies fitted parameters to data
These three methods do different things depending on which object instance of sklearn you use. Here is how they work in general:
.fit(): fits the data to the model.transform(): transforms the model.fit_transform(): fits the data to the model, and then returns the transformed data.
standard_scaler = StandardScaler()
scaled_df = standard_scaler.fit_transform(data)
Feature importance
Feature importance matters because you want to find the most important features that contribute the most to a target value, as focusing on those features will make your predictor stronger.
Here are the core benefits
- improves performance, reduces model complexity: It helps you focus on the most relevant data, improving prediction accuracy and reducing model complexity.
- improves explainability: lowering the number of features makes explainign those features and thus improves model interpretability.
Feature engineering
Feature engineering is the process of creating new features from existing raw data and other existing features to use new features that would benefit the model training in order to improve a model's performance
NOTE
Good feature engineering makes the difference between an average model and an excellent one, as it helps the model to focus on the most relevant patterns in the data.

Here are examples of common feature engineering tasks to create, modify, or delete features for facilitating better input data to the model:
- combining existing features together: you can achieve a lower complexity model (and thus less variance) by making the data lower complexity by combining several features into one single, powerful predictive feature.
- removing highly correlated features: features that are highly correlated with each other will be redundant. Therefore removing them lets you make your model simpler.
- impute missing data: features with missing data are bad data, so it's better to fill them via imputation or remove the feature entirely.
Imputation
Imputation is the process of filling in missing data values in your dataset so that your machine learning model can train effectively.
Instead of leaving gaps or incomplete information, imputation estimates those missing values based on other data, ensuring the dataset is complete and consistent.
NOTE
Imputation is important because most models require complete data to learn patterns accurately, and imputation allows you to infer reasonable estimates for missing data instead of throwing it away.
Data preparation
Data analysis
Here are basic data analysis tools you can do:
- verify which features have missing values: it's important to understand which features may have null values
# Verify which features have missing values
housing_df.isnull().sum()
# Calculate the % of missing data
housing_df['total_bedrooms'].isnull().sum()/housing_df.shape[0] * 100
- find most important features: use pandas to find feature importances
# Determine feature importance - random forest algorithm is that it gives you the ‘feature importance’ for all the variables in the data
# plot the 6 most important features
plt.figure(figsize=(10,6))
feat_importances = pd.Series(rf_model.feature_importances_, index = X_train.columns)
feat_importances.nlargest(6).plot(kind='barh');
Data preparation
One-hot encoding
One-hot encoding is the process of converting text enum data into numerical data, where if an enum is of cardinality , then the one-hot encoded vector of a datum of that enum will be of dimensionality , where one value 1 and all other values are 0.
That one position where the value is 1 corresponds to that datum's position in the enum, meaning that one-hot encoded vector representation uniquely represents that enum.
pd.get_dummies(housing_df['ocean_proximity'])
Training, Validation, Test
Generalization error
Generalization error is how much your model errors on new data it hasn't seen before.
NOTE
Therefore, to get an accurate view of how your model is doing, you need to make sure that you cannot reuse a training set as your test set when doing the generalization error. You must calculate the generalization error on new data that you haven't seen before, otherwise you introduce biases in the data.
Let's take a look at training performance vs generalization performance:
- in-sample performance: when training on a train set, which we use to iteratively find a good hypothesis, the error on the training set will be optimistic and having fit the model to the training set, the model would be biased to performing well on the training set.
- out-of-sample performance: When training on new data, the model has never seen before, it can't use its underlying biases to get a lower error, and therefore is a more accurate depiction of a model's performance
- overfitting: When we overfit a data set, essentially the model does well on the training data but fails to generalize on the testing data. It means training score is less than testing score.
- underfitting: When we underfit a data set, it performs poorly on both the training set and and test set.
- internal validity : how well the model performs on the training data
- external validity : how well the model performs on the test data
Train test-split
The performance on the validation and test sets will be roughly the same, but the test set will be slightly worse since you will not have trained on the test set at all.
- train set: The dataset partition which you use to train your model - around 60% of dataset
- validation set: The dataset partition which you use to choose the best hyperparameters for your model - around 20% of dataset
- test set: The dataset partition which you use to test your model - around 20% of dataset
validation is the process of evaluating a model on data that it hasn't been trained on yet.

The validation set is a portion of the dataset with the purpose of evaluating how well the model would perform on new unseen data.
NOTE
The reason why the test set can only be tested once is because if you test on it multiple times, you're essentially just using the test set as a validation set and you're fitting to it.
sklearn implementation
Here is an example of how we can do in sklearn:
The tts() method takes in an array of features and an array of target data, and then returns an array of 4 nested arrays, representing the training data and testing data respectively.
from sklearn.model_selection import train_test_split as tts
X_train, X_test, y_train, y_test = tts(x, y, test_size=0.2, shuffle=True, random_state=201)
The first two arguments you pass to the tts() method are the array of features, x, and the array of target values y. Then after that here are the kwargs you can supply:
test_size=: the percentage of data to allocate to testing. 20% is pretty good here.shuffle=: whether to randomly sort data or not.random_state=:int. a random seed to set so that you get back the same split every time.
k-fold cross validation
The idea of K-fold validation is to make each batch of data as the testing data so that there are no discrepancies and bias between what we're choosing for training data and testing data.
You will make models during this process.
- Split data into equally sized groups, called folds
- Use the first fold as a validation fold, which is the testing data, and then merge the rest of the folds into a group as the training data, called train folds.
- Move on to make the second fold as the validation fold, and then merge the rest of the folds into a group as the training data.
- Continue this process for iterations, covering all folds, and then compute the error for each iteration
- The total error is the average of all error yields.
In total, each fold will be in the training data times and in the testing data 1 time. Per iteration, you have train folds and 1 validation fold.

NOTE
K-fold cross validation provides a more robust estimate of the model's true performance and helps detect underfitting or overfitting.
Theory of k
The value of has a bias-variance tradeoff.
- As the number of folds increases, You do better on training data and do worse on testing data (testing data is a smaller portion of data) and thus variance increases
- As the number of folds decreases, You do worse on training data and do better on testing data (testing data is a bigger portion of data, and you're training on less of data) and thus bias increases
kfold in code
-
Import
kfoldfrom sklearn.model_selection import KFold -
Create a kfold instance from the
KFoldclasskf = KFold(n_splits = 10, random_state=146, shuffle=True) -
Use the
kf.split(data)method to create kfold splits on your data, which should be a 2D array. This method returns a generator, so you should iterate through itfor idxTrain, idxTest in kf.split(x):Xtrain = x[idxTrain]Xtest = x[idxTest]ytrain = y[idxTrain]ytest = y[idxTest]
And here's a convenience method:
from sklearn.model_selection import KFold
def do_Kfold(model,X,y,k,scaler = None, random_state = 146):
kf = KFold(n_splits=k, random_state = random_state, shuffle=True)
train_scores = []
test_scores = []
for idxTrain, idxTest in kf.split(X):
Xtrain = X[idxTrain, :]
Xtest = X[idxTest, :]
ytrain = y[idxTrain]
ytest = y[idxTest]
if scaler != None:
Xtrain = scaler.fit_transform(Xtrain)
Xtest = scaler.transform(Xtest)
model.fit(Xtrain,ytrain)
train_scores.append(model.score(Xtrain,ytrain))
test_scores.append(model.score(Xtest,ytest))
return train_scores, test_scores
Leave-one-out-validation
Leave one out validation is where you train on every single observation except for one, which is your test set.
NOTE
This is extremely useful when paired with K nearest neighbors.
Grid Search
Here are the essential components to grid search:
- estimator : the model to run grid search on
- param grid : the dictionary of hyperparameters to try and optimize
- cv : the KFold validation and number of folds to use
- scoring: how the Grid model shoudl evaluate performance to select the best estimator.
Grid search kwargs
cv=: the custom cross validation to use, like a custom KFold instance. By default it's 5 folds KFold validation.scoring=: string, how the Grid model should evaluate performance to select the best estimator."accuracy": uses accuracy as the score. Useful for classification"neg_mean_squared_error": uses negative mean squared error as score. Useful for regression"recall": uses recall as the score