Detecting and Handling Outliers in Python: Methods and Strategies

Updated: at 05:23 PM

Outliers are data points that differ significantly from other observations in a dataset. They can arise due to variability in the data, errors in measurements, or anomalous occurrences outside the normal behavior. Identifying and properly handling outliers is an important part of data cleaning and preprocessing. This guide provides a comprehensive overview of techniques for detecting and addressing outliers in Python.

Table of Contents

Open Table of Contents

Definition and Causes of Outliers

An outlier is a data point that is distant from other observations in a dataset. More formally, outliers are observations that deviate markedly from other members of the sample in which they occur.

There are several potential causes of outliers:

Dangers of Outliers

Outliers can negatively impact the performance and reliability of machine learning models if not handled properly. Some key dangers of outliers include:

Detecting Outliers in Univariate Data

For univariate data where we only have a single variable, there are several simple methods to identify potential outliers:

Visualization for Univariate Outliers

Visual inspection of the data using histograms, boxplots, and scatter plots can reveal outliers.

import matplotlib.pyplot as plt
import numpy as np

# Generate sample data
x = np.random.normal(size=1000)
x[20] = 30 # Inject outlier

# Histogram
plt.figure(figsize=(8, 6))
plt.hist(x, color='blue', edgecolor='black', alpha=0.7)
plt.title('Histogram of Data with Outlier')
plt.xlabel('Value')
plt.ylabel('Frequency')
plt.axvspan(30 - 1.5, 30 + 1.5, color='red', alpha=0.2)
plt.axvline(30, color='red', linestyle='--', label='Outlier')
plt.legend()
plt.show()

# Boxplot
fig, ax = plt.subplots(figsize=(8, 6))
ax.boxplot(x, vert=False, patch_artist=True)
ax.set_ylim(-12, 32)
ax.set_title('Boxplot of Data with Outlier')
ax.set_xlabel('Value')
plt.yticks([])
plt.show()

# Scatter plot
plt.figure(figsize=(8, 6))
plt.scatter(range(len(x)), x, color='blue', label='Data')
plt.scatter(20, 30, color='red', label='Outlier')
plt.title('Scatter Plot of Data with Outlier')
plt.xlabel('Index')
plt.ylabel('Value')
plt.legend()
plt.show()

Histogram of Data with Outlier Fig 1. Histogram visualization of data with an outlier.

Boxplot of Data with Outlier Fig 2. Boxplot visualization of data with an outlier.

Scatter Plot of Data with Outlier Fig 3. Scatter plot visualization of data with an outlier.

The visual plots clearly reveal the presence of an outlier value.

Standard Deviation Method

Any points that are a certain number of standard deviations away from the mean may be flagged as outliers. Commonly used thresholds are 2, 2.5 or 3 standard deviations:

import numpy as np

# Sample dataset (replace with your data)
x = np.array([1, 2, 3, 4, 5, 6, 1000])

# Set threshold
threshold = 3

# Calculate mean and standard deviation
mean = np.mean(x)
std = np.std(x)

# Determine outliers
outlier_idx = np.abs(x - mean) > threshold * std
print(x[outlier_idx])

This will print any values exceeding the chosen threshold, which is typically set based on domain knowledge and exploratory analysis. A lower threshold will flag more potential outliers.

Interquartile Range Method

Another approach is to use the interquartile range (IQR) to identify outliers. The IQR represents the middle 50% of the data and is less influenced by outliers. Any points below Q1 - 1.5IQR or above Q3 + 1.5IQR are marked as outliers, where Q1 and Q3 are the first and third quartiles respectively. The 1.5 multiplier helps set a stringent boundary for outliers while accounting for regular variability around the upper and lower quartiles.

import numpy as np

# Define your dataset 'x' (replace this with your data)
x = [10, 15, 20, 22, 25, 30, 35, 100, 105, 110]

# Calculate Q1, Q3, and IQR
Q1 = np.percentile(x, 25)
Q3 = np.percentile(x, 75)
IQR = Q3 - Q1

# Determine outlier bounds
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

# Find outlier indices
outlier_indices = np.where((x < lower_bound) | (x > upper_bound))
outliers = np.array(x)[outlier_indices]  # Convert x to a NumPy array and then use the indices
print("Outliers:", outliers)

This relies on the IQR rather than standard deviation to set the outlier bounds.

Detecting Outliers in Multivariate Data

Multivariate data analysis involves considering the relationships between different features, making the identification of outliers a more intricate task. In this guide, we will explore various methods for detecting multivariate outliers, and we will use Python in a Jupyter notebook to demonstrate these techniques.

Visual Methods

Visual methods provide a powerful way to spot multivariate outliers through plots. Two commonly used visual methods are Scatterplot Matrices and Parallel Coordinate Plots. We will generate sample data and use these plots to identify outliers.

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from pandas.plotting import parallel_coordinates

# Generate sample data
np.random.seed(0)
num_samples = 100
num_features = 4

data = {
    'Feature1': np.random.normal(0, 1, num_samples),
    'Feature2': np.random.normal(0, 1, num_samples),
    'Feature3': np.random normal(0, 1, num_samples),
    'Feature4': np.random.normal(0, 1, num_samples),
}
df = pd.DataFrame(data)

# Scatterplot matrix
pd.plotting.scatter_matrix(df)
plt.show()

# Parallel coordinates plot
df['NameOfClassifyingColumn'] = np.random.choice(['A', 'B', 'C'], num_samples)
parallel_coordinates(df, 'NameOfClassifyingColumn')
plt.show()

Points outside the general scope of the data will stand out as outliers in these plots.

Statistical Models

Statistical models, like regression or multivariate density estimation, can be used to identify outliers by calculating the residuals and setting a threshold. Let’s apply Linear Regression for outlier detection.

from sklearn.linear_model import LinearRegression

# Generate sample data
X = df[['Feature1', 'Feature2', 'Feature3', 'Feature4']]
y = np.random.normal(0, 1, num_samples)

reg = LinearRegression().fit(X, y)
y_pred = reg.predict(X)

residuals = y - y_pred
threshold = 3 * residuals.std()
outlier_idx = np.abs(residuals) > threshold

Proximity-Based Methods

Proximity-based methods involve using metrics like k-nearest neighbors and isolation forests to identify anomalies. These methods are based on the idea that outliers are isolated or distant from their neighbors.

Let’s use Local Outlier Factor and Isolation Forest for outlier detection.

from sklearn.neighbors import LocalOutlierFactor
from sklearn.ensemble import IsolationForest

# Local Outlier Factor
lof = LocalOutlierFactor(n_neighbors=10)
outlier_scores = lof.fit_predict(X)
outlier_idx = outlier_scores == -1

# Isolation Forest
iso = IsolationForest(n_estimators=100)
outlier_scores = iso.fit_predict(X)
outlier_idx = outlier_scores == -1

In the code snippets above, we’ve demonstrated the steps and techniques for detecting multivariate outliers using Python in a Jupyter notebook. You can apply these methods to your own datasets, whether they come from .csv files or are generated within the code.

Complete Code Example

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from pandas.plotting import parallel_coordinates
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import LocalOutlierFactor
from sklearn.ensemble import IsolationForest

# Generate sample data
np.random.seed(0)
num_samples = 100
num_features = 4

# Create a DataFrame with random data
data = {
    'Feature1': np.random.normal(0, 1, num_samples),
    'Feature2': np.random.normal(0, 1, num_samples),
    'Feature3': np.random.normal(0, 1, num_samples),
    'Feature4': np.random.normal(0, 1, num_samples),
}
df = pd.DataFrame(data)

# Scatterplot matrix
pd.plotting.scatter_matrix(df)
plt.show()

# Parallel coordinates plot
df['NameOfClassifyingColumn'] = np.random.choice(['A', 'B', 'C'], num_samples)
parallel_coordinates(df, 'NameOfClassifyingColumn')
plt.show()

# Statistical Models - Linear Regression for Outlier Detection
X = df[['Feature1', 'Feature2', 'Feature3', 'Feature4']]
y = np.random.normal(0, 1, num_samples)

reg = LinearRegression().fit(X, y)
y_pred = reg.predict(X)

residuals = y - y_pred
threshold = 3 * residuals.std()
outlier_idx = np.abs(residuals) > threshold

# Proximity-Based Methods - Local Outlier Factor
lof = LocalOutlierFactor(n_neighbors=10)
outlier_scores = lof.fit_predict(X)
outlier_idx = outlier_scores == -1

# Proximity-Based Methods - Isolation Forest
iso = IsolationForest(n_estimators=100)
outlier_scores = iso.fit_predict(X)
outlier_idx = outlier_scores == -1

This code generates random sample data for multivariate outlier detection, creates scatterplot matrices, parallel coordinates plots, and applies linear regression, local outlier factor, and isolation forest for outlier detection.

Make sure you have the required libraries (matplotlib, pandas, numpy, and scikit-learn) installed in your Jupyter notebook environment.

Multivariate Outlier Detection Multivariate Outlier Detection Fig 4: Exploratory data analysis and outlier detection using Python’s matplotlib, pandas, and scikit-learn libraries on a synthetic dataset with scatterplot matrix, parallel coordinates plot, and the application of Linear Regression, Local Outlier Factor, and Isolation Forest algorithms for outlier detection.

Handling Outliers

Once outliers have been identified, there are several techniques to handle them:

Delete Outliers

The simplest approach is to completely remove outliers from the dataset. However, this could discard potentially useful data and should be done cautiously.

X_filtered = X[~outlier_idx]
y_filtered = y[~outlier_idx]

Impute Missing Values

It is important to first handle any missing values in the data prior to outlier detection and removal. Imputing missing values with outliers present may skew results. Simple imputation or robust methods can be used to fill missing values.

Impute Outliers

Instead of deleting, outliers can be replaced with substituted values like the mean, median, or values from a model prediction. One common approach is to use a regression model to predict the values of outliers based on the non-outliers.

Imputing Outliers with a Regression Model

One approach to handling outliers is to impute them using a regression model. This method involves training a model on the data without outliers, predicting the outlier values, and replacing them with the model’s predictions. While this technique can be effective in reducing the impact of outliers on your analysis or models, it’s important to consider its pros and cons and when it’s most suitable.

Pros:

  1. Retaining Data Points: Imputing outliers allows you to retain all data points in your dataset. This can be crucial when you want to preserve information and maintain a complete dataset for further analysis.

  2. Reduces Outlier Influence: By imputing outliers with predicted values from a regression model, you mitigate the extreme influence of outliers on your analysis or machine learning models. This results in more robust and reliable results.

  3. Preserves Data Characteristics: The imputed values are derived from the relationships within the data, which helps preserve the overall characteristics of the dataset.

Cons:

  1. Model Sensitivity: The success of this method heavily depends on the quality of the regression model. If the model used for imputation is not well-suited to the data, it may lead to inaccurate imputations.

  2. Assumptions: This approach assumes that the relationships between variables can be well-captured by a linear regression model. If the data has complex or nonlinear relationships, this method may not perform well.

  3. Overfitting: There is a risk of overfitting, particularly when the dataset contains a small number of data points or when the model is too complex. Overfit models may generate imputed values that do not generalize well to new data.

When to Use It:

It’s worth noting that there are alternative methods for imputing outliers, such as using the median or other robust statistics. These methods may be preferred when the dataset has unique characteristics that make regression-based imputation less suitable. The choice of imputation method should be guided by a thorough understanding of the data and the problem context.

Let’s see how this can be done using Python and the scikit-learn library.

import numpy as np
from sklearn.linear_model import LinearRegression

# Generate synthetic data with outliers
X, y = make_regression(n_samples=100, n_features=1, noise=5, random_state=42)
# Introduce an outlier
X[0] = 2.0
y[0] = 300.0

# Identify outliers based on some threshold (e.g., 2 standard deviations)
threshold = 2 * np.std(y)
outliers = np.abs(y) > threshold

# Create a copy of the data
X_clean = X.copy()
y_clean = y.copy()

# Impute outliers by training a simple linear regression model
regressor = LinearRegression()
regressor.fit(X[~outliers], y[~outliers])
y_clean[outliers] = regressor.predict(X[outliers].reshape(-1, 1))

# Now, y_clean contains the imputed values for outliers

In this code, we first generate synthetic data with an outlier. We then identify outliers based on a threshold (in this case, two times the standard deviation of the target variable). After identifying outliers, we create a copy of the data and use a simple linear regression model (you can replace it with more complex models if needed) to impute the outlier values. The resulting y_clean variable contains the imputed values for outliers, and you can use it for further analysis or modeling.

Imputing outliers with a regression model can be a useful strategy when you want to retain the data points while reducing the influence of extreme values on your analysis or models.

Capping

Capping replaces outliers with the maximum non-outlier values in the tails of the distribution. For example, outliers above the upper quartile can be capped to the maximum value below the upper whisker. This retains presence of outliers without distorting the distribution.

# Set upper cap
x[x > Q3 + 1.5*IQR] = max(x[x <= Q3 + 1.5*IQR])

# Set lower cap
x[x < Q1 - 1.5*IQR] = min(x[x >= Q1 - 1.5*IQR])

Capping preserves outliers while limiting their impact.

Robust Methods

Robust statistical methods like RANSAC or Theil-Sen estimators can fit models that are resilient to outliers in data.

from sklearn.linear_model import RANSACRegressor
from sklearn.linear_model import TheilSenRegressor

# RANSAC robust regression
ransac = RANSACRegressor()
model = ransac.fit(X, y)

# Theil-Sen robust regression
theil = TheilSenRegressor()
model = theil.fit(X, y)

Robust models limit the influence of outliers when fitting and making predictions.

Weighting

Downweight outliers by assigning lower sample weights when model training. This reduces their impact without discarding them entirely.

# Generate outlier weight vector
outlier_weights = [1 if not o else 0.2 for o in outlier_idx]

# Train model with outlier weighting
model = LinearRegression()
model.fit(X, y, sample_weight=outlier_weights)

Here we assign a weight of 0.2 to outliers while keeping other points at 1. Weighting preserves outliers but reduces their significance during modeling.

Handling Outliers in Multivariate Data

Handling outliers in multivariate data is essential to ensure the robustness and reliability of machine learning models. When dealing with multiple features, the impact of outliers can be more complex and significant. We will explore two approaches for handling outliers in multivariate datasets: Isolation Forest and Local Outlier Factor (LOF).

Isolation Forest

The Isolation Forest is a tree-based algorithm that excels in isolating anomalies in the data. It works by constructing isolation trees, which are binary trees where nodes represent feature splits, and the depth of a data point in the tree corresponds to its anomaly score. Data points that reach a smaller depth are considered more isolated and are more likely to be outliers.

Here’s how you can use the Isolation Forest to handle outliers in a multivariate dataset:

from sklearn.ensemble import IsolationForest

# Create an Isolation Forest model
iso_forest = IsolationForest(contamination=0.05, random_state=42)

# Fit the model on the data
iso_forest.fit(X)

# Predict outliers
outlier_pred = iso_forest.predict(X)

# Filter data to remove outliers
X_filtered_iso_forest = X[outlier_pred != -1]

In this example, we set the contamination parameter to 0.05, indicating that we expect approximately 5% of the data to be outliers. Adjust this parameter based on your domain knowledge and the characteristics of your dataset.

Local Outlier Factor (LOF)

The Local Outlier Factor (LOF) is a density-based method that identifies anomalies by comparing the local density of a data point with the density of its neighbors. Data points with significantly lower densities than their neighbors are considered outliers.

Here’s how you can use the LOF algorithm to handle outliers in a multivariate dataset:

from sklearn.neighbors import LocalOutlierFactor

# Create a Local Outlier Factor model
lof = LocalOutlierFactor(n_neighbors=20, contamination=0.05)

# Fit the model on the data
lof.fit(X)

# Predict outliers
outlier_pred_lof = lof.fit_predict(X)

# Filter data to remove outliers
X_filtered_lof = X[outlier_pred_lof != -1]

In this example, we set the n_neighbors parameter to 20, indicating that the algorithm considers the density of 20 nearest neighbors. Adjust this parameter based on your dataset and desired sensitivity to outliers.

Choosing the Right Method

The choice between Isolation Forest and LOF depends on the nature of your data and the specific problem you are addressing. Isolation Forest is suitable for cases where anomalies are expected to be more isolated and sparse, while LOF is appropriate for cases where anomalies have a more local impact on density.

Remember that both methods provide a way to filter out outliers and create a cleaner dataset for modeling. You can evaluate the performance of your machine learning models on both the filtered and unfiltered datasets to assess the impact of outlier removal.

Handling outliers in multivariate data is crucial for improving the reliability and accuracy of your machine learning models. Proximity-based methods like Isolation Forest and Local Outlier Factor offer effective techniques for identifying and removing outliers, ultimately leading to more robust data analysis and modeling results.

Example: Detecting and Handling Outliers in the California Housing Dataset

In this practical example, we will demonstrate how to detect and handle outliers in a real-world dataset using Python. We’ll use the California housing dataset from scikit-learn, which contains information about median house values from different districts in California.

Loading and Visualizing the Data

First, we load the dataset and visualize potential outliers using boxplots for each feature. This step helps us identify any data points that deviate significantly from the majority of the data.

# Import necessary libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression

# Load the California housing dataset
data = fetch_california_housing()
X = data.data

# Visualize potential outliers using boxplots
plt.figure(figsize=(10, 6))
plt.boxplot(X)
plt.xticks(range(1, X.shape[1] + 1), data.feature_names, rotation=45)
plt.title("Boxplot of California Housing Dataset Features")
plt.show()

Data Preprocessing and Outlier Detection

Next, we preprocess the data by standardizing the features to have a mean of 0 and a standard deviation of 1. We then fit a linear regression model to the standardized data, make predictions, and calculate residuals. Outliers are detected based on a threshold derived from the standard deviation of residuals.

# Standardize features
X_scaled = StandardScaler().fit_transform(X)

# Fit a linear regression model
model = LinearRegression()
model.fit(X_scaled, y)

# Make predictions and calculate residuals
y_pred = model.predict(X_scaled)
residuals = y - y_pred

# Set residual threshold for outlier detection
outlier_threshold = 3 * residuals.std()

# Detect outlier indices
outlier_idx = (residuals < -outlier_threshold) | (residuals > outlier_threshold)

Handling Outliers

Once outliers are identified, we create a new dataset with outliers removed. This step allows us to build a more robust model by eliminating the influence of extreme data points.

# Create a new dataset with outliers removed
X_filtered = X_scaled[~outlier_idx]
y_filtered = y[~outlier_idx]

Retraining the Model

Finally, we retrain the linear regression model on the filtered data, which results in an improved model that is less affected by outliers.

# Retrain the model on the filtered data
model.fit(X_filtered, y_filtered)

Complete Code Example

# Import necessary libraries
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.datasets import fetch_california_housing
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression

# Load the California housing dataset
data = fetch_california_housing()
X = data.data
y = data.target

# Visualize potential outliers using boxplots
plt.figure(figsize=(10, 6))
plt.boxplot(X)
plt.xticks(range(1, X.shape[1] + 1), data.feature_names, rotation=45)
plt.title("Boxplot of California Housing Dataset Features")
plt.show()

# Standardize features
X_scaled = StandardScaler().fit_transform(X)

# Fit a linear regression model
model = LinearRegression()
model.fit(X_scaled, y)

# Make predictions and calculate residuals
y_pred = model.predict(X_scaled)
residuals = y - y_pred

# Set residual threshold for outlier detection
outlier_threshold = 3 * residuals.std()

# Detect outlier indices
outlier_idx = (residuals < -outlier_threshold) | (residuals > outlier_threshold)

# Create a new dataset with outliers removed
X_filtered = X_scaled[~outlier_idx]
y_filtered = y[~outlier_idx]

# Retrain the model on the filtered data
model.fit(X_filtered, y_filtered)

# Print the number of outliers and the improved model's R-squared score
print("Number of outliers:", np.sum(outlier_idx))
print("Improved Model R-squared:", model.score(X_filtered, y_filtered))

Boxplot of California Housing Dataset Features Fig 5. Boxplot visualization displaying the distribution of features in the California Housing Dataset, aiding in the identification of potential outliers and data distribution characteristics.

By following this step-by-step process, we can effectively detect and handle outliers in the California housing dataset, leading to a more reliable and accurate machine learning model.

This example showcases how to apply outlier detection and handling techniques to a real-world dataset, but these principles can be adapted to various data analysis and modeling tasks.

Conclusion

Outliers can significantly impact the performance of machine learning models. This guide covered key methods like visualization, standard deviation, interquartile ranges, proximity-based models, and robust statistical techniques to effectively detect outliers in both univariate and multivariate data in Python. We also discussed strategies like deletion, imputation, capping, robust methods, and weighting to appropriately handle outliers. With a sound understanding of outlier detection and treatment, practitioners can build more reliable, resilient, and accurate data pipelines and models.