Healthcare Data Analysis with Pandas in Python

Updated: at 05:15 AM

Healthcare data analysis has become increasingly important with the rise of big data and advanced analytics techniques. Python is a popular programming language for healthcare data science due to its versatility, simplicity, and vast array of data tools and libraries.

Pandas is an open-source Python library that provides high-performance, easy-to-use data structures and data analysis tools for working with structured data. It is one of the most widely used tools for healthcare data manipulation and analysis using Python.

In this comprehensive guide, we will explore techniques for healthcare data analysis using Pandas in Python, with a focus on the Philippines healthcare system. We will learn how to load, explore, clean, visualize, analyze, and gain insights from healthcare datasets using Pandas and other Python libraries.

Real-world healthcare datasets will be used to showcase key data manipulation and analysis tasks. By the end, you will have strong foundational skills to process and analyze healthcare data with Python. Let’s get started!

Table of Contents

Open Table of Contents

Loading Healthcare Datasets

The first step in any data analysis project is importing the dataset. Pandas provides various functions to load datasets from different sources into DataFrames, the primary Pandas data structure.

Some common ways to load data in Pandas:

# Load CSV file into DataFrame
df = pd.read_csv('healthcare_data.csv')

# Load Excel file
df = pd.read_excel('healthcare_data.xlsx')

# Load SQL database table into DataFrame
df = pd.read_sql('SELECT * FROM healthcare_table', conn)

For demonstration, we will use a publicly available Philippines health survey dataset from Kaggle. This dataset contains information on demographics, health conditions, and habits of 5000 respondents:

import pandas as pd

health_df = pd.read_csv('healthcare_survey_ph.csv')

Let’s explore the DataFrame:

# Check data type of df
print(type(health_df))

# Print first 5 rows
health_df.head()

# Number of rows and columns
health_df.shape

# Column names
health_df.columns

This shows key information on the structure of the loaded data, which is important before further analysis.

Exploring and Cleaning Data

Once the data is loaded, it is common to explore the DataFrame contents and clean the data to handle missing values and formatting inconsistencies before analysis.

Exploring Data

We can check DataFrame information using various Pandas functions:

# Summary statistics for each column
health_df.describe()

# Data types for each column
health_df.dtypes

# Number of null values in each column
health_df.isnull().sum()

# Distribution of categorical columns
health_df['Gender'].value_counts()

This gives a quick glimpse into the properties of the dataset. We can see potential data issues like missing values and inconsistent formats.

Cleaning Data

Now let’s clean the dirty data for downstream analysis.

Handle missing values:

# Drop rows with missing values
health_df.dropna(inplace=True)

# Fill missing values
health_df.fillna(method='ffill', inplace=True)

Fix formatting errors:

# Convert string column to numeric
health_df['Age'] = pd.to_numeric(health_df['Age'])

# Standardize date columns
health_df['VisitDate'] = pd.to_datetime(health_df['VisitDate'])

Filter outliers:

# Filter blood pressure between 60-220
health_df = health_df[(health_df['BloodPressure'] > 60) &
                      (health_df['BloodPressure'] < 220)]

Data cleaning ensures higher quality data for the analysis procedures.

Data Visualization

Visualizing healthcare data is key to understand distributions, spot trends and identify relationships. Pandas has tight integration with Matplotlib to create informative plots.

Let’s visualize the cleaned survey data to explore patterns:

# Histogram of age
health_df['Age'].plot.hist()

# Scatter plot of blood pressure vs body mass index
health_df.plot.scatter(x='BMI', y='BloodPressure')

# Box plots of cholesterol by gender
health_df.boxplot(column='Cholesterol', by='Gender')

We can immediately notice insights like increasing blood pressure with BMI and higher cholesterol in females from the visual patterns. Visual data exploration leads to asking more targeted analytical questions later.

Data Analysis with Pandas

Now that our data is prepared, we can perform analysis to derive healthcare insights. Pandas provides a versatile set of functions for various analytical tasks:

Summary Statistics

Aggregate statistics like mean, median, counts cater to many healthcare analysis needs:

# Average BMI by gender
health_df.groupby('Gender')['BMI'].mean()

# Total patients surveyed by region
health_df['Region'].value_counts()

# Percentage of patients with diabetes
(health_df['Diabetes'] == 1).sum() / len(health_df) * 100

Hypothesis Testing

Statistical tests can evaluate healthcare hypotheses:

# Compare mean BMI between males and females
from scipy import stats
stats.ttest_ind(health_df.loc[health_df['Gender'] == 'Male']['BMI'],
                health_df.loc[health_df['Gender'] == 'Female']['BMI'])

# Chi-square test for smoking vs cancer
contingency = [[50, 150], [10, 100]]
chi2, p_val, dof, expected = stats.chi2_contingency(contingency)

Correlations

Find relationships between variables:

# Correlation between age and blood pressure
health_df['Age'].corr(health_df['BloodPressure'])

# Correlation matrix for all numeric columns
health_df.corr()

Regression Analysis

Model and predict outcomes using regression:

# Linear regression
import statsmodels.formula.api as smf
model = smf.ols('BloodPressure ~ Age + BMI', data=health_df).fit()
model.params

# Logistic regression
from sklearn.linear_model import LogisticRegression
X = health_df[['Age', 'BMI']]
y = health_df['Hypertension']
logit = LogisticRegression()
logit.fit(X, y)

Pandas combined with SciPy, StatsModels, Scikit-Learn makes Python a powerful environment for in-depth healthcare data analysis.

Case Study - Patient Health Risk Prediction

Let’s walk through a real-world healthcare data analysis case study using our survey dataset.

Business Problem

A healthcare organization wants to identify patients at high risk of developing chronic illnesses like heart disease and diabetes to proactively monitor their health. We need to build a predictive model using patient demographic and clinical data.

Analyzing and Preparing Data

We divide the relevant columns into features (X) and target (y):

X = health_df[['Age', 'BMI', 'Cholesterol', 'BloodPressure']]
y = health_df['Diabetes']

We split the data into training and test sets:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)

Model Training and Evaluation

We train a logistic regression model on the data:

from sklearn.linear_model import LogisticRegression

logreg = LogisticRegression()
logreg.fit(X_train, y_train)

We evaluate model performance:

print('Accuracy:', logreg.score(X_test, y_test))
print('Precision:', precision_score(y_test, y_pred))
print('Recall:', recall_score(y_test, y_pred))

Making Predictions

We use the model to identify high-risk patients:

y_pred = logreg.predict(X_test)

high_risk = X_test[y_pred==1]
print('Number of high risk patients:', len(high_risk))

By leveraging Pandas and Scikit-Learn, we built a predictive model to improve patient risk stratification using healthcare data.

Conclusion

Pandas is undoubtedly an indispensable tool for healthcare data analysis in Python. We explored key Pandas capabilities like data cleaning, visualization, aggregation, modeling, and more to derive insights from healthcare data.

By following this comprehensive guide, you should feel equipped to start analyzing real-world healthcare datasets and extracting meaningful patterns to improve clinical and business decisions in the Philippines healthcare system. Pandas, along with the extensive Python data analysis ecosystem, empowers data-driven advancement in healthcare services.