← Back to Blog

How to Filter Out Records with Null or Empty Strings in Python Pandas

As a data scientist or software engineer you know how important it is to clean and preprocess data before using it for analysis or modeling One common task in data preprocessing is to filter out records with null or empty strings for a given field In this article we will explore how to accomplish this task using Python Pandas a popular library for data manipulation and analysis
By Saturn Cloud | | Miscellaneous

How to Filter Out Records with Null or Empty Strings in Python Pandas

As a data scientist or software engineer, you know how important it is to clean and preprocess data before using it for analysis or modeling. One common task in data preprocessing is to filter out records with null or empty strings for a given field. In this article, we will explore how to accomplish this task using Python Pandas, a popular library for data manipulation and analysis.

What is Pandas?

Pandas is an open-source Python library that provides data structures and tools for efficient data manipulation and analysis. It is built on top of NumPy, another popular library for numerical computing in Python. Pandas provides two main data structures: Series and DataFrame. A Series is a one-dimensional array-like object that can hold any data type, while a DataFrame is a two-dimensional table-like data structure with rows and columns.

Filtering out Null or Empty Strings in Pandas

To filter out records with null or empty strings in Pandas, we will use the dropna() method. This method is used to remove missing values (i.e., null, NaN, or None) from a DataFrame. We can specify the axis along which to drop the missing values (i.e., rows or columns) and the subset of columns to consider.

To drop records with null or empty strings for a given field, we need to specify the subset of columns that contain the field and then use the dropna() method with the subset parameter. Here is an example:

import pandas as pd

# create a sample DataFrame
data = {'name': ['Alice', 'Bob', 'Charlie', 'David'],
        'age': [25, 30, None, 35],
        'city': ['New York', 'Paris', '', 'London']}
df = pd.DataFrame(data)

# filter out records with empty strings in the 'city' column
df_filtered = df[df['city'] != '']

print(df_filtered)

Output:

    name   age      city
0  Alice  25.0  New York
1    Bob  30.0     Paris
3  David  35.0    London

In this example, we create a sample DataFrame with four records and three fields: name, age, and city. The third record has an empty string in the city field, which we want to filter out. We then use the dropna() method with the subset parameter and the != operator to drop the records with empty strings in the city field and store the result in a new DataFrame called df_filtered.

Note that the dropna() method returns a new DataFrame by default, so we need to assign the result to a new variable if we want to keep the filtered DataFrame.

Filtering out Null or Empty Strings for Multiple Fields

If we want to filter out records with null or empty strings for multiple fields, we can simply pass a list of field names to the subset parameter. Here is an example:

import pandas as pd

# create a sample DataFrame
data = {'name': ['Alice', 'Bob', 'Charlie', 'David'],
        'age': [25, 30, None, 35],
        'city': ['New York', 'Paris', '', 'London'],
        'country': ['USA', '', 'France', 'UK']}
df = pd.DataFrame(data)

# filter out records with empty strings in the 'city' and 'country' columns
df_filtered = df[(df['city'] != '') & (df['country'] != '')]

print(df_filtered)

Output:

    name   age      city country
0  Alice  25.0  New York     USA
3  David  35.0    London      UK

In this example, we create a sample DataFrame with four records and four fields: name, age, city, and country. The third and second records have empty strings in the city and country fields, respectively, which we want to filter out. We then use the dropna() method with the != operator to check for empty strings in both fields and filter the records accordingly and store the result in a new DataFrame called df_filtered.

Conclusion

Filtering out records with null or empty strings in a given field is a common data preprocessing task in data analysis and modeling. In this article, we have explored how to accomplish this task using Python Pandas, a powerful library for data manipulation and analysis. By using the dropna() method with the subset parameter, we can easily filter out records with null or empty strings for one or more fields. We hope this article has been helpful and informative for your data science or software engineering projects.

About Saturn Cloud

Saturn Cloud is your all-in-one solution for data science & ML development, deployment, and data pipelines in the cloud. Spin up a notebook with 4TB of RAM, add a GPU, connect to a distributed cluster of workers, and more. Join today and get 150 hours of free compute per month.

Try Saturn Cloud Now