menu

login

Log in

Linear Algebra

Prob and Stats

Other math topics

Machine Learning

Dagster (NEW)

search

Search

Login

Unlock 100+ guides

menu

menu

search toc

close

Outline

Return Value Examples Identifying rows where certain value is not null in PySpark DataFrame Getting rows where certain value is not null in PySpark DataFrame

Comments

Log in or sign up

Cancel

Post

account_circle

exit_to_app

Sign out

What does this mean?

Why is this true?

Give me some examples!

search

keyboard_voice

close

Searching Tips

Search for a recipe:
"Creating a table in MySQL"

Search for an API documentation: "@append"

Search for code: "!dataframe"

Apply a tag filter: "#python"

Useful Shortcuts

/ to open search panel

Esc to close search panel

↑↓ to navigate between search results

⌘d to clear all current filters

⌘Enter to expand content preview

icon_star

Doc Search

icon_star

Code Search Beta

SORRY NOTHING FOUND!

mic

Start speaking...

Voice search is only supported in Safari and Chrome.

fullscreen_exit

Shrink

Navigate to

PySpark

147 guides

keyboard_arrow_down

Linear Algebra

Prob and Stats

Machine Learning

Other math topics

chevron_leftPySpark Column

Method alias Method cast Method contains Method dropFields Method endswith Method getItem Method isin Method isNotNull Method isNull Method otherwise Method rlike Method startswith Method substr Method withField

check_circle

Mark as learned

thumb_up

0

thumb_down

0

chat_bubble_outline

0

Comment

auto_stories Bi-column layout

settings

PySpark Column | isNotNull method

schedule Aug 12, 2023

Last updated

local_offer

PySpark

Tags

tocTable of Contents

expand_more

Return Value Examples Identifying rows where certain value is not null in PySpark DataFrame Getting rows where certain value is not null in PySpark DataFrame

Master the mathematics behind data science with 100+ top-tier guides
Start your free 7-days trial now!

PySpark Column's isNotNull() method identifies rows where the value is not null.

Return Value

A PySpark Column (pyspark.sql.column.Column).

Examples

Consider the following PySpark DataFrame:


        
        
            
                
                
                    df = spark.createDataFrame([["Alex", 25], ["Bob", 30], ["Cathy", None]], ["name", "age"])
df.show()
                
            
            +-----+----+
| name| age|
+-----+----+
| Alex|  25|
|  Bob|  30|
|Cathy|null|
+-----+----+

Identifying rows where certain value is not null in PySpark DataFrame

To identify rows where the value for age is not null:


        
        
            
                
                
                    df.select(df.age.isNotNull()).show()
                
            
            +-------------+
|(age IS NULL)|
+-------------+
|        false|
|        false|
|         true|
+-------------+

Getting rows where certain value is not null in PySpark DataFrame

To get rows where the age field is not null:


        
        
            
                
                
                    df.filter(df.age.isNotNull()).show()
                
            
            +----+---+
|name|age|
+----+---+
|Alex| 25|
| Bob| 30|
+----+---+

Here, the filter(~) extracts rows that correspond to True in the boolean column returned by isNotNull() method.

robocat

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.Column.isNotNull.html

thumb_up

0

thumb_down

0

chat_bubble_outline

0

settings

Enjoy our search

Hit / to insta-search docs and recipes!

Navigation

Contact us

Resources

Python Pandas MySQL Beautiful Soup Matplotlib NumPy PySpark

Community

Join our Discord

Join our newsletter for updates on new comprehensive DS/ML guides

|