
        
        
            
                
                
                    df = spark.createDataFrame([["Alex", 20], ["Bob", 30], ["Cathy", 40]], ["name", "age"])
df.show()
                
            
            +-----+---+
| name|age|
+-----+---+
| Alex| 20|
|  Bob| 30|
|Cathy| 40|
+-----+---+

Computing the length of column strings in PySpark

To compute the length of each value of the name column, use the length(~) method:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.select(F.length("name")).show()
                
            
            +------------+
|length(name)|
+------------+
|           4|
|           3|
|           5|
+------------+

We could also pass in a Column object instead of a column label like so:


        
        
            
                
                
                    # df.select(F.length(df.name)).show()
df.select(F.length(F.col("name"))).show()
                
            
            +------------+
|length(name)|
+------------+
|           4|
|           3|
|           5|
+------------+

Note that we can append a new column containing the length of the strings using withColumn(~):


        
        
            
                
                
                    df_new = df.withColumn("name_length", F.length("name"))
df_new.show()
                
            
            +-----+---+-----------+
| name|age|name_length|
+-----+---+-----------+
| Alex| 20|          4|
|  Bob| 30|          3|
|Cathy| 40|          5|
+-----+---+-----------+

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/3.2.0/api/python/reference/api/pyspark.sql.functions.length.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!