The label of the new column. If colName already exists, then supplied col will update the existing column. If colName does not exist, then col will be a new column.

2. col | Column

The new column.

Return Value

A PySpark DataFrame (pyspark.sql.dataframe.DataFrame).

Examples

Consider the following PySpark DataFrame:


        
        
            
                
                
                    df = spark.createDataFrame([["Alex", 25], ["Bob", 30], ["Cathy", 50]], ["name", "age"])
df.show()
                
            
            +-----+---+
| name|age|
+-----+---+
| Alex| 25|
|  Bob| 30|
|Cathy| 50|
+-----+---+

Updating column values based on original column values in PySpark

To update an existing column, supply its column label as the first argument:


        
        
            
                
                
                    df.withColumn("age", 2 * df.age).show()
                
            
            +-----+---+
| name|age|
+-----+---+
| Alex| 50|
|  Bob| 60|
|Cathy|100|
+-----+---+

Note that you must pass in a Column object as the second argument, and so you cannot simply use a list as the new column values.

Adding a new column to a PySpark DataFrame

To add a new column AGEE with 0s:


        
        
            
                
                
                    import pyspark.sql.functions as F
df.withColumn("AGEE", F.lit(0)).show()
                
            
            +-----+---+----+
| name|age|AGEE|
+-----+---+----+
| Alex| 25|   0|
|  Bob| 30|   0|
|Cathy| 50|   0|
+-----+---+----+

Here, F.lit(0) returns a Column object holding 0s. Note that since column labels are case insensitive, if you pass in "AGE" as the first argument, you would end up overwriting the age column.

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.withColumn.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!