
        
        
            
                
                
                    from pyspark.sql import Row
data = [
    Row(name="Alex", age=20, friend=Row(name="Bob",age=30)),
    Row(name="Cathy", age=40, friend=Row(name="Doge",age=40))
]
df = spark.createDataFrame(data)
df.show()
                
            
            +-----+---+----------+
| name|age|    friend|
+-----+---+----------+
| Alex| 20| {Bob, 30}|
|Cathy| 40|{Doge, 40}|
+-----+---+----------+

Here, the friend column contains nested Row, which can be confirmed by printing out the schema:


        
        
            
                
                
                    df.printSchema()
                
            
            root
 |-- name: string (nullable = true)
 |-- age: long (nullable = true)
 |-- friend: struct (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- age: long (nullable = true)

Updating nested rows in PySpark

To update nested rows, use the withField(~) method like so:


        
        
            
                
                
                    import pyspark.sql.functions as F
updated_col = df["friend"].withField("name", F.lit("BOB"))
df.withColumn("friend", updated_col).show()
                
            
            +-----+---+---------+
| name|age|   friend|
+-----+---+---------+
| Alex| 20|{BOB, 30}|
|Cathy| 40|{BOB, 40}|
+-----+---+---------+

Note the following:

we are updating the name field of the friend column with a constant string "BOB".
the F.lit("BOB") returns a Column object whose values are filled with the string "BOB".
the withColumn(~) method replaces the friend column of our DataFrame with the updated column returned by withField(~).

Updating nested rows using original values in PySpark

To update nested rows using original values, use the withField(~) method like so:


        
        
            
                
                
                    updated_col = df["friend"].withField("name", F.upper("friend.name"))
df.withColumn("friend", updated_col).show()
                
            
            +-----+---+----------+
| name|age|    friend|
+-----+---+----------+
| Alex| 20| {BOB, 30}|
|Cathy| 40|{DOGE, 40}|
+-----+---+----------+

Here, we are uppercasing the name field of the friend column using F.upper("friend.name"), which returns a Column object.

Adding new field values in nested rows in PySpark

The withField(~) column can also be used to add new field values in nested rows:


        
        
            
                
                
                    updated_col = df["friend"].withField("upper_name", F.upper("friend.name"))
df_new = df.withColumn("friend", updated_col)
df_new.show()
                
            
            +-----+---+----------------+
| name|age|          friend|
+-----+---+----------------+
| Alex| 20|  {Bob, 30, BOB}|
|Cathy| 40|{Doge, 40, DOGE}|
+-----+---+----------------+

Now, checking our schema of our new PySpark DataFrame:


        
        
            
                
                
                    df_new.printSchema()
                
            
            root
 |-- name: string (nullable = true)
 |-- age: long (nullable = true)
 |-- friend: struct (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- age: long (nullable = true)
 |    |-- upper_name: string (nullable = true)

We can see the new nested field upper_name has been added!

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/3.2.0/api/python/reference/api/pyspark.sql.Column.withField.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!