
        
        
            
                
                
                    df = spark.createDataFrame([("Alex", 20), ("Bob", 30), ("Cathy", 40)], ["name", "age"])
df.show()
                
            
            +-----+---+
| name|age|
+-----+---+
| Alex| 20|
|  Bob| 30|
|Cathy| 40|
+-----+---+

Converting PySpark column type to string

To convert the type of the DataFrame's age column from numeric to string:


        
        
            
                
                
                    df_new = df.withColumn("age", df["age"].cast("string"))
df_new.show()
                
            
            +-----+---+
| name|age|
+-----+---+
| Alex| 20|
|  Bob| 30|
|Cathy| 40|
+-----+---+

Equivalently, we can pass in the StringType() method like so:


        
        
            
                
                
                    from pyspark.sql.types import StringType
df_new = df.withColumn("age", df["age"].cast(StringType()))
df_new.show()
                
            
            +-----+---+
| name|age|
+-----+---+
| Alex| 20|
|  Bob| 30|
|Cathy| 40|
+-----+---+

I recommend passing in "string" instead of StringType() for simplicity.

To confirm that the column type has been converted to string, use the printSchema() method:


        
        
            
                
                
                    df_new.printSchema()
                
            
            root
 |-- name: string (nullable = true)
 |-- age: string (nullable = true)

Converting PySpark column type to integer

To convert the column type to integer, use cast("int"):


        
        
            
                
                
                    df_new = df.withColumn("age", df["age"].cast("int"))
df_new.printSchema()
                
            
            root
 |-- name: string (nullable = true)
 |-- age: integer (nullable = true)

Converting PySpark column type to float

To convert the column type to float, use cast("float"):


        
        
            
                
                
                    df_new = df.withColumn("age", df["age"].cast("float"))
df_new.printSchema()
                
            
            root
 |-- name: string (nullable = true)
 |-- age: float (nullable = true)

Converting PySpark column type to date

To convert the PySpark column type to date, use the to_date(~) method instead of cast(~).

Published by Isshin Inada

Edited by 0 others

Did you find this page useful?

thumb_up

thumb_down

Comment

Citation

Ask a question or leave a feedback...

Official PySpark Documentation

https://spark.apache.org/docs/3.2.0/api/python/reference/api/pyspark.sql.Column.cast.html

thumb_up

thumb_down

chat_bubble_outline

settings

Enjoy our search

Hit / to insta-search docs and recipes!