
        
        
            
                
                
                    df = spark.createDataFrame([["Alex", 20], ["Bob", 25], ["Bob", 30]], ["name", "age"])
df.show()
                
            
            +----+---+
|name|age|
+----+---+
|Alex| 20|
| Bob| 25|
| Bob| 30|
+----+---+

Getting summary statistics of certain columns in PySpark DataFrame

To get the summary statistics of the name and age columns:


        
        
            
                
                
                    df.describe("name","age").show()
                
            
            +-------+----+----+
|summary|name| age|
+-------+----+----+
|  count|   3|   3|
|   mean|null|25.0|
| stddev|null| 5.0|
|    min|Alex|  20|
|    max| Bob|  30|
+-------+----+----+

Getting summary statistics of all numeric and string columns in PySpark DataFrame

To get the summary statistics of all numeric and string columns:


        
        
            
                
                
                    df.describe().show()
                
            
            +-------+----+----+
|summary|name| age|
+-------+----+----+
|  count|   3|   3|
|   mean|null|25.0|
| stddev|null| 5.0|
|    min|Alex|  20|
|    max| Bob|  30|
+-------+----+----+