CODE EXAMPLE FOR PYTHON

how to add new column in Spark

from pyspark.sql.functions import expr

# Using withColumn() method
foo2 = (foo.withColumn(
"status", expr("CASE WHEN delay <= 10 THEN 'On-time' ELSE 'Delayed' END")
))

# Output:
# +--------+-----+--------+------+-----------+-------+
# |    date|delay|distance|origin|destination| status|
# +--------+-----+--------+------+-----------+-------+
# |01010710|   31|     590|   SEA|        SFO|Delayed|
# |01010955|  104|     590|   SEA|        SFO|Delayed|
# |01010730|    5|     590|   SEA|        SFO|On-time|
# +--------+-----+--------+------+-----------+-------+