Python

Search

CSV data source does not support array data type

# In this example, change the field column_as_array to column_as_string before saving.

from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

def array_to_string(my_list):
    return '[' + ','.join([str(elem) for elem in my_list]) + ']'

array_to_string_udf = udf(array_to_string, StringType())

df = df.withColumn('column_as_str', array_to_string_udf(df["column_as_array"]))

# Then you can drop the old column (array type) before saving.
df.drop("column_as_array").write.csv(...)

Comment

PREVIOUS	NEXT

Code Example
Python :: Code of recursive binary search
Python :: python extract substring
Python :: python string cut left
Python :: python mathematics
Python :: change forms labels django
Python :: python turtle fill
Python :: how to add char to string python
Python :: how to make a dict from a file py
Python :: find optimal number of clusters sklearn
Python :: python access key in dictionary
Python :: numpy random choice
Python :: how to swap two variables without using third variable python
Python :: while activating env show error of unautorize access in vscode
Python :: overriding update in serializer django
Python :: aws django migrate
Python :: how to show mean values on histogram in seaborn
Python :: get source code selenium python
Python :: python if type dict
Python :: How to Adjust Title Position in Matplotlib
Python :: parse int python
Python :: python squared math function
Python :: prime numbers python
Python :: multiprocessing pool pass additional arguments
Python :: _set in django
Python :: sql like equivalent in python
Python :: Delete python text after 1 sec
Python :: start virtual environment python linux
Python :: Using a list with index and column names to Convert List to Dataframe
Python :: argmax implementation
Python :: opencv load image python

Search

PYTHON

CSV data source does not support array data type

ADD CONTENT