How to convert json file to csv file using python or spark dataframe
NickName:pythonUser Ask DateTime:2019-04-09T23:00:44

How to convert json file to csv file using python or spark dataframe

I have to convert json file to csv file using spark dataframe in databricks. I have tried the below code to convert the json to csv but i'm getting the CSV data source does not support array data type in spark dataframe . I'm unable to convert to csv file .can someone help me on this issue how to remove _corrupt_string?

 import json
    data=r'/dbfs/FileStore/tables/ABC.json'
    print ("This is json data ", data)
    def js_r(data):
       with open(data, encoding='utf-8') as f_in:
           return(json.load(f_in))

    if __name__ == "__main__":
        dic_data_first = js_r(data)
        print("This is my dictionary", dic_data_first)
    keys= dic_data_first.keys()
    print ("The original dict keys",keys)
    dic_data_second={'my_items':dic_data_first['Data']for key in keys}
    with open('/dbfs/FileStore/tables/ABC_1.json', 'w') as f:   
         json.dump(dic_data_first, f)
    df = sqlContext.read.json('dbfs:/FileStore/tables/ABC_1.json')   # reading a json and writing a  parquet
    print(df)
df.write.mode("overwrite").format("com.databricks.spark.csv").option("header","true").csv("/dbfs/FileStore/tables/ABC_1.csv")
JSON data as follows:
{"Table":"test1",
  "Data":[
{"aa":"1",
 "bb":"2"},
{"aa" :"ss",
"bb":"dc"}            
}]
}

Copyright Notice:Content Author:「pythonUser」,Reproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/55595804/how-to-convert-json-file-to-csv-file-using-python-or-spark-dataframe

More about “How to convert json file to csv file using python or spark dataframe” related questions

How to convert json file to csv file using python or spark dataframe

I have to convert json file to csv file using spark dataframe in databricks. I have tried the below code to convert the json to csv but i'm getting the CSV data source does not support array data ...

Show Detail

Split Spark Dataframe to each row and convert to JSON - Python

I am a newbie to Spark and am trying to read & research as much as I can. Currently I am stuck on this and I have spent few days for solving. I have successfully set up a Spark Clusters on 3 ma...

Show Detail

How to convert Spark dataframe output to json?

I am reading file with CSV file with Spark SQL Context. Code : m.put("path", CSV_DIRECTORY+file.getOriginalFilename()); m.put("inferSchema", "true"); // Automatically infer data types else strin...

Show Detail

Convert spark to pandas dataframe has Exception: arrow is not supported when using file-based collect

I am trying to convert a spark dataframe to pandas dataframe on Azure databricks. But I get the following error: Exception: arrow is not supported when using file-based collect I have tried the

Show Detail

Convert Dask Dataframe to Spark dataframe using Python

I want to convert Dask Dataframe to Spark Dataframe. Let's consider this example: import dask.dataframe as dd dask_df = dd.read_csv("file_name.csv") # convert dask df to spark df spark_d...

Show Detail

Convert csv to parquet file using python

I am trying to convert a .csv file to a .parquet file. The csv file (Temp.csv) has the following format 1,Jon,Doe,Denver I am using the following python code to convert it into parquet from

Show Detail

Convert csv to parquet file using python

I am trying to convert a .csv file to a .parquet file. The csv file (Temp.csv) has the following format 1,Jon,Doe,Denver I am using the following python code to convert it into parquet from

Show Detail

Writing a big Spark Dataframe into a csv file

I'm using Spark 2.3 and I need to save a Spark Dataframe into a csv file and I'm looking for a better way to do it.. looking over related/similar questions, I found this one, but I need a more spec...

Show Detail

Importing csv file into spark dataframe

I'm trying to import csv file using pyspark. I tried this and this. Using the first method I could read the csv file. But number of variables are quite large. So manually mentioning the variable n...

Show Detail

Converting a spark dataframe that contains Vector as a feature into CSV file

I need to convert a spark dataframe into a CSV file. The problem is that one of the features is a Vector structure and I am not sure how to deal with it. I got this dataframe as a result of a lda.

Show Detail