NickName:Brajesh Kumar Mishra Ask DateTime:2022-03-10T20:29:40

Wrong FS s3://ss-pprd-v2-dart//tempdir/962c6007-77c0-4294-b021-b9498e3d66ab/manifest.json -expected s3a://ss-pprd-v2-dart

I am using spark 3.2.1, Java8 ->1.8.0_292 (AdoptOpenJDK), Scala 2.12.10 and trying to read and write the data from/to redshift by using below mentioned jars and packages. But i am not able write the data back. While writing the data back to redshift. it was creating avro files with one manifest.json file in the temp directory but in my current versions it is not able to create the manifest.json file but it is creating all the avro files.

Jars and Packages:-

RedshiftJDBC42-no-awssdk-1.2.54.1082.jar,
hadoop-aws-3.3.1.jar,aws-java-sdk-1.12.173.jar ,
org.apache.spark:spark-avro_2.12:3.2.1,
io.github.spark-redshift-community:spark-redshift_2.12:5.0.3,
com.eclipsesource.minimal-json:minimal-json:0.9.5

Code i am trying to run:

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession

conf=SparkConf().setAppName("Testing")
sc=SparkContext.getOrCreate(conf)
sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", AWS_ACCESS_KEY)
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", AWS_SECRET_KEY)

df.write \
        .format("io.github.spark_redshift_community.spark.redshift")\
        .option("url", REDSHIFT_JDBC_URL) \
        .option("dbtable",MASTER_TABLE) \
        .option("forward_spark_s3_credentials", "true") \
        .option("extracopyoptions", EXTRACOPYOPTIONS) \
        .option("tempdir", "s3a://" + str(S3_BUCKET) + "/tempdir") \
        .mode("append") \
        .save()

print("Sucesss")

Stack Trace:

Traceback (most recent call last):
  File "/Users/brajeshmishra/Documents/TEMP/Temp_Py.py", line 65, in <module>
    .mode("append") \
  File "/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 738, in save
  File "/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/python/lib/py4j-0.10.9.3-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/opt/homebrew/Cellar/apache-

List item

spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/sql/utils.py", line 117, in deco
pyspark.sql.utils.IllegalArgumentException: Wrong FS s3://ss-pprd-v2-dart//tempdir/962c6007-77c0-4294-b021-b9498e3d66ab/manifest.json -expected s3a://ss-pprd-v2-dart

Copyright Notice：Content Author:「Brajesh Kumar Mishra」，Reproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article：https://stackoverflow.com/questions/71424185/wrong-fs-s3-ss-pprd-v2-dart-tempdir-962c6007-77c0-4294-b021-b9498e3d66ab-mani