Setting external jars to hadoop classpath

Your external jar is missing on the node running maps. You have to add it to the cache to make it available. Try :\n\nDistributedCache.addFileToClassPath(new Path(\"pathToJar\"), conf);\n\n\nNot sure in which version DistributedCache was deprecated, but from Hadoop 2.2.0 onward you can use :\n\njob.addFileToClassPath(new Path(\"pathToJar\")); \n",

If you are adding the external JAR to the Hadoop classpath then its better to copy your JAR to one of the existing directories that hadoop is looking at. On the command line run the command \"hadoop classpath\" and then find a suitable folder and copy your jar file to that location and hadoop will pick up the dependencies from there. This wont work with CloudEra etc as you may not have read/write rights to copy files to the hadoop classpath folders.\n\nLooks like you tried the LIBJARs option as well, did you edit your driver class to implement the TOOL interface? First make sure that you edit your driver class as shown below:\n\n public class myDriverClass extends Configured implements Tool {\n\n public static void main(String[] args) throws Exception {\n int res = ToolRunner.run(new Configuration(), new myDriverClass(), args);\n System.exit(res);\n }\n\n public int run(String[] args) throws Exception\n {\n\n // Configuration processed by ToolRunner \n Configuration conf = getConf();\n Job job = new Job(conf, \"My Job\");\n\n ...\n ...\n\n return job.waitForCompletion(true) ? 0 : 1;\n }\n }\n\n\nNow edit your \"hadoop jar\" command as shown below: \n\nhadoop jar YourApplication.jar [myDriverClass] args -libjars path/to/jar/file\n\n\nNow lets understand what happens underneath. Basically we are handling the new command line arguments by implementing the TOOL Interface. ToolRunner is used to run classes implementing Tool interface. It works in conjunction with GenericOptionsParser to parse the generic hadoop command line arguments and modifies the Configuration of the Tool. \n\nWithin our Main() we are calling ToolRunner.run(new Configuration(), new myDriverClass(), args) - this runs the given Tool by Tool.run(String[]), after parsing with the given generic arguments. It uses the given Configuration, or builds one if it's null and then sets the Tool's configuration with the possibly modified version of the conf.\n\nNow within the run method, when we call getConf() we get the modified version of the Configuration. So make sure that you have the below line in your code. If you implement everything else and still make use of Configuration conf = new Configuration(), nothing would work.\n\nConfiguration conf = getConf();\n",

I tried setting the opencsv jar in the hadoop classpath but it didn't work.We need to explicitly copy the jar in the classpath for this to work.It did worked for me.\nBelow are the steps i followed:\n\nI have done this in a HDP CLuster.I ahave copied my opencsv jar in hbase libs and exported it before running my jar\n\nCopy ExternalJars to HDP LIBS:\n\nTo Run Open CSV Jar:\n1.Copy the opencsv jar in directory /usr/hdp/2.2.9.1-11/hbase/lib/ and /usr/hdp/2.2.9.1-11/hadoop-yarn/lib\n\n**sudo cp /home/sshuser/Amedisys/lib/opencsv-3.7.jar /usr/hdp/2.2.9.1-11/hbase/lib/**\n\n\n2.Give the file permissions using \n sudo chmod 777 opencsv-3.7.jar\n3.List Files\n ls -lrt\n\n4.exporthadoop classpath:hbase classpath\n\n5.Now run your Jar.It will pick up the opencsv jar and will execute properly.",

I found another workaround by implementing ToolRunner like below. With this approach hadoop accepts command line options. We can avoid hard coding of adding files to DistributedCache\n\n public class FlightsByCarrier extends Configured implements Tool {\n\n public int run(String[] args) throws Exception {\n // Configuration processed by ToolRunner\n Configuration conf = getConf();\n\n // Create a JobConf using the processed conf\n JobConf job = new JobConf(conf, FlightsByCarrier.class);\n\n // Process custom command-line options\n Path in = new Path(args[1]);\n Path out = new Path(args[2]);\n\n // Specify various job-specific parameters \n job.setJobName(\"my-app\");\n job.setInputPath(in);\n job.setOutputPath(out);\n job.setMapperClass(MyMapper.class);\n job.setReducerClass(MyReducer.class);\n\n // Submit the job, then poll for progress until the job is complete\n JobClient.runJob(job);\n return 0;\n }\n\n public static void main(String[] args) throws Exception {\n // Let ToolRunner handle generic command-line options \n int res = ToolRunner.run(new Configuration(), new FlightsByCarrier(), args);\n\n System.exit(res);\n }\n }\n",