WARN TaskSetManager: Lost task 0.0 in stage 0.0 ，TID 0, worker1: java.lang.ClassNotFoundException: com.spark.firstApp.HelloSpark$$anonfun$2

2024-03-08 08:50•java•阅读 4001

进行如下设置，解决报错信息。

val conf = new SparkConf().setAppName("helloSpark").setMaster("spark://master:7077").set("spark.executor.memory", "2g")

val sc = new SparkContext(conf)

sc.addJar("/home/spark/IdeaProjects/FirstApp/out/artifacts/FirstAppjar1/FirstApp.jar")

这对如下报错信息：

16/02/23 16:39:53 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, worker1): java.lang.ClassNotFoundException: com.spark.firstApp.HelloSpark$$anonfun$2

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)

at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)

at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)

......

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

16/02/23 16:39:53 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 2, worker0, partition 0,NODE_LOCAL, 2134 bytes)

16/02/23 16:39:53 INFO TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1) on executor worker2: java.lang.ClassNotFoundException (com.spark.firstApp.HelloSpark$$anonfun$2) [duplicate 1]

16/02/23 16:39:53 INFO TaskSetManager: Starting task 1.1 in stage 0.0 (TID 3, worker2, partition 1,NODE_LOCAL, 2134 bytes)

16/02/23 16:39:53 INFO TaskSetManager: Lost task 1.1 in stage 0.0 (TID 3) on executor worker2: java.lang.ClassNotFoundException (com.spark.firstApp.HelloSpark$$anonfun$2) [duplicate 2]

16/02/23 16:39:53 INFO TaskSetManager: Starting task 1.2 in stage 0.0 (TID 4, worker1, partition 1,NODE_LOCAL, 2134 bytes)

16/02/23 16:39:53 INFO TaskSetManager: Lost task 1.2 in stage 0.0 (TID 4) on executor worker1: java.lang.ClassNotFoundException (com.spark.firstApp.HelloSpark$$anonfun$2) [duplicate 3]

16/02/23 16:39:53 INFO TaskSetManager: Starting task 1.3 in stage 0.0 (TID 5, worker1, partition 1,NODE_LOCAL, 2134 bytes)

16/02/23 16:39:53 INFO TaskSetManager: Lost task 1.3 in stage 0.0 (TID 5) on executor worker1: java.lang.ClassNotFoundException (com.spark.firstApp.HelloSpark$$anonfun$2) [duplicate 4]

16/02/23 16:39:53 ERROR TaskSetManager: Task 1 in stage 0.0 failed 4 times; aborting job

16/02/23 16:39:53 INFO TaskSchedulerImpl: Cancelling stage 0

16/02/23 16:39:53 INFO TaskSchedulerImpl: Stage 0 was cancelled

16/02/23 16:39:53 INFO DAGScheduler: ShuffleMapStage 0 (map at HelloSpark.scala:20) failed in 10.015 s

16/02/23 16:39:53 INFO DAGScheduler: Job 0 failed: sortByKey at HelloSpark.scala:21, took 10.156984 s

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure: Lost task 1.3 in stage 0.0 (TID 5, worker1): java.lang.ClassNotFoundException: com.spark.firstApp.HelloSpark$$anonfun$2