Skip to content

Instantly share code, notes, and snippets.

@zyazxr

zyazxr/DNN回归

Created August 9, 2019 08:28
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save zyazxr/929e31e79f8c9f481f5f793c0b020570 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save zyazxr/929e31e79f8c9f481f5f793c0b020570 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
DNN回归
"C:\Program Files\Java\jdk1.8.0_191\bin\java.exe" -agentlib:jdwp=transport=dt_socket,address=127.0.0.1:52686,suspend=y,server=n -javaagent:C:\Users\zhaoy\.IntelliJIdea2018.3\system\captureAgent\debugger-agent.jar -Dfile.encoding=UTF-8 -classpath C:\Users\zhaoy\AppData\Local\Temp\classpath61260294.jar com.nari.bdp.mine_server_test_behavior.classification.DNNClassificationTest
Connected to the target VM, address: '127.0.0.1:52686', transport: 'socket'
/C:/Users/zhaoy/IdeaProjects/nari.work/work2/execute_behavior/mine/test_behavior/target/classes/test_xml/bdp/regression/DNNClassfication.xml
[INFO ] 2019-08-09 16:01:50 [main] c.n.b.m.e.service.MineJobExecutor - =========开始执行=========
########### className = com.nari.bdp.mine.cloud.io.readdatabank.behavior.ReadDataBankBehavior
########### className = com.nari.bdp.mine.cloud.dp.setrole.behavior.SetroleBehavior
########### className = com.nari.bdp.mine.cloud.classification.classificationdnn.behavior.ClassificationdnnBehavior
cost time: 1.035000s
[INFO ] 2019-08-09 16:01:50 [main] c.n.b.m.c.i.r.b.ReadDataBankBehavior - ===========开始执行 com.nari.bdp.mine.cloud.io.readdatabank.behavior.ReadDataBankBehavior节点=============
文件数据源 cost time: 27.648000s
Temp--static column：pollution,dew,temp,press,wnd_spd,snow,rain used 4.202 seconds
Temp--static column：null,null,null,null,null,null,null,null,null used 0.0 seconds
+--------------+---------+---+----+-----+-------+-------+----+----+
| date_time|pollution|dew|temp|press|wnd_dir|wnd_spd|snow|rain|
+--------------+---------+---+----+-----+-------+-------+----+----+
| 2014/1/1 0:00| 24|-20| 7| 1014| NW| 143.48| 0| 0|
| 2014/1/1 1:00| 53|-20| 7| 1013| NW| 147.5| 0| 0|
| 2014/1/1 2:00| 65|-20| 6| 1013| NW| 151.52| 0| 0|
| 2014/1/1 3:00| 70|-20| 6| 1013| NW| 153.31| 0| 0|
| 2014/1/1 4:00| 79|-18| 3| 1012| cv| 0.89| 0| 0|
| 2014/1/1 5:00| 92|-18| 4| 1012| NW| 4.02| 0| 0|
| 2014/1/1 6:00| 106|-19| 6| 1012| NW| 8.94| 0| 0|
| 2014/1/1 7:00| 75|-19| 6| 1013| NW| 16.09| 0| 0|
| 2014/1/1 8:00| 58|-18| 6| 1013| NW| 21.9| 0| 0|
| 2014/1/1 9:00| 33|-18| 7| 1014| NW| 26.82| 0| 0|
|2014/1/1 10:00| 51|-18| 8| 1015| NW| 31.74| 0| 0|
|2014/1/1 11:00| 32|-18| 9| 1015| NW| 39.79| 0| 0|
|2014/1/1 12:00| 23|-17| 10| 1015| NW| 48.73| 0| 0|
|2014/1/1 13:00| 28|-18| 11| 1014| NW| 55.88| 0| 0|
|2014/1/1 14:00| 23|-17| 11| 1014| NW| 63.93| 0| 0|
|2014/1/1 15:00| 24|-17| 11| 1014| NW| 71.08| 0| 0|
|2014/1/1 16:00| 26|-17| 11| 1014| NW| 76.89| 0| 0|
|2014/1/1 17:00| 26|-16| 10| 1015| NW| 81.81| 0| 0|
|2014/1/1 18:00| 27|-16| 9| 1015| NW| 84.94| 0| 0|
|2014/1/1 19:00| 43|-16| 9| 1016| SE| 1.79| 0| 0|
+--------------+---------+---+----+-----+-------+-------+----+----+
only showing top 20 rows
设置角色 cost time: 0.569000s
+--------------+---------+---+----+-----+-------+-------+----+----+
| date_time|pollution|dew|temp|press|wnd_dir|wnd_spd|snow|rain|
+--------------+---------+---+----+-----+-------+-------+----+----+
| 2014/1/1 0:00| 24|-20| 7| 1014| NW| 143.48| 0| 0|
| 2014/1/1 1:00| 53|-20| 7| 1013| NW| 147.5| 0| 0|
| 2014/1/1 2:00| 65|-20| 6| 1013| NW| 151.52| 0| 0|
| 2014/1/1 3:00| 70|-20| 6| 1013| NW| 153.31| 0| 0|
| 2014/1/1 4:00| 79|-18| 3| 1012| cv| 0.89| 0| 0|
| 2014/1/1 5:00| 92|-18| 4| 1012| NW| 4.02| 0| 0|
| 2014/1/1 6:00| 106|-19| 6| 1012| NW| 8.94| 0| 0|
| 2014/1/1 7:00| 75|-19| 6| 1013| NW| 16.09| 0| 0|
| 2014/1/1 8:00| 58|-18| 6| 1013| NW| 21.9| 0| 0|
| 2014/1/1 9:00| 33|-18| 7| 1014| NW| 26.82| 0| 0|
|2014/1/1 10:00| 51|-18| 8| 1015| NW| 31.74| 0| 0|
|2014/1/1 11:00| 32|-18| 9| 1015| NW| 39.79| 0| 0|
|2014/1/1 12:00| 23|-17| 10| 1015| NW| 48.73| 0| 0|
|2014/1/1 13:00| 28|-18| 11| 1014| NW| 55.88| 0| 0|
|2014/1/1 14:00| 23|-17| 11| 1014| NW| 63.93| 0| 0|
|2014/1/1 15:00| 24|-17| 11| 1014| NW| 71.08| 0| 0|
|2014/1/1 16:00| 26|-17| 11| 1014| NW| 76.89| 0| 0|
|2014/1/1 17:00| 26|-16| 10| 1015| NW| 81.81| 0| 0|
|2014/1/1 18:00| 27|-16| 9| 1015| NW| 84.94| 0| 0|
|2014/1/1 19:00| 43|-16| 9| 1016| SE| 1.79| 0| 0|
|2014/1/1 20:00| 62|-14| 3| 1017| SE| 2.68| 0| 0|
|2014/1/1 21:00| 70|-14| 0| 1017| NE| 1.79| 0| 0|
|2014/1/1 22:00| 81|-12| -1| 1018| SE| 0.89| 0| 0|
|2014/1/1 23:00| 111|-12| 0| 1019| NW| 1.79| 0| 0|
| 2014/1/2 0:00| 144|-13| -2| 1019| NE| 0.89| 0| 0|
| 2014/1/2 1:00| 170|-12| -4| 1019| cv| 0.89| 0| 0|
| 2014/1/2 2:00| 174|-12| -4| 1019| cv| 1.34| 0| 0|
| 2014/1/2 3:00| 174|-12| -4| 1019| NW| 0.89| 0| 0|
| 2014/1/2 4:00| 172|-12| -5| 1020| cv| 0.89| 0| 0|
| 2014/1/2 5:00| 149|-10| -2| 1020| SE| 1.79| 0| 0|
| 2014/1/2 6:00| 166| -7| -2| 1020| SE| 3.58| 0| 0|
| 2014/1/2 7:00| 187| -9| -5| 1020| NE| 0.89| 0| 0|
| 2014/1/2 8:00| 107| -9| -5| 1021| NW| 3.13| 0| 0|
| 2014/1/2 9:00| 114| -7| -2| 1021| NW| 4.92| 0| 0|
|2014/1/2 10:00| 108| -7| 2| 1021| NW| 6.71| 0| 0|
|2014/1/2 11:00| 102| -8| 4| 1020| cv| 0.89| 0| 0|
|2014/1/2 12:00| 95| -8| 5| 1019| cv| 1.78| 0| 0|
|2014/1/2 13:00| 127| -9| 7| 1017| cv| 2.67| 0| 0|
|2014/1/2 14:00| 125| -9| 7| 1016| cv| 4.46| 0| 0|
|2014/1/2 15:00| 128|-10| 7| 1016| cv| 5.35| 0| 0|
|2014/1/2 16:00| 146|-10| 6| 1016| cv| 5.8| 0| 0|
|2014/1/2 17:00| 165| -9| 4| 1016| cv| 6.69| 0| 0|
|2014/1/2 18:00| 173| -8| 3| 1016| cv| 7.58| 0| 0|
|2014/1/2 19:00| 195| -9| 1| 1016| cv| 8.47| 0| 0|
|2014/1/2 20:00| 239| -8| 0| 1017| cv| 9.36| 0| 0|
|2014/1/2 21:00| 232| -8| 0| 1017| SE| 0.89| 0| 0|
|2014/1/2 22:00| 242| -8| -1| 1017| NE| 1.79| 0| 0|
|2014/1/2 23:00| 269| -7| -1| 1018| NW| 4.02| 0| 0|
| 2014/1/3 0:00| 264| -9| 0| 1018| NW| 7.15| 0| 0|
| 2014/1/3 1:00| 220| -9| 1| 1018| cv| 0.45| 0| 0|
| 2014/1/3 2:00| 146| -9| 0| 1019| NW| 4.02| 0| 0|
| 2014/1/3 3:00| 34| -9| 1| 1019| cv| 0.89| 0| 0|
| 2014/1/3 4:00| 34| -9| -2| 1020| NW| 3.13| 0| 0|
| 2014/1/3 5:00| 35| -9| -1| 1020| NW| 7.15| 0| 0|
| 2014/1/3 6:00| 45|-13| 0| 1020| cv| 0.89| 0| 0|
| 2014/1/3 7:00| 43|-13| -1| 1022| NW| 3.13| 0| 0|
| 2014/1/3 8:00| 43|-12| -1| 1023| NW| 7.15| 0| 0|
| 2014/1/3 9:00| 36|-13| 4| 1024| NW| 11.17| 0| 0|
|2014/1/3 10:00| 36|-16| 7| 1025| NW| 14.3| 0| 0|
|2014/1/3 11:00| 23|-16| 8| 1024| NW| 17.43| 0| 0|
|2014/1/3 12:00| 25|-16| 8| 1023| NW| 20.56| 0| 0|
|2014/1/3 13:00| 29|-17| 9| 1022| NW| 22.35| 0| 0|
|2014/1/3 14:00| 26|-18| 9| 1022| NE| 1.79| 0| 0|
|2014/1/3 15:00| 21|-17| 8| 1022| NE| 3.58| 0| 0|
|2014/1/3 16:00| 25|-17| 7| 1022| SE| 1.79| 0| 0|
|2014/1/3 17:00| 31|-17| 5| 1023| SE| 3.58| 0| 0|
|2014/1/3 18:00| 43|-17| 3| 1023| cv| 0.45| 0| 0|
|2014/1/3 19:00| 46|-17| 3| 1024| cv| 1.34| 0| 0|
|2014/1/3 20:00| 50|-16| 1| 1024| cv| 2.23| 0| 0|
|2014/1/3 21:00| 68|-13| -1| 1024| cv| 2.68| 0| 0|
|2014/1/3 22:00| 60|-11| -1| 1025| SE| 1.79| 0| 0|
|2014/1/3 23:00| 103| -8| -1| 1025| SE| 4.92| 0| 0|
| 2014/1/4 0:00| 85| -6| -1| 1026| SE| 6.71| 0| 0|
| 2014/1/4 1:00| 86| -6| -2| 1025| SE| 8.5| 0| 0|
| 2014/1/4 2:00| 89| -7| -2| 1025| cv| 0.89| 0| 0|
| 2014/1/4 3:00| 77| -7| -2| 1025| SE| 1.79| 0| 0|
| 2014/1/4 4:00| 77| -9| -5| 1025| cv| 0.45| 0| 0|
| 2014/1/4 5:00| 75| -9| -5| 1025| cv| 1.34| 0| 0|
| 2014/1/4 6:00| 80| -9| -6| 1025| cv| 1.79| 0| 0|
| 2014/1/4 7:00| 86| -7| -4| 1025| SE| 0.89| 0| 0|
| 2014/1/4 8:00| 95| -7| -3| 1025| cv| 0.89| 0| 0|
| 2014/1/4 9:00| 101| -6| -2| 1025| SE| 0.89| 0| 0|
|2014/1/4 10:00| 132| -6| -1| 1025| SE| 1.78| 0| 0|
|2014/1/4 11:00| 153| -5| -1| 1024| SE| 3.57| 0| 0|
|2014/1/4 12:00| 173| -6| 0| 1023| NE| 1.79| 0| 0|
|2014/1/4 13:00| 178| -7| 1| 1021| cv| 0.89| 0| 0|
|2014/1/4 14:00| 176| -7| 2| 1020| SE| 1.79| 0| 0|
|2014/1/4 15:00| 209| -7| 2| 1020| cv| 0.89| 0| 0|
|2014/1/4 16:00| 219| -7| 2| 1020| NW| 1.79| 0| 0|
|2014/1/4 17:00| 224| -7| 1| 1020| NW| 3.58| 0| 0|
|2014/1/4 18:00| 212| -8| -2| 1020| NW| 6.71| 0| 0|
|2014/1/4 19:00| 221| -8| -3| 1021| NW| 8.5| 0| 0|
|2014/1/4 20:00| 221| -8| -4| 1021| NW| 10.29| 0| 0|
|2014/1/4 21:00| 217| -8| -4| 1021| cv| 0.89| 0| 0|
|2014/1/4 22:00| 203| -8| -4| 1021| cv| 1.78| 0| 0|
|2014/1/4 23:00| 221| -9| -5| 1021| NW| 1.79| 0| 0|
| 2014/1/5 0:00| 192| -8| -4| 1021| NW| 3.58| 0| 0|
| 2014/1/5 1:00| 183|-10| -6| 1022| NW| 6.71| 0| 0|
| 2014/1/5 2:00| 175| -8| -5| 1022| NW| 10.73| 0| 0|
| 2014/1/5 3:00| 177| -9| -6| 1022| NW| 14.75| 0| 0|
+--------------+---------+---+----+-----+-------+-------+----+----+
only showing top 100 rows
root
|-- date_time: string (nullable = true)
|-- pollution: integer (nullable = true)
|-- dew: integer (nullable = true)
|-- temp: integer (nullable = true)
|-- press: integer (nullable = true)
|-- wnd_dir: string (nullable = true)
|-- wnd_spd: double (nullable = true)
|-- snow: integer (nullable = true)
|-- rain: integer (nullable = true)
|-- predicition: string (nullable = true)
|-- probability: vector (nullable = false)
|-- prob_SE: double (nullable = true)
|-- prob_NW: double (nullable = true)
|-- prob_cv: double (nullable = true)
|-- prob_NE: double (nullable = true)
DNN分类 cost time: 712.532000s
org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 28.0 failed 1 times, most recent failure: Lost task 3.0 in stage 28.0 (TID 79, localhost, executor driver): java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, date_time), StringType), true, false) AS date_time#335
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, pollution), IntegerType) AS pollution#336
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, dew), IntegerType) AS dew#337
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, temp), IntegerType) AS temp#338
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, press), IntegerType) AS press#339
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, wnd_dir), StringType), true, false) AS wnd_dir#340
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 6, wnd_spd), DoubleType) AS wnd_spd#341
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 7, snow), IntegerType) AS snow#342
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 8, rain), IntegerType) AS rain#343
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 9, label_numeric), DoubleType) AS label_numeric#344
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS featureoutput#345
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS minMaxOutput#346
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 12, predicition), DoubleType) AS predicition#347
newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS probability#348
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:291)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:216)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:295)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:266)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.Invoke2$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields3_5$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:288)
... 25 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1599)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1587)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1586)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1586)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1820)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1769)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1758)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2027)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2067)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:363)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3272)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2484)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2484)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3253)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3252)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2484)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2698)
at com.nari.bdp.mine.base.utils.MineBaseUtil.convertDfToJavaListDColumns(MineBaseUtil.java:76)
at com.nari.bdp.mine.base.utils.MineBaseUtil.convertDfToJavaList(MineBaseUtil.java:28)
at com.nari.bdp.mine.operator.impl.AutoBehavior.getResuleTable(AutoBehavior.java:646)
at com.nari.bdp.mine.operator.impl.AutoBehavior.getDatasetMeta(AutoBehavior.java:475)
at com.nari.bdp.mine.operator.impl.AutoBehavior.generateDatasetInsight(AutoBehavior.java:499)
at com.nari.bdp.mine.operator.impl.AutoBehavior.generatePortInsight(AutoBehavior.java:523)
at com.nari.bdp.mine.operator.impl.AutoBehavior.leave(AutoBehavior.java:612)
at com.nari.bdp.mine.operator.impl.AutoBehavior.executeBehavior(AutoBehavior.java:437)
at com.nari.bdp.mine.operator.impl.AutoBehavior.leave(AutoBehavior.java:635)
at com.nari.bdp.mine.operator.impl.AutoBehavior.executeBehavior(AutoBehavior.java:437)
at com.nari.bdp.mine.operator.impl.AutoBehavior.leave(AutoBehavior.java:635)
at com.nari.bdp.mine.operator.impl.AutoBehavior.executeBehavior(AutoBehavior.java:437)
at com.nari.bdp.mine.operator.impl.AutoBehavior.leave(AutoBehavior.java:635)
at com.nari.bdp.mine.operator.impl.AutoBehavior.executeBehavior(AutoBehavior.java:437)
at com.nari.bdp.mine.operator.impl.ExecutionImpl.executeNode(ExecutionImpl.java:75)
at com.nari.bdp.mine.operator.impl.ProcessImpl.run(ProcessImpl.java:79)
at com.nari.bdp.mine.executor.executor.Executor.rProcessExecute(Executor.java:125)
at com.nari.bdp.mine.executor.executor.Executor.rProcessExecute(Executor.java:111)
at com.nari.bdp.mine.executor.executor.Executor.excute(Executor.java:89)
at com.nari.bdp.mine.executor.service.MineJobExecutor.jobExecutor(MineJobExecutor.java:75)
at com.nari.bdp.mine_server_test_behavior.classification.DNNClassificationTest.main(DNNClassificationTest.java:34)
Caused by: java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, date_time), StringType), true, false) AS date_time#335
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, pollution), IntegerType) AS pollution#336
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, dew), IntegerType) AS dew#337
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, temp), IntegerType) AS temp#338
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, press), IntegerType) AS press#339
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, wnd_dir), StringType), true, false) AS wnd_dir#340
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 6, wnd_spd), DoubleType) AS wnd_spd#341
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 7, snow), IntegerType) AS snow#342
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 8, rain), IntegerType) AS rain#343
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 9, label_numeric), DoubleType) AS label_numeric#344
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS featureoutput#345
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS minMaxOutput#346
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 12, predicition), DoubleType) AS predicition#347
newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS probability#348
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:291)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:216)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:295)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:266)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.Invoke2$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields3_5$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:288)
... 25 more
process: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 28.0 failed 1 times, most recent failure: Lost task 3.0 in stage 28.0 (TID 79, localhost, executor driver): java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, date_time), StringType), true, false) AS date_time#335
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, pollution), IntegerType) AS pollution#336
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, dew), IntegerType) AS dew#337
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, temp), IntegerType) AS temp#338
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, press), IntegerType) AS press#339
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, wnd_dir), StringType), true, false) AS wnd_dir#340
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 6, wnd_spd), DoubleType) AS wnd_spd#341
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 7, snow), IntegerType) AS snow#342
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 8, rain), IntegerType) AS rain#343
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 9, label_numeric), DoubleType) AS label_numeric#344
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS featureoutput#345
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS minMaxOutput#346
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 12, predicition), DoubleType) AS predicition#347
newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS probability#348
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:291)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:216)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:295)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:266)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.Invoke2$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields3_5$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:288)
... 25 more
Driver stacktrace:
process: java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, date_time), StringType), true, false) AS date_time#335
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, pollution), IntegerType) AS pollution#336
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, dew), IntegerType) AS dew#337
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, temp), IntegerType) AS temp#338
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, press), IntegerType) AS press#339
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, wnd_dir), StringType), true, false) AS wnd_dir#340
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 6, wnd_spd), DoubleType) AS wnd_spd#341
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 7, snow), IntegerType) AS snow#342
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 8, rain), IntegerType) AS rain#343
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 9, label_numeric), DoubleType) AS label_numeric#344
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS featureoutput#345
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS minMaxOutput#346
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 12, predicition), DoubleType) AS predicition#347
newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS probability#348
process: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
Match Error: Job aborted due to stage failure: Task 3 in stage 28.0 failed 1 times, most recent failure: Lost task 3.0 in stage 28.0 (TID 79, localhost, executor driver): java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, date_time), StringType), true, false) AS date_time#335
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, pollution), IntegerType) AS pollution#336
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, dew), IntegerType) AS dew#337
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, temp), IntegerType) AS temp#338
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, press), IntegerType) AS press#339
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, wnd_dir), StringType), true, false) AS wnd_dir#340
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 6, wnd_spd), DoubleType) AS wnd_spd#341
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 7, snow), IntegerType) AS snow#342
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 8, rain), IntegerType) AS rain#343
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 9, label_numeric), DoubleType) AS label_numeric#344
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS featureoutput#345
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS minMaxOutput#346
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 12, predicition), DoubleType) AS predicition#347
newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS probability#348
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:291)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:216)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:295)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:266)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.Invoke2$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields3_5$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:288)
... 25 more
Driver stacktrace:
2019-08-09 16:18:50,780 - com.nari.bdp.mine.operator.impl.ExecutionImpl -5 [main] ERROR - Job aborted due to stage failure: Task 3 in stage 28.0 failed 1 times, most recent failure: Lost task 3.0 in stage 28.0 (TID 79, localhost, executor driver): java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, date_time), StringType), true, false) AS date_time#335
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, pollution), IntegerType) AS pollution#336
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, dew), IntegerType) AS dew#337
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, temp), IntegerType) AS temp#338
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, press), IntegerType) AS press#339
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, wnd_dir), StringType), true, false) AS wnd_dir#340
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 6, wnd_spd), DoubleType) AS wnd_spd#341
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 7, snow), IntegerType) AS snow#342
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 8, rain), IntegerType) AS rain#343
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 9, label_numeric), DoubleType) AS label_numeric#344
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS featureoutput#345
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS minMaxOutput#346
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 12, predicition), DoubleType) AS predicition#347
newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS probability#348
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:291)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:216)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:295)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:266)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.Invoke2$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields3_5$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:288)
... 25 more
Driver stacktrace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 28.0 failed 1 times, most recent failure: Lost task 3.0 in stage 28.0 (TID 79, localhost, executor driver): java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, date_time), StringType), true, false) AS date_time#335
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, pollution), IntegerType) AS pollution#336
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, dew), IntegerType) AS dew#337
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, temp), IntegerType) AS temp#338
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, press), IntegerType) AS press#339
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, wnd_dir), StringType), true, false) AS wnd_dir#340
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 6, wnd_spd), DoubleType) AS wnd_spd#341
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 7, snow), IntegerType) AS snow#342
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 8, rain), IntegerType) AS rain#343
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 9, label_numeric), DoubleType) AS label_numeric#344
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS featureoutput#345
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS minMaxOutput#346
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 12, predicition), DoubleType) AS predicition#347
newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS probability#348
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:291)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:216)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:295)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:266)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.Invoke2$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields3_5$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:288)
... 25 more
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1599)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1587)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1586)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1586)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1820)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1769)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1758)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2027)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2067)
at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:363)
at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:3272)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2484)
at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2484)
at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3253)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3252)
at org.apache.spark.sql.Dataset.head(Dataset.scala:2484)
at org.apache.spark.sql.Dataset.take(Dataset.scala:2698)
at com.nari.bdp.mine.base.utils.MineBaseUtil.convertDfToJavaListDColumns(MineBaseUtil.java:76)
at com.nari.bdp.mine.base.utils.MineBaseUtil.convertDfToJavaList(MineBaseUtil.java:28)
at com.nari.bdp.mine.operator.impl.AutoBehavior.getResuleTable(AutoBehavior.java:646)
at com.nari.bdp.mine.operator.impl.AutoBehavior.getDatasetMeta(AutoBehavior.java:475)
at com.nari.bdp.mine.operator.impl.AutoBehavior.generateDatasetInsight(AutoBehavior.java:499)
at com.nari.bdp.mine.operator.impl.AutoBehavior.generatePortInsight(AutoBehavior.java:523)
at com.nari.bdp.mine.operator.impl.AutoBehavior.leave(AutoBehavior.java:612)
at com.nari.bdp.mine.operator.impl.AutoBehavior.executeBehavior(AutoBehavior.java:437)
at com.nari.bdp.mine.operator.impl.AutoBehavior.leave(AutoBehavior.java:635)
at com.nari.bdp.mine.operator.impl.AutoBehavior.executeBehavior(AutoBehavior.java:437)
at com.nari.bdp.mine.operator.impl.AutoBehavior.leave(AutoBehavior.java:635)
at com.nari.bdp.mine.operator.impl.AutoBehavior.executeBehavior(AutoBehavior.java:437)
at com.nari.bdp.mine.operator.impl.AutoBehavior.leave(AutoBehavior.java:635)
at com.nari.bdp.mine.operator.impl.AutoBehavior.executeBehavior(AutoBehavior.java:437)
at com.nari.bdp.mine.operator.impl.ExecutionImpl.executeNode(ExecutionImpl.java:75)
at com.nari.bdp.mine.operator.impl.ProcessImpl.run(ProcessImpl.java:79)
at com.nari.bdp.mine.executor.executor.Executor.rProcessExecute(Executor.java:125)
at com.nari.bdp.mine.executor.executor.Executor.rProcessExecute(Executor.java:111)
at com.nari.bdp.mine.executor.executor.Executor.excute(Executor.java:89)
at com.nari.bdp.mine.executor.service.MineJobExecutor.jobExecutor(MineJobExecutor.java:75)
at com.nari.bdp.mine_server_test_behavior.classification.DNNClassificationTest.main(DNNClassificationTest.java:34)
Caused by: java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, date_time), StringType), true, false) AS date_time#335
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, pollution), IntegerType) AS pollution#336
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, dew), IntegerType) AS dew#337
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, temp), IntegerType) AS temp#338
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, press), IntegerType) AS press#339
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, wnd_dir), StringType), true, false) AS wnd_dir#340
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 6, wnd_spd), DoubleType) AS wnd_spd#341
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 7, snow), IntegerType) AS snow#342
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 8, rain), IntegerType) AS rain#343
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 9, label_numeric), DoubleType) AS label_numeric#344
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS featureoutput#345
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS minMaxOutput#346
validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 12, predicition), DoubleType) AS predicition#347
newInstance(class org.apache.spark.ml.linalg.VectorUDT).serialize AS probability#348
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:291)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:589)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
at org.apache.spark.sql.execution.UnsafeExternalRowSorter.sort(UnsafeExternalRowSorter.java:216)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:295)
at org.apache.spark.sql.execution.exchange.ShuffleExchangeExec$$anonfun$2.apply(ShuffleExchangeExec.scala:266)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
at org.apache.spark.scheduler.Task.run(Task.scala:109)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.spark.mllib.linalg.DenseVector is not a valid external type for schema of vector
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.Invoke2$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields3_5$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:288)
... 25 more
com.meritdata.tempo.force.exit is:null
Disconnected from the target VM, address: '127.0.0.1:52686', transport: 'socket'
Process finished with exit code 1024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment