Spark Component Release Notes
This page lists the main features added to the Spark Component.
Change Log
Version 2023.1.0
Version 5.3.8 (Component Pack)
Feature improvements
-
DI-5872: The decimal precision defined is unexpectedly ignored.
-
DI-6088: The LOAD Hdfs XML to Spark template is now available.
-
DI-6397: Spark 1.6 templates have been removed.
-
DI-6571: The LOAD Hdfs XML to Spark template has been updated to ensure that the datastore and column names are not truncated.
Version 2.0.5 (Spark Component)
Feature improvements
-
DI-4011: Addition of ability to specify deploy mode (cluster or client).
-
DI-4012: Addition of ability to work with resource files through a new dedicated node in Metadata and a new parameter on the Spark Submit TOOL.
-
DI-4042: Addition of ability to handle spark session configuration when multiple targets are loaded with Spark.
Version 2.0.4 (Spark Component)
Feature improvements
-
DI-3580: Template - LOAD Hdfs File to Spark - new parameter "In File List".
-
DI-3581: Template - LOAD Hdfs File to Spark - support compressed files when using fileDriver Read Method.
-
DI-3719: Template - Load Hdfs Json to Spark - new Template to load JSON files stored in HDFS into Spark.
-
DI-3800: TOOL - Spark Execution Unit Launcher- add number of partitions to debug prints.