Release Notes - Spring XD - Version M6 - HTML format

Technical task

  • [XD-747] - Bootstrap XD on Yarn

Bug

  • [XD-630] - StreamCommandTests - asserting sink contents sometimes failing
  • [XD-1017] - Spring Batch Admin UI looks to localhost when getting status updates
  • [XD-1162] - Column option of JDBC sink should not convert underscore to property name.
  • [XD-1169] - Column name of JDBC sink module should not hard code to "payload".
  • [XD-1170] - Splunk module is broken
  • [XD-1173] - Duplicate messages on tap
  • [XD-1264] - Update SI to latest 4.0 M3 and Spring AMQP to 1.3.0.M2
  • [XD-1266] - Twitterstream is broken
  • [XD-1280] - JMX endpoints not functioning
  • [XD-1316] - UI:Fix E2E test warning
  • [XD-1318] - XD container can not be started before the admin server
  • [XD-1327] - Rabbit source module with outputType fails to deploy
  • [XD-1352] - Job Launch should throw exception if the job is not deployed in the container
  • [XD-1356] - Hadoop distro option hdp20 is broken
  • [XD-1373] - HSQL always started, even when using other database
  • [XD-1384] - Improve job launch functionality with distributed nodes
  • [XD-1408] - Container ID is not equal to its application context ID
  • [XD-1494] - OS commands no longer supports whitespace/arguments in M6
  • [XD-1495] - xd:>runtime modules gives error from CLI
  • [XD-1502] - Investigate failing LocalSingleNodeStreamDeploymentIntegrationTests
  • [XD-1505] - Documentation typo in JSON SPEL filter
  • [XD-1508] - All jobs end up on the same container node
  • [XD-1526] - Exception when accessing CDH4 namenode
  • [XD-1530] - Error when removing HDFS files in shell
  • [XD-1532] - Clean up MBean registration for failed module deployments
  • [XD-1533] - Admin needs to clean up failed deployment attempts
  • [XD-1534] - Twitter Search results are not deserialized
  • [XD-1535] - TwitterSearch does not deploy correctly if previous deploy fails
  • [XD-1541] - Batch Basic Fails to launch job when rabbit is data transport
  • [XD-1546] - Batch Notification Sample fails to execute
  • [XD-1547] - clean up dead entries in ZooKeeper /xd/deployments/modules
  • [XD-1553] - syslog source is not capturing log info.
  • [XD-1554] - JMS source can only connect to localhost
  • [XD-1556] - Batch hashtag count throws exception when launched
  • [XD-1557] - Batch wordcount sample returns zero counts
  • [XD-1560] - Payload Conversion Sample throws exception.
  • [XD-1562] - Gauge & Rich Gauge fail to write results to redis for singlenode
  • [XD-1571] - JLine 1 is brought up (and shows in IDE) through ZK/curator
  • [XD-1577] - Changing externalized module config properties at runtime
  • [XD-1581] - XD config home should use XD_CONFIG_LOCATION if this is set
  • [XD-1582] - Temporary race condition between deployment and "runtime modules" command

Epic

  • [XD-235] - Modules (sinks, processors, sources) should be able to be easily tested inside the IDE using JUnit
  • [XD-464] - Deploy Spring XD on Hadoop YARN
  • [XD-996] - Support for executing batch functionality on and off Hadoop
  • [XD-1257] - Deploy XD nightly builds to EC2 and perform basic stream testing across singlenode, and distributed mode 1 server on redis, rabbitmq
  • [XD-1258] - Get basic project infrastructure based on new JS stack in place
  • [XD-1259] - Have supported distributions of Hadoop 'always on' in EC2.
  • [XD-1260] - Batch Job that uses partitioning and can poll FTP directory and transfer to HDFS directly via output streams
  • [XD-1267] - Improve configuration option handling
  • [XD-1317] - Support for deploying and running XD on YARN
  • [XD-1391] - Rewrite of the existing UI to use AngluarJS

Improvement

  • [XD-639] - Update error message for usage of hadoop rm with --recursive option
  • [XD-944] - Clean up unused JSON mapping classes
  • [XD-1047] - Allow Aggregate Counter to use timestamp field in data.
  • [XD-1084] - Create REST API for getting information on a given step execution
  • [XD-1087] - Create REST API for restarting a specific job instance
  • [XD-1090] - Create shell command for restarting a specific job instance
  • [XD-1093] - Create shell command for getting information on a given step execution
  • [XD-1185] - Add redisConnectionFactory with connection pool
  • [XD-1246] - To be able to run the tests without conflicting with an existing XD admin server/launcher
  • [XD-1247] - Remove RedisStreamDeploymentIntegrationTests and RabbitStreamDeploymentIntegrationTests
  • [XD-1248] - Merge AbstractStreamDeploymentIntegrationTests and AbstractSingleNodeStreamDeploymentIntegrationTests
  • [XD-1249] - AbstractShellIntegrationTests should start and stop server once.
  • [XD-1250] - Investigate RabbitSingleNodeStreamDeploymentIntegrationTests performance
  • [XD-1253] - RabbitMessageBus queue name prefix
  • [XD-1254] - Optimize AbstractSingleNodeStreamDeploymentIntegrationTests
  • [XD-1262] - Provide a clean way to get a reference to the MessageBus running in SingleNodeApplication
  • [XD-1273] - The use of labelled modules and taps needs more explanation
  • [XD-1358] - Remove launcher.xml and [transport]-launcher.xml configuration.
  • [XD-1363] - Tap XD batch Job output
  • [XD-1371] - Clarify API or syntax for managing deployment parameters
  • [XD-1424] - Docs could use link to Tuple artifacts
  • [XD-1428] - Log Hadoop Distro and ZK client connect info on Container startup
  • [XD-1429] - Create Shared Server Context
  • [XD-1438] - RabbitMQ port wrong in Docs
  • [XD-1486] - eliminate package tangle
  • [XD-1527] - Modify module metadata to exclude default module properties
  • [XD-1550] - Fix 'cannot find MessageBuilderFactory' warning
  • [XD-1551] - Fix Startup Messages
  • [XD-1552] - Remove --transport option except for single node
  • [XD-1565] - Document append support, else filepollhdfs writes empty file to hdfs
  • [XD-1566] - Document append configuration, else jdbchdfs writes empty file to hdfs

Story

  • [XD-157] - Verify use of JMX managed bean to shutdown cleanly the xd-admin and xd-container servers
  • [XD-158] - Expose shutdown operation over http
  • [XD-159] - Parameter parsing does not work if an argument contains '--'.
  • [XD-302] - User wants ability to create a mock source
  • [XD-304] - User wants ability to test processors
  • [XD-305] - User wants ability to test sinks
  • [XD-306] - User wants ability to test sources
  • [XD-307] - User to send a message directly to module and receive a message from a module
  • [XD-343] - Investigate JMX object naming of deployed modules and inbound/outbound channel adapters.
  • [XD-396] - Add section to documentation that shows command line options available for each server
  • [XD-433] - Homogenize Container Initialization Failures
  • [XD-467] - JMX shouldn't register taps or streams if the creation fails
  • [XD-535] - Create shell integration tests for trigger lifeycle
  • [XD-586] - Document queue channel capacity configurable when using local transport
  • [XD-613] - Deployed streams should be restarted on container start
  • [XD-647] - HTTP source should emit raw payload
  • [XD-726] - Test sink module in isolation
  • [XD-765] - Remove AutoLaunch feature from batch jobs
  • [XD-780] - Avoid use of module name twice in location when using a custom modules
  • [XD-786] - Add Warning-level log to postProcessAfterInitialization if Job name is not "job"
  • [XD-795] - Refactor mail and imap source into one "mail" module, leveraging Profiles
  • [XD-804] - Add Named Channel API
  • [XD-805] - Get notified when created named channel "is ready"
  • [XD-807] - Shell: Standardize counter name parameter
  • [XD-812] - Re-enable support for tapping labels and named channels
  • [XD-822] - TupleBuilder.fromString() should not overwrite original id and timestamp fields
  • [XD-840] - Fix Compiler Warnings
  • [XD-850] - JAR version mismatches
  • [XD-874] - For file based item reader jobs, step/job completion message should have name of file sent on named channel
  • [XD-879] - Cleanup hsqldb data directory used by tests after each test completion
  • [XD-885] - Add Batch Job Listeners Automatically
  • [XD-931] - Format option to display runtime module properties in shell
  • [XD-947] - Spike for Deployment SPI
  • [XD-954] - Stream definition completion REST layer + Shell adapter
  • [XD-1021] - Fix undeploy of stream with a composed module
  • [XD-1022] - Switch "module list" to horizontal display
  • [XD-1037] - Support ModuleOptions for composed modules
  • [XD-1050] - Improve Module Options support
  • [XD-1052] - Enforce consistent naming across CLI options, and command/template/operations method names
  • [XD-1060] - Add support for Hortonworks Data Platform 2.0
  • [XD-1068] - Remove existing 'purpose built' json processors and ensure all functionality is still available with #jsonPath based SpEL expression based processors
  • [XD-1076] - Create an FTP tasklet to get remote files and put them in the local file system.
  • [XD-1105] - Add some test coverage to mqtt modules
  • [XD-1114] - Investigate dropped Module Deployment Requests
  • [XD-1122] - Add jmxPort to list of coerced cmd line options
  • [XD-1127] - Add support for deploying a batch job with partitioning across multiple XD nodes.
  • [XD-1139] - Add TaskLet to Stream from (S)FTP to HDFS
  • [XD-1140] - Add (S)FTP Gateway and Batch Partitioner to List/Process Remote Files
  • [XD-1142] - Spike to model the cluster nodes.
  • [XD-1143] - Spike for matching algorithm for stream/job deployments and nodes
  • [XD-1146] - With Partitioned Jobs, Wire Partitioner and StepExecutionHandlers with the MessageBus
  • [XD-1156] - Refactor/Simplify JobPlugin
  • [XD-1167] - Mail Source ModuleOptions (+ profiles)
  • [XD-1182] - Update to spring-data-hadoop 2.0.0.M5
  • [XD-1190] - Setup precedence order for module properties' property resolver
  • [XD-1194] - CommandLine default values from container & admin options can not be overridden
  • [XD-1206] - Garbage in Job Repo Causes List Failure
  • [XD-1208] - Create Module options metadata for OOTB jobs
  • [XD-1209] - Add support for XD_CONFIG environment variable in windows shell scripts
  • [XD-1213] - Allow conditional validation for module options
  • [XD-1214] - Make job names unique across tests that use the same JobRepository
  • [XD-1215] - Distributed JobLocator should return a valid job
  • [XD-1220] - Batch jobs should use application.yml provided connection as default
  • [XD-1221] - XD modules should not have 'build' directory upon running gradle build
  • [XD-1222] - Duplicate MBean server definition by MBeanExportingPlugin
  • [XD-1223] - Unit/Integration tests need appropriate cleanup of test data during teardown
  • [XD-1224] - Add code coverage to gradle build
  • [XD-1225] - Integrate code coverage reports into the CI process
  • [XD-1226] - Review abstract base classes used in test cases to ensure proper resource cleanup.
  • [XD-1227] - Investigate long running tests and create refactoring issues
  • [XD-1228] - Provide a easy, prescriptive means to perform unit and basic stream integration tests.
  • [XD-1229] - Create build infrastructure for Angular based UI
  • [XD-1230] - Integrate grunt based UI build into the XD's gradle build
  • [XD-1231] - Investigate if we should use RequreJS with Angular
  • [XD-1232] - Create EC2 AMI for single-node install of Apache Hadoop 1.2.1
  • [XD-1233] - Create EC2 AMI for single-node install of Apache Hadoop 2.2.0
  • [XD-1234] - Create EC2 AMI for single-node install of Pivotal HD 1.1
  • [XD-1235] - Create EC2 AMI for single-node install of Cloudera CDH 4.5.0
  • [XD-1236] - Create EC2 AMI for single-node install of Hortonworks Data Platform 1.3
  • [XD-1237] - Add option to stop all running XD EC2 instances that match a given naming pattern
  • [XD-1238] - Add to Acceptance Test EC2 CI build plan a stage that stops any existing CI EC2 deployments
  • [XD-1239] - Add stage to Acceptance Test EC2 build plan that runs a basic acceptance test application against the single-node deployment
  • [XD-1240] - Add to Acceptance Test EC2 CI build plan a stage that uses XD distributed mode with rabbit
  • [XD-1241] - Add to Acceptance Test EC2 job a stage that uses XD distributed mode with redis
  • [XD-1242] - Optimize deployment of xd-admin and multiple nodes in spring-xd-ec2 to occur in parallel.
  • [XD-1243] - Create App for CI to Shutdown XD EC2 Cluster
  • [XD-1244] - Create CI Plan for XD EC2 deployment
  • [XD-1245] - Develop basic acceptance test application to exercise based XD-EC2 deployment from CI
  • [XD-1268] - Remove unused code related to 'accepted media type' in MessageBus
  • [XD-1271] - Investigate missing boot's actuator endpoints in XD
  • [XD-1272] - Remove job options that are handled at module level from shell
  • [XD-1274] - spring-integration-hadoop.xsd (and reactor) Imports the SI 3.0 Instead of SI 4.0 Schema
  • [XD-1276] - Out of the box batch jobs should add xdJobExecutionListener and xdStepExecutionListener
  • [XD-1278] - Rename avro sink to hdfs-dataset and add support for parquet format
  • [XD-1285] - Support shell completions for module names
  • [XD-1287] - Tap definitions should verify stream name
  • [XD-1288] - Remove references to XD-1050 in documentation
  • [XD-1291] - Handle container shutdown gracefully
  • [XD-1294] - Update spring-xd-extension-reactor dependency
  • [XD-1295] - Module message conversion fails to work if JMX is enabled
  • [XD-1296] - Few integration tests fail if JMX is enabled
  • [XD-1300] - Handling boolean type module option properties defaults in option metadata
  • [XD-1302] - Add documentation for using FTP->HDFS partitioned jobs
  • [XD-1303] - Create REST API for getting information on a given job instance
  • [XD-1304] - Create shell command for getting information on a given job instance
  • [XD-1305] - File Sink should support Replace as an option
  • [XD-1309] - JSR303 validation of options interferes with dsl completion
  • [XD-1312] - Job execution restart fails with NPE
  • [XD-1314] - Create XD .zip distribution for YARN
  • [XD-1315] - Create POJO based FileSink module metadata
  • [XD-1319] - Allow mixins of ModuleOptionsMetadata
  • [XD-1320] - Make Batch Job Restarts Work with Distributed Nodes
  • [XD-1321] - Add XD deployment for YARN
  • [XD-1322] - Add way to provide module config options for XD on YARN
  • [XD-1324] - Make Hadoop22 the default for the build
  • [XD-1325] - Deploy XD on YARN for a distribution other than Apache Hadoop 2.2
  • [XD-1328] - Modularize XD UI
  • [XD-1330] - Enhance HadoopFileSystemTestSupport to obtain resource for a specific hadoop distro
  • [XD-1331] - Make Batch Job Restarts Work using Single Node
  • [XD-1343] - Provide a conventional way to extend XD Container configuration
  • [XD-1345] - Use dot as the composed module option separator
  • [XD-1349] - Make hdfs configurable via application.yml
  • [XD-1350] - Remove enum for transport options
  • [XD-1351] - Replace BeanDefinitionAddingBeanPostProcessor with Ordered Plugins
  • [XD-1354] - Remove XDContainer and rename LauncherApplication
  • [XD-1357] - Container nodes should write attributes to ZooKeeper
  • [XD-1359] - Create an embedded ZooKeeper server process
  • [XD-1362] - Make Job notification channels subscribable
  • [XD-1364] - Upgrade to SHDP 2.0 M6
  • [XD-1367] - Exclude commons-logging from final distro1
  • [XD-1368] - Refactor container to remove shared module context as a separate context
  • [XD-1369] - Using hdfs sink throwing an error
  • [XD-1372] - Update Reactor integration to align 1.1 changes
  • [XD-1374] - Update to use spring-data-hadoop 2.0 M6
  • [XD-1375] - Create spike of web app that maps UI design docs to MVC components in Angluar
  • [XD-1376] - XD Shell crashes when the stream DSL has "!"
  • [XD-1378] - Change module placeholder names and remove context:property-placeholder usage
  • [XD-1379] - Avoid false negative test failures related to HSQLDB
  • [XD-1382] - tcp source requires a \r\n to suffix all inbound data
  • [XD-1383] - Add acceptance tests for stream with sources of TCP, HTTP, and Time and sinks of File and Log
  • [XD-1385] - Add JDBC Sink to acceptance tests
  • [XD-1386] - Add option to xd-admin and xd-container YARN scripts to allow copying ot HDFS and no execution.
  • [XD-1387] - Add documentation on how to deploy XD to YARN
  • [XD-1388] - Upgrade to Spring Batch 3.0.0 M3
  • [XD-1393] - Support explicit client connect string for ZooKeeper
  • [XD-1394] - Upgrade to Spring 4.0.2.RELEASE
  • [XD-1395] - Container and Single Node do not update their associated log
  • [XD-1396] - Container fails to start if JMX is enabled and manage_port is set
  • [XD-1397] - Admin leader should watch Container nodes in ZooKeeper
  • [XD-1398] - Admin servers should write streams to and delete them from ZooKeeper
  • [XD-1399] - Admin leader should watch ZooKeeper for Stream deployment requests
  • [XD-1400] - Containers should listen for Module deployment requests and their deletions
  • [XD-1401] - Add new reactor tcp module
  • [XD-1402] - Create a ‘throughput-sampler’ module for benchmarking
  • [XD-1403] - Create benchmarking application to demonstrate high performance message processing
  • [XD-1404] - Test against Spring Boot Snapshot build
  • [XD-1407] - Create a throughput sink
  • [XD-1409] - GemFire sink properties missmatch
  • [XD-1410] - XD EC2 needs to bootstrap ZOOKEEPER at installation time.
  • [XD-1411] - Create xd-yarn script
  • [XD-1412] - Composed options does not trigger profile activation
  • [XD-1413] - Create spring-xd-test-fixtures project
  • [XD-1414] - Create one xd-yarn shell script that encompases the functionality of seperate shell scripts
  • [XD-1417] - Create RPM for distribution
  • [XD-1418] - Create subproject spring-xd-machine-learning-analytics
  • [XD-1419] - Create subproject spring-xd-machine-learning-analytics-jpmml
  • [XD-1420] - Create a JPMML module that will evaluate a model.
  • [XD-1421] - Create documentation for the core analytical model abstractions and use of jpmml processor
  • [XD-1422] - Create a simple sample application for the jpmml module
  • [XD-1423] - Create documentation for how module properties are resolved.
  • [XD-1427] - Upgrade to ZooKeeper 3.4.6
  • [XD-1431] - Support multiple admin servers on a same host
  • [XD-1432] - Configure servers to use VanillaHealthEndpoint
  • [XD-1433] - Update to Spring Boot RC5
  • [XD-1434] - Improvements to Modules Tab
  • [XD-1435] - Improvements to Executions Tab
  • [XD-1436] - Misc cleanup in UI
  • [XD-1437] - Exclude slf4j Transitive Dependencies
  • [XD-1439] - Investigate module classloader leakage
  • [XD-1441] - Implement XD_MODULE_CONFIG_LOCATION & NAME
  • [XD-1443] - Update Spring Framework dependency to 4.0.3 GA
  • [XD-1444] - Update to Spring Boot 1.0 GA
  • [XD-1445] - Don't swallow unexpected exceptions in StacktraceFingerprintingCompletionRecoveryStrategy
  • [XD-1447] - Add documentation for a 'stdin' source module
  • [XD-1449] - Update to Spring Shell 1.1 RC1
  • [XD-1451] - Support CI build in Travis
  • [XD-1452] - http module leaks threads
  • [XD-1453] - Update Spring-AMQP to 1.3.1.RELEASE
  • [XD-1454] - User should not be required to specify a control channel
  • [XD-1458] - Upgrade to Spring Hadoop 2.0 RC2
  • [XD-1459] - Prevent accidental pickup of ENV var as module option
  • [XD-1464] - Upgrade to Spring Integration 4.0.0.M4
  • [XD-1465] - Update management context path to <root>/management
  • [XD-1467] - Do not eagerly use repositories in completion's *Strategy
  • [XD-1468] - Update to Reactor 1.1.0 M3
  • [XD-1469] - Error in correlation strategy in aggregator.xml
  • [XD-1470] - Rationalize inclusion of JSON jars in xd/lib and in modules
  • [XD-1481] - Remove spring-xd-analytics-ml-pmml project and module xml - to be put in seperate repository
  • [XD-1482] - Add initial support for DeploymentManifest
  • [XD-1483] - StreamListener should watch /xd/deployments/streams
  • [XD-1484] - JobListener should watch /xd/deployments/jobs
  • [XD-1485] - The Container's DeploymentListener should watch /xd/deployments/modules
  • [XD-1487] - Test scripts on windows that use XD_MODULE_CONFIG_LOCATION/NAME
  • [XD-1488] - Create a test case for insulating environment variables in module property lookup
  • [XD-1489] - Add documentation for new module property configuration support
  • [XD-1491] - Allow users to set attribute values on a container
  • [XD-1492] - Support groups container attribute
  • [XD-1498] - Create documentation for module property configuration
  • [XD-1499] - change xd-config.yml and xd-modules-config.yml to servers.yml and modules.yml
  • [XD-1506] - Add admin-ui to YARN zip packaging
  • [XD-1511] - Create documentation for ZK runtime
  • [XD-1513] - Change the default deploy option to false for stream/job deploy commands
  • [XD-1514] - Create documentation for the deployment manifest
  • [XD-1515] - Allow the value of xd.module.number to be used as property placeholder value
  • [XD-1521] - Create a dedicated plugin for ${xd.stream.name} and similar
  • [XD-1524] - Create small documentation section on jmx/monitoring functionalty
  • [XD-1528] - Support external datasource for single node application
  • [XD-1537] - Create documentation for Job listeners support
  • [XD-1538] - Update docs and samples now that deploy is false on job/stream creation
  • [XD-1540] - stack trace on xd-admin restart when redploying streams
  • [XD-1555] - transform processor with script option is broken
  • [XD-1558] - shell cp command fails
  • [XD-1559] - Payload Conversion will need to migrated to M6 as soon as M6 is available
  • [XD-1561] - When using transform --script=foo.groovy shell displays error
  • [XD-1567] - Update XD-Ec2 deployer to use XD_TRANSPORT
  • [XD-1568] - Update documentation related to transport and controlTransport
  • [XD-1569] - Rename reactor-tcp module to reactor-ip since it also supports udp
  • [XD-1572] - Unable to delete composed module
  • [XD-1573] - Update diagrams that show control transport
  • [XD-1574] - JDBC Acceptance tests must jdbc props vs. configProps setting.
  • [XD-1576] - Remove unused .properties files in config and update docs
  • [XD-1579] - Release 1.0.0.M6

Edit/Copy Release Notes

The text area below allows the project release notes to be edited and copied to another document.