├── book.json ├── README.md ├── rest-api └── restful_api_with_jax-rs.md ├── set_up_jberet └── job-repository-tables-diagram.png ├── .gitignore ├── SUMMARY.md ├── custom_cdi_scopes └── README.md ├── kafka └── kafkaitemreader_and_kafkaitemwriter.md ├── bean_validation_api_for_data_pojo └── README.md ├── batch_jsl_inheritance_and_composition └── README.md ├── programmatic_job_definition_with_java └── README.md ├── rest_item_reader_writer └── RestItemReaderWriter.md ├── jmsitemreader_and_jmsitemwriter └── README.md ├── xmlitemreader_and_xmlitemwriter └── README.md ├── mongodb_itemreader_and_itemwriter └── README.md ├── generate_reports_with_jasper_reports └── README.md ├── read_and_write_csv_data_with_super_csv └── README.md ├── hornetqitemreader_and_hornetqitemwriter └── README.md ├── read_and_write_csv_data └── README.md ├── read_and_write_data_through_jdbc └── README.md ├── LICENSE ├── beanio_itemreader_and_itemwriter └── README.md ├── batch_properties └── README.md ├── read_and_write_csv_data_with_jackson-dataformat-csv └── README.md ├── jsonitemreader_and_jsonitemwriter └── README.md ├── excel_itemreaders_and_itemwriters └── README.md └── develop_batch_artifacts_in_script_languages └── README.md /book.json: -------------------------------------------------------------------------------- 1 | {} -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | JBeret User Guide 2 | -------------------------------------------------------------------------------- /rest-api/restful_api_with_jax-rs.md: -------------------------------------------------------------------------------- 1 | # RESTful API with JAX-RS 2 | 3 | -------------------------------------------------------------------------------- /set_up_jberet/job-repository-tables-diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jberet/jberet-user-guide/HEAD/set_up_jberet/job-repository-tables-diagram.png -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Logs 2 | logs 3 | *.log 4 | 5 | # Runtime data 6 | pids 7 | *.pid 8 | *.seed 9 | 10 | # Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files) 11 | .grunt 12 | 13 | # Compiled binary addons (http://nodejs.org/api/addons.html) 14 | build/Release 15 | 16 | # Dependency directory 17 | # Deployed apps should consider commenting this line out: 18 | # see https://npmjs.org/doc/faq.html#Should-I-check-my-node_modules-folder-into-git 19 | node_modules 20 | 21 | _book/ 22 | 23 | -------------------------------------------------------------------------------- /SUMMARY.md: -------------------------------------------------------------------------------- 1 | # Summary 2 | 3 | * [Introduction](README.md) 4 | * [Set up JBeret](set_up_jberet/README.md) 5 | * [Batch JSL Inheritance and Composition](batch_jsl_inheritance_and_composition/README.md) 6 | * [Batch Properties](batch_properties/README.md) 7 | * [Programmatic Job Definition with Java](programmatic_job_definition_with_java/README.md) 8 | * [Custom CDI Scopes](custom_cdi_scopes/README.md) 9 | * [Develop Batch Artifacts in Script Languages](develop_batch_artifacts_in_script_languages/README.md) 10 | * [Bean Validation API for Data POJO](bean_validation_api_for_data_pojo/README.md) 11 | * [RESTful API with JAX-RS](rest-api/restful_api_with_jax-rs.md) 12 | * [Generate Reports with Jasper Reports](generate_reports_with_jasper_reports/README.md) 13 | * [Read and Write CSV Data with super-csv](read_and_write_csv_data/README.md) 14 | * [Read and Write CSV Data with jackson-dataformat-csv](read_and_write_csv_data_with_jackson-dataformat-csv/README.md) 15 | * [JsonItemReader and JsonItemWriter](jsonitemreader_and_jsonitemwriter/README.md) 16 | * [Read and Write Data through JDBC](read_and_write_data_through_jdbc/README.md) 17 | * [XmlItemReader and XmlItemWriter](xmlitemreader_and_xmlitemwriter/README.md) 18 | * [Excel ItemReaders and ItemWriters](excel_itemreaders_and_itemwriters/README.md) 19 | * [MongoDB ItemReader and ItemWriter](mongodb_itemreader_and_itemwriter/README.md) 20 | * [BeanIO ItemReader and ItemWriter](beanio_itemreader_and_itemwriter/README.md) 21 | * [JmsItemReader and JmsItemWriter](jmsitemreader_and_jmsitemwriter/README.md) 22 | * [HornetQItemReader and HornetQItemWriter](hornetqitemreader_and_hornetqitemwriter/README.md) 23 | * [RestItemReader and RestItemWriter](rest_item_reader_writer/RestItemReaderWriter.md) 24 | * [KafkaItemReader and KafkaItemWriter](kafka/kafkaitemreader_and_kafkaitemwriter.md) 25 | 26 | -------------------------------------------------------------------------------- /custom_cdi_scopes/README.md: -------------------------------------------------------------------------------- 1 | # Custom CDI Scopes 2 | 3 | JBeret supports 3 custom CDI scopes with the corresponding annotations: 4 | * `org.jberet.cdi.JobScoped` 5 | * `org.jberet.cdi.StepScoped` 6 | * `org.jberet.cdi.PartitionScoped` 7 | 8 | These annotations can be applied to CDI beans in a batch application to specify their scopes within the batch application. For instance, a `@JobScoped` singleton bean can be used to share and exchange application state between various artifacts within the same job execution. Or with a `@PartitionScoped` bean, different instances will be injected into targets in different partitions. 9 | 10 | Note that they should not be applied to any standard batch artifact types such as batchlet, item reader/writer/processor, batch listener, partition reducer/collector/analyzer/mapper, etc. 11 | 12 | ## `JobScoped` 13 | CDI beans with `@JobScoped` annotation are scoped to the current job execution. Injections of a bean type within the same job execution share the same bean instance. Injections of the same bean type across different job executions refers to different bean instances. `org.jberet.cdi.JobScoped` annotation is declared as follows: 14 | 15 | ```java 16 | @Target({TYPE}) 17 | @Retention(RUNTIME) 18 | @Documented 19 | @NormalScope 20 | @Inherited 21 | public @interface JobScoped {} 22 | ``` 23 | To apply the annotation to CDI bean at class-level: 24 | ```java 25 | @Named 26 | @JobScoped 27 | public class Foo {...} 28 | ``` 29 | To inject the above bean into a batch artifact: 30 | ```java 31 | @Named 32 | public class JobScopeBatchlet1 extends AbstractBatchlet { 33 | @Inject 34 | private Foo foo; 35 | ... 36 | } 37 | ``` 38 | 39 | ## `StepScoped` 40 | Injections of a bean type within the same step execution share the same bean instance. Injections of the same bean type across different step executions refers to different bean instances. `org.jberet.cdi.StepScoped` annotation is declared as follows: 41 | 42 | ```java 43 | @Target({TYPE}) 44 | @Retention(RUNTIME) 45 | @Documented 46 | @NormalScope 47 | @Inherited 48 | public @interface StepScoped {} 49 | ``` 50 | To apply the annotation to CDI bean at class-level: 51 | ```java 52 | @Named 53 | @StepScoped 54 | public class Foo {...} 55 | ``` 56 | To inject the above bean into a batch artifact: 57 | ```java 58 | @Named 59 | public class StepScopedListener implements StepListener { 60 | @Inject 61 | private Foo foo; 62 | ... 63 | } 64 | ``` 65 | 66 | ## `PartitionScoped` 67 | Injections of a bean type within the same partition execution share the same bean instance. Injections of the same bean type across different partition executions refers to different bean instances. `org.jberet.cdi.PartitionScoped` annotation is declared as follows: 68 | 69 | ```java 70 | @Target({TYPE}) 71 | @Retention(RUNTIME) 72 | @Documented 73 | @NormalScope 74 | @Inherited 75 | public @interface PartitionScoped {} 76 | ``` 77 | To apply the annotation to CDI bean at class-level: 78 | ```java 79 | @Named 80 | @PartitionScoped 81 | public class Foo {...} 82 | ``` 83 | To inject the above bean into a batch artifact: 84 | ```java 85 | @Named 86 | public class PartitionScopePartitionAnalyzer implements PartitionAnalyzer { 87 | @Inject 88 | private Foo foo; 89 | ... 90 | } 91 | ``` 92 | 93 | 94 | -------------------------------------------------------------------------------- /kafka/kafkaitemreader_and_kafkaitemwriter.md: -------------------------------------------------------------------------------- 1 | # KafkaItemReader and KafkaItemWriter 2 | 3 | Apache Kafka is a distributed publish-subscribe messaging system. jberet-support module includes `kafkaItemReader` and `kafkaItemWriter` to read from or write to Kafka topics. `kafkaItemReader` reads and binds data to instances of custom POJO bean provided by the batch application. 4 | 5 | `kafkaItemReader` keeps track of the current read position, including current topic name, topic partition number, and topic partition offset. Therefore, it is recommended to disable Kafka auto commit in Kafka consumer properties, e.g., `enable.auto.commit=false`. Kafka consumer properties are specified in batch property `configFile`. 6 | 7 | This reader class supports retry and restart, using the tracked read position as checkpoint info. It is also recommended to turn off Kafka consumer automatic group management; instead manually assign topics and partitions for the consumer. See batch property `topicPartitions`. 8 | 9 | The following dependencies are required for `kafkaItemReader` and `kafkaItemWriter`: 10 | 11 | ```xml 12 | 13 | org.apache.kafka 14 | kafka-clients 15 | 0.9.0.0 16 | 17 | ``` 18 | 19 | ## Batch Configuration Properties in Job XML 20 | 21 | `kafkaItemReader` and `kafkaItemWriter` are configured through `` or `` batch properties in job xml. All properties are of type String, unless noted otherwise. The following is an example job xml that references `kafkaItemReader` and `kafkaItemWriter`: 22 | 23 | ```xml 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | ... 33 | 34 | ``` 35 | 36 | ```xml 37 | 38 | ... 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | ``` 48 | 49 | ### Batch Properties for Both `kafkaItemReader` and `kafkaItemWriter` 50 | 51 | #### configFile 52 | 53 | The file path or URL to the Kafka configuration properties file. See Kafka docs for valid property keys and values. 54 | 55 | ### Batch Properties for `kafkaItemReader` 56 | 57 | #### topicPartitions 58 | 59 | `java.util.List` 60 | 61 | A list of topic-and-partition in the form of "topicName1:partitionNumber1, topicName2:partitionNumber2, ". For example, "orders:0, orders:1, returns:0, returns:1". 62 | 63 | #### pollTimeout 64 | 65 | `long` 66 | 67 | The time, in milliseconds, spent waiting in poll if data is not available. If `0`, returns immediately with any records that are available now. Must not be negative. 68 | 69 | 70 | ### Batch Properties for `kafkaItemWriter` 71 | 72 | #### topicPartition 73 | 74 | A topic partition in the form of `:`. For example, "orders:0". Unlike KafkaItemReader, which accepts multiple TopicPartition as source, this writer class only accepts 1 TopicPartition as destination. 75 | 76 | #### recordKey 77 | 78 | The key used when sending `ProducerRecord`. -------------------------------------------------------------------------------- /bean_validation_api_for_data_pojo/README.md: -------------------------------------------------------------------------------- 1 | # Bean Validation API for Data POJO 2 | Batch applications can leverage Bean Validation API to perform data validation, by embedding validation constraints in data POJO class. It is supported in all `ItemReader` implementation classes in jberet-support module. 3 | 4 | In each `ItemReader` class, if the configured `beanType` for the incoming data is a custom POJO (i.e., not `java.util.Map`, or `java.util.List`, for example, `Person`, `StockTrade`, `Company`), the reader class will invoke `javax.validation.Validator#validate` to perform the validation. 5 | 6 | Any validation constraint failure will cause the reader to throw `javax.validation.ConstraintViolationException`, a subclass of `java.lang.RuntimeException` and `javax.validation.ValidationException`. 7 | 8 | The following POJO class, `StockTrade`, demonstrates certain usage of Bean Validation constraints: 9 | ```java 10 | public class StockTrade implements Serializable { 11 | 12 | private static final long serialVersionUID = -55209987994863611L; 13 | 14 | @NotNull 15 | @Past 16 | Date date; 17 | 18 | @NotNull 19 | @Size(min = 3, max = 5) 20 | @Pattern(regexp = "^([0-9]|0[0-9]|1[0-9]|2[0-3]):[0-5][0-9]$") 21 | String time; 22 | 23 | @NotNull 24 | @Max(1000) 25 | @Min(1) 26 | double open; 27 | 28 | @Max(1000) 29 | @Min(1) 30 | double high; 31 | 32 | @Max(1000) 33 | @Min(1) 34 | double low; 35 | 36 | @Max(1000) 37 | @Min(1) 38 | double close; 39 | 40 | @NotNull 41 | @DecimalMin("100") 42 | @DecimalMax("9999999999") 43 | double volume; 44 | 45 | ... 46 | ``` 47 | 48 | ## Dependencies 49 | All `ItemReader` implementation classes in jberet-support module has a compile-time dependency on Bean Validation API: 50 | ```xml 51 | 52 | javax.validation 53 | validation-api 54 | ${version.javax.validation} 55 | 56 | ``` 57 | 58 | If a batch application triggers bean validation at runtime, then the following runtime dependencies are also required: 59 | ```xml 60 | 61 | org.hibernate 62 | hibernate-validator 63 | ${version.org.hibernate.validator} 64 | 65 | 66 | 67 | org.jboss.spec.javax.el 68 | jboss-el-api_3.0_spec 69 | ${version.org.jboss.spec.javax.el.jboss-el-api_3.0_spec} 70 | 71 | 72 | 73 | org.glassfish 74 | javax.el 75 | ${version.org.glassfish.javax.el} 76 | 77 | ``` 78 | Bean Validation may not be triggered when running a batch application, because 79 | * item reader `beanType` property is not a custom POJO type; 80 | * bean validation is disabled in job xml for a particular `ItemReader` class. 81 | 82 | ## Disable Bean Validation 83 | Bean validation is enabled by default for all `ItemReader` classes in jberet-support. It can be turned off by setting batch property `skipBeanValidation` to true on `reader` element in job.xml (its default value is false, which means bean validation is on): 84 | ```xml 85 | 86 | ``` 87 | 88 | Performing bean validation in your batch application has its performance overhead. So if processing speed and performance is high priority, consider disabling bean validation on `ItemReader` artifacts, or set item reader `beanType` property to a non-POJO type. 89 | 90 | -------------------------------------------------------------------------------- /batch_jsl_inheritance_and_composition/README.md: -------------------------------------------------------------------------------- 1 | # Batch JSL Inheritance and Composition 2 | 3 | In batch applications, it is a common practice to factor out common parts of job definition. It can be achieved with: 4 | 5 | * inheritance: inherit common parts from parent job xml; 6 | * composition: reference common parts as external entities. 7 | 8 | Batch job xml inheritance is not included in JSR 352 Batch Spec 1.0. JBeret implements JSL inheritance based on the draft [JSL Inheritance v1](https://java.net/projects/jbatch/downloads). Refer to that document for inheritance rules and restrictions. Note that this is an experimental feature and may undergo significant changes in future releases. 9 | 10 | ## JSL Inheritance Examples 11 | Here are some examples illustrating how to use JSL inheritance with JBeret. 12 | 13 | ### Inherit step and flow within the same job xml document 14 | Parent elements (step, flow, etc) are marked with attribute `abstract = "true"` to exclude them from direct execution. Child elements contains `parent` attribute, which points to the parent element. 15 | 16 |

inheritance.xml

17 | 18 | ```xml 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | ``` 36 | 37 | ### Inherit step from a different job xml document 38 | Child elements (step, job, etc) contain a `jsl-name` attribute, which specifies the job xml name (without `.xml` extension) containing the parent element. 39 | 40 |

chunk-child.xml

41 | 42 | ```xml 43 | 44 | 45 | 46 | 47 | ``` 48 | 49 |

chunk-parent.xml

50 | 51 | ```xml 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | ``` 81 | ## Resolve Custom XML External Entities 82 | Compared to inheritance, custom XML external entity offers a more direct, low-level means of JSL composition. For example, 83 | 84 |

job-with-xml-entities.xml

85 | 86 | ```xml 87 | 88 | 89 | 91 | ]> 92 | 93 | 94 | 95 | &job-segment; 96 | 97 | 98 | 99 | 100 | 101 | ``` 102 | 103 |

job-segment.xml

104 | 105 | ```xml 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | ``` 117 | 118 | The target file of the entity (`job-segment.xml` in the above example) should be accessible and loadable by JBeret batch runtime, and typically reside in the same location as the referencing job xml (`job-with-xml-entities.xml` in the above example). 119 | -------------------------------------------------------------------------------- /programmatic_job_definition_with_java/README.md: -------------------------------------------------------------------------------- 1 | # Programmatic Job Definition with Java 2 | 3 | In addition to defining batch jobs with job XML, one can also define a batch job programmatically using builder classes in `org.jberet.job.model` package: 4 | 5 | * `org.jberet.job.model.JobBuilder` 6 | * `org.jberet.job.model.StepBuilder` 7 | * `org.jberet.job.model.FlowBuilder` 8 | * `org.jberet.job.model.SplitBuilder` 9 | * `org.jberet.job.model.DecisionBuilder` 10 | 11 | ##`JobBuilder` 12 | 13 | Builder class for building a single job. After the job is built, the same `JobBuilder` instance should not be reused to build another job. 14 | 15 | Jobs built programmatically with this builder and other related builder classes do not support JSL inheritance. 16 | 17 | This class does not support multi-threaded access or modification. 18 | 19 | Usage example, 20 | ```java 21 | Job job = new JobBuilder(jobName) 22 | .restartable(false) 23 | .property("jobk1", "J") 24 | .property("jobk2", "J") 25 | .listener("jobListener1", new String[]{"jobListenerk1", "#{jobParameters['jobListenerPropVal']}"}, 26 | new String[]{"jobListenerk2", "#{jobParameters['jobListenerPropVal']}"}) 27 | 28 | .step(new StepBuilder(stepName) 29 | .properties(new String[]{"stepk1", "S"}, new String[]{"stepk2", "S"}) 30 | .batchlet(batchlet1Name, new String[]{"batchletk1", "B"}, new String[]{"batchletk2", "B"}) 31 | .listener("stepListener1", stepListenerProps) 32 | .stopOn("STOP").restartFrom(stepName).exitStatus() 33 | .endOn("END").exitStatus("new status for end") 34 | .failOn("FAIL").exitStatus() 35 | .nextOn("*").to(step2Name) 36 | .build()) 37 | 38 | .step(new StepBuilder(step2Name) 39 | .batchlet(batchlet1Name) 40 | .build()) 41 | 42 | .build(); 43 | ``` 44 | 45 | ##`StepBuilder` 46 | 47 | Builder class for building a single step. After the step is built, the same `StepBuilder` instance should not be reused to build another step. 48 | 49 | This class tries to model the `jsl:Step` element in job XML, while keeping a somewhat flattened structure. Methods for setting step sub-elements resides directly under `StepBuilder` where possible, avoiding the need for drilling down to sub-elements and popping back to parent context. For instance, 50 | 51 | * batchlet-specific method, `batchlet(String, java.util.Properties)` and `batchlet(String, java.util.Properties)` are directly in `StepBuilder`; 52 | 53 | * chunk-specific method, `reader(String, java.util.Properties)`, `processor(String, java.util.Properties)`, `writer(String, String[]...)`, etc are directly in `StepBuilder`, with no intermediary chunk builder; 54 | 55 | * partition-specific method, `partitionPlan(int, int, List)`, `partitionMapper(String, String[]...)`, `partitionReducer(String, java.util.Properties)`, `partitionCollector(String, String[]...)`, etc are directly in `StepBuilder`, with no intermediary partition builder. 56 | 57 | However, transition methods, such as `endOn(String)`, `stopOn(String)`, `failOn(String)`, and `nextOn(String)` will drill down to `Transition.End`, `Transition.Stop`, `Transition.Fail`, and `Transition.Next` respectively. These classes all contain a terminating method, which pops the context back to the current `StepBuilder`. 58 | 59 | This class does not support multi-threaded access or modification. 60 | 61 | Usage example, 62 | ```java 63 | Step step1 = new StepBuilder(step1Name).batchlet(batchlet1Name).build(); 64 | 65 | Step step2 = new StepBuilder(step2Name) 66 | .properties(new String[]{"stepk1", "S"}, new String[]{"stepk2", "S"}) 67 | .batchlet(batchlet1Name, new String[]{"batchletk1", "B"}, new String[]{"batchletk2", "B"}) 68 | .listener("stepListener1", stepListenerProps) 69 | .stopOn("STOP").restartFrom(step1Name).exitStatus() 70 | .endOn("END").exitStatus("new status for end") 71 | .failOn("FAIL").exitStatus() 72 | .nextOn("*").to(step3Name) 73 | .build()); 74 | 75 | Step step3 = new StepBuilder(step3Name) 76 | .reader("integerArrayReader", new String[]{"data.count", "30"}) 77 | .writer("integerArrayWriter", new String[]{"fail.on.values", "-1"}, new String[]{"writer.sleep.time", "0"}) 78 | .processor("integerProcessor") 79 | .checkpointPolicy("item") 80 | .listener("chunkListener1", new String[]{"stepExitStatus", stepExitStatusExpected}) 81 | .itemCount(10) 82 | .allowStartIfComplete() 83 | .startLimit(2) 84 | .skipLimit(8) 85 | .timeLimit(2, TimeUnit.MINUTES) 86 | .build()); 87 | 88 | Step step4 = new StepBuilder(stepName) 89 | .reader("integerArrayReader", new String[]{"data.count", "30"}, 90 | new String[]{"partition.start", "#{partitionPlan['partition.start']}"}, 91 | new String[]{"partition.end", "#{partitionPlan['partition.end']}"}) 92 | .writer("integerArrayWriter", new String[]{"fail.on.values", "-1"}, new String[]{"writer.sleep.time", "0"}) 93 | .partitionMapper("partitionMapper1", new String[]{"partitionCount", String.valueOf(partitionCount)}) 94 | .partitionCollector("partitionCollector1") 95 | .partitionAnalyzer("partitionAnalyzer1") 96 | .partitionReducer("partitionReducer1") 97 | .build()) 98 | ``` 99 | 100 | ##`DecisionBuilder` 101 | 102 | Builder class for building a single `Decision`. After the decision is built, the same `DecisionBuilder` instance should not be reused to build another decision. 103 | 104 | This class does not support multi-threaded access or modification. 105 | 106 | Usage example: 107 | ```java 108 | Decision decision = new DecisionBuilder("decision1", "decider1") 109 | .failOn("FAIL").exitStatus() 110 | .stopOn("STOP").restartFrom(stepName).exitStatus() 111 | .nextOn("NEXT").to(stepName) 112 | .endOn("*").exitStatus(stepName) 113 | .build() 114 | ``` 115 | 116 | ##`FlowBuilder` 117 | 118 | Builder class for building a single `Flow`. After the flow is built, the same `FlowBuilder` instance should not be reused to build another flow. 119 | 120 | This class does not support multi-threaded access or modification. 121 | 122 | Usage example: 123 | ```java 124 | Flow flow = new FlowBuilder(flowName) 125 | .step(new StepBuilder(stepName).batchlet(batchlet1Name) 126 | .next(step2Name) 127 | .build()) 128 | .step(new StepBuilder(step2Name).batchlet(batchlet1Name) 129 | .build()) 130 | .build()) 131 | ``` 132 | 133 | ##`SplitBuilder` 134 | 135 | Builder class for building a single `Split`. After the split is built, the same `SplitBuilder` instance should not be reused to build another split. 136 | 137 | This class does not support multi-threaded access or modification. 138 | 139 | Usage example: 140 | ```java 141 | Split split = new SplitBuilder(splitName) 142 | .flow(new FlowBuilder(flowName) 143 | .step(new StepBuilder(stepName).batchlet(batchlet1Name).build()) 144 | .build()) 145 | .flow(new FlowBuilder(flow2Name) 146 | .step(new StepBuilder(step2Name).batchlet(batchlet1Name).build()) 147 | .build()) 148 | .next(step3Name) 149 | .build()) 150 | ``` 151 | -------------------------------------------------------------------------------- /rest_item_reader_writer/RestItemReaderWriter.md: -------------------------------------------------------------------------------- 1 | # RestItemReader and RestItemWriter 2 | 3 | `jberet-support` module includes `restItemReader` and `restItemWriter` to support reading and writing batch data from and to a REST resource destination. `restItemReader` reads and binds data to instances of custom POJO bean provided by the batch application. 4 | 5 | `restItemReader` and `restItemWriter` leverages the standard JAX-RS API to interact with REST resources. The following dependencies are required: 6 | 7 | ```xml 8 | 9 | org.jboss.spec.javax.ws.rs 10 | jboss-jaxrs-api_2.0_spec 11 | provided 12 | 13 | ``` 14 | 15 | Depending on your JAX-RS implementation, for instance, Jersey or RESTEasy, you may need additional dependencies. For example, when using JBoss RESTEasy, you will typically need the following additional dependencies: 16 | 17 | ```xml 18 | 19 | com.fasterxml.jackson.core 20 | jackson-core 21 | 22 | 23 | 24 | com.fasterxml.jackson.core 25 | jackson-databind 26 | 27 | 28 | 29 | com.fasterxml.jackson.core 30 | jackson-annotations 31 | 32 | 33 | 34 | com.fasterxml.jackson.module 35 | jackson-module-jaxb-annotations 36 | 37 | 38 | ``` 39 | 40 | If your batch application is a web application or Java EE application, the target application server, e.g., WildFly or JBoss EAP, already provides all Java EE API classes. So there is no need to package JAX-RS API jar in your application archive, and its dependency scope can be `provided`. When deployed to WildFly or JBoss EAP 7, your application can declare the above jackson-related implementation dependencies by referencing corresponding JBoss modules hosted in the application server. For example, this can be done by adding `WEB-INF/jboss-deployment-structure` file: 41 | 42 | ```xml 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | ``` 54 | 55 | ## Batch Configuration Properties in Job XML 56 | 57 | `restItemReader` and `restItemWriter` are configured through `` or `` batch properties in job xml. All properties are of type String, unless noted otherwise. The following is an example job xml that references `restItemReader` and `restItemWriter`: 58 | 59 | ```xml 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | ... 76 | 77 | ``` 78 | 79 | ```xml 80 | 81 | ... 82 | 83 | 84 | 85 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | ``` 95 | 96 | ### Batch Properties for Both `RestItemReader` and `RestItemWriter` 97 | 98 | #### restUrl 99 | `java.net.URI` 100 | 101 | The base URI for the REST call. It usually points to a collection resource URI. For `RestItemReader`, data may be retrieved via HTTP GET or less commonly DELETE method. The URI may include additional query parameters other than `offset` (starting position to read) and `limit` (maximum number of items to return in each response). Query parameter `offset` and `limit` are specified by their own batch properties. 102 | 103 | For example, http://localhost:8080/restReader/api/movies 104 | 105 | For `RestItemWriter`, data may be submitted via HTTP POST or PUT method. The URI may include additional query parameters. 106 | 107 | For example, http://localhost:8080/restReader/api/movies?param1=value1 108 | 109 | This is a required batch property. 110 | 111 | #### httpMethod 112 | 113 | HTTP method to use in the REST call to read or write data. Its value should corresponds to the media types accepted by the target REST resource. 114 | 115 | For `RestItemReader`, valid values are GET and less commonly DELETE. If not specified, this property defaults to GET. 116 | 117 | For `RestItemWriter`, valid values are POST and PUT. If not specified, this property defaults to POST. 118 | 119 | ### Batch Properties for `RestItemReader` Only 120 | 121 | In addition to the above common properties, `RestItemReader` also supports the following batch properties in job xml: 122 | 123 | #### offsetKey 124 | 125 | Configures the key of the query parameter that specifies the starting position to read in the target REST resource. For example, some REST resource may require `start` instead of `offset` query parameter for the same purpose. 126 | 127 | This batch property is optional. If not set, the default key `offset` is used. 128 | 129 | #### offset 130 | 131 | The value of the offset property, which specifies the starting point for reading. If not specified, it defaults to `0`. 132 | 133 | #### limitKey 134 | 135 | Configures the key of the query parameter that specifies the maximum number of items to return in the REST response. For example, some REST resource may require `count` instead of `limit` query parameter for the same purpose. 136 | 137 | This batch property is optional. If not set, the default key `limit` is used. 138 | 139 | #### limit 140 | 141 | The value of the `limit` property, which specifies the maximum number of items to read. If not specified, it defaults to `10`. 142 | 143 | #### beanType 144 | 145 | The class of individual element of the response message entity. For example, 146 | * java.lang.String 147 | * org.jberet.samples.wildfly.common.Movie 148 | 149 | ### Batch Properties for `RestItemWriter` Only 150 | 151 | In addition to the above common properties, `restItemWriter` also supports the following batch properties in job xml: 152 | 153 | #### mediaType 154 | 155 | Media type to use in the REST call to write data. Its value should be valid for the target REST resource. If not specified, this property defaults to `application/json`. -------------------------------------------------------------------------------- /jmsitemreader_and_jmsitemwriter/README.md: -------------------------------------------------------------------------------- 1 | # JmsItemReader and JmsItemWriter 2 | 3 | jberet-support module includes `jmsItemReader` and `jmsItemWriter` to handle JMS messages. `jmsItemReader` reads and binds data to instances of custom POJO bean provided by the batch application. 4 | 5 | `jmsItemReader` and `jmsItemWriter` invokes standard JMS API (1.1 or later) for reading and writing JMS messages. Any compliant JMS implementation (e.g., HornetQ) can be used as the MQ provider. The following dependencies are required: 6 | 7 | ```xml 8 | 9 | org.jboss.spec.javax.jms 10 | jboss-jms-api_2.0_spec 11 | compile 12 | 13 | 14 | 15 | 16 | org.hornetq 17 | hornetq-core-client 18 | 19 | 20 | org.hornetq 21 | hornetq-commons 22 | 23 | 24 | io.netty 25 | netty-all 26 | 27 | 28 | 29 | 30 | org.hornetq 31 | hornetq-server 32 | 33 | 34 | org.hornetq 35 | hornetq-jms-server 36 | 37 | ``` 38 | 39 | ##Batch Configuration Properties in Job XML 40 | 41 | `jmsItemReader` and `jmsItemWriter` are configured through `` or `` batch properties in job xml, and CDI field injections into `org.jberet.support.io.JmsItemReader` and `org.jberet.support.io.JmsItemWriter`. All properties are of type `String`, unless noted otherwise. The following is an example job xml that references `jmsItemReader` and `jmsItemWriter`: 42 | 43 | ```xml 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | ``` 70 | 71 | ```xml 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | ``` 98 | 99 | ###Batch Properties for `jmsItemReader` and `jmsItemWriter` 100 | 101 | ####destinationLookupName 102 | JNDI lookup name for the JMS `Destination`. Optional property and defaults to `null`. When specified in job xml, it has higher precedence over `destinationInstance` injection 103 | 104 | ####connectionFactoryLookupName 105 | JNDI lookup name for the JMS `ConnectionFactory`. Optional property and defaults to `null`. When specified in job xml, it has higher precedence over `connectionFactoryInstance` injection. 106 | 107 | ####sessionMode 108 | The string name of the `sessionMode` used to create JMS session from a JMS connection. Optional property, and defaults to `null`. When not specified, JMS API `Connection.createSession()` is invoked to create the JMS session. When this property is specified, its value must be either `AUTO_ACKNOWLEDGE` or `DUPS_OK_ACKNOWLEDGE`. 109 | 110 | An example property in job xml: 111 | ```xml 112 | 113 | ``` 114 | See JMS API `Connection.createSession(int)` for more details. 115 | 116 | 117 | ###CDI Field Injections for `jmsItemReader` and `jmsItemWriter` 118 | ####destinationInstance 119 | `javax.enterprise.inject.Instance` 120 | 121 | This field holds an optional injection of `javax.jms.Destination`. When `destinationLookupName` property is specified in job xml, this field is ignored and `destinationLookupName` is used to look up JMS destination. The application may implement a `javax.enterprise.inject.Produces` method to satisfy this dependency injection. 122 | 123 | ####connectionFactoryInstance 124 | `javax.enterprise.inject.Instance` 125 | 126 | This field holds an optional injection of `javax.jms.ConnectionFactory`. When `connectionFactoryLookupName` property is specified in job xml, this field is ignored and `connectionFactoryLookupName` is used to look up JMS `ConnectionFactory`. The application may implement a `javax.enterprise.inject.Produces` method to satisfy this dependency injection. 127 | 128 | 129 | ###Batch Properties for `jmsItemReader` Only 130 | In addition to the above common properties, `jmsItemReader` also supports the following batch properties in job xml: 131 | 132 | ####skipBeanValidation 133 | `boolean` 134 | 135 | Indicates whether the current batch reader will invoke Bean Validation API to validate the incoming data POJO. Optional property and defaults to `false`, i.e., the reader will validate data POJO bean where appropriate. 136 | 137 | ####receiveTimeout 138 | `long` 139 | The number of milliseconds a JMS `MessageConsumer` blocks until a message arrives. Optional property, and defaults to `0`, which means it blocks indefinitely. 140 | 141 | ####messageSelector 142 | Only messages with properties matching the message selector expression are delivered. A value of `null` or an empty string indicates that there is no message selector for the message consumer. See JMS API `Session.createConsumer(javax.jms.Destination, java.lang.String)` 143 | 144 | ####beanType 145 | `java.lang.Class` 146 | 147 | The fully-qualified class name of the data item to be returned from `readItem()` method. Optional property and defaults to `null`. If it is specified, its valid value is: 148 | 149 | * `javax.jms.Message`: an incoming JMS message is returned as is. 150 | 151 | When this property is not specified, `readItem()` method returns an object whose actual type is determined by the incoming JMS message type. 152 | 153 | ###Batch Properties for `jmsItemWriter` Only 154 | None 155 | 156 | 157 | -------------------------------------------------------------------------------- /xmlitemreader_and_xmlitemwriter/README.md: -------------------------------------------------------------------------------- 1 | # XmlItemReader and XmlItemWriter 2 | 3 | jberet-support module includes `xmlItemReader` and `xmlItemWriter` to handle XML chunk-oriented data. `xmlItemReader` reads and binds data to instances of custom POJO bean provided by the batch application. 4 | 5 | `xmlItemReader` and `xmlItemWriter` leverages `jackson` and `jackson-dataformat-xml` libraries to handle XML data parsing and binding. The following `jackson` dependencies are required: 6 | 7 | ```xml 8 | 9 | com.fasterxml.jackson.core 10 | jackson-core 11 | ${version.com.fasterxml.jackson} 12 | 13 | 14 | com.fasterxml.jackson.core 15 | jackson-databind 16 | ${version.com.fasterxml.jackson} 17 | 18 | 19 | com.fasterxml.jackson.core 20 | jackson-annotations 21 | ${version.com.fasterxml.jackson} 22 | 23 | 24 | 25 | com.fasterxml.jackson.dataformat 26 | jackson-dataformat-xml 27 | ${version.com.fasterxml.jackson} 28 | 29 | 30 | 31 | 32 | 33 | com.fasterxml.jackson.module 34 | jackson-module-jaxb-annotations 35 | ${version.com.fasterxml.jackson} 36 | 37 | ``` 38 | 39 | ##Batch Configuration Properties in Job XML 40 | `xmlItemReader` and `xmlItemWriter` are configured through `` or `` batch properties in job xml. All properties are of type `String`, unless noted otherwise. The following is an example job xml that references `xmlItemReader` and `xmlItemWriter`: 41 | 42 | ```xml 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | ``` 66 | 67 | ###Batch Properties for Both `xmlItemReader` and `xmlItemWriter` 68 | 69 | ####resource 70 | 71 | The resource to read from (for batch readers), or write to (for batch writers). 72 | 73 | ####xmlFactoryLookup 74 | 75 | JNDI lookup name for `com.fasterxml.jackson.dataformat.xml.XmlFactory`, which is used for constructing `com.fasterxml.jackson.dataformat.xml.deser.FromXmlParser` in `xmlItemReader` and `com.fasterxml.jackson.dataformat.xml.ser.ToXmlGenerator` in `xmlItemWriter`. See class [org.jberet.support.io.XmlFactoryObjectFactory](http://docs.jboss.org/jberet/latest/javadoc/jberet-support/org/jberet/support/io/XmlFactoryObjectFactory.html) for more details. 76 | 77 | ####customDataTypeModules 78 | 79 | A comma-separated list of Jackson datatype module type ids that extend `com.fasterxml.jackson.databind.Module`. These modules will be registered with `xmlMapper`. For example, 80 | 81 | `com.fasterxml.jackson.datatype.joda.JodaModule, 82 | com.fasterxml.jackson.datatype.jsr353.JSR353Module, 83 | com.fasterxml.jackson.datatype.jsr310.JSR310Module, 84 | com.fasterxml.jackson.module.afterburner.AfterburnerModule` 85 | 86 | 87 | ###Batch Properties for `xmlItemReader` Only 88 | In addition to the above common properties, `xmlItemReader` also supports the following batch properties in job xml: 89 | 90 | ####skipBeanValidation 91 | `boolean` 92 | 93 | Indicates whether the current batch reader will invoke Bean Validation API to validate the incoming data POJO. Optional property and defaults to `false`, i.e., the reader will validate data POJO bean where appropriate. 94 | 95 | ####beanType 96 | `java.lang.Class` 97 | 98 | The bean class that represents individual data item in the `resource` XML, and the `readItem()` method reads one item at a time and binds it to the provided bean type. Required property. For example, 99 | 100 | * `org.jberet.support.io.StockTrade` 101 | * `org.jberet.support.io.Person` 102 | * `my.own.custom.ItemBean` 103 | 104 | 105 | ####start 106 | `int` 107 | 108 | Specifies the start position (a positive integer starting from `1`) to read the data. If reading from the beginning of the input XML, there is no need to specify this property. 109 | 110 | ####end 111 | `int` 112 | 113 | Specify the end position in the data set (inclusive). Optional property, and defaults to `Integer.MAX_VALUE`. If reading till the end of the input XML, there is no need to specify this property. 114 | 115 | ####inputDecorator 116 | `java.lang.Class` 117 | 118 | Fully-qualified class name of `com.fasterxml.jackson.core.io.InputDecorator`, which can be used to decorate input sources. Optional property, and defaults to `null`. See `com.fasterxml.jackson.core.io.InputDecorator`, and `com.fasterxml.jackson.core.JsonFactory#setInputDecorator(com.fasterxml.jackson.core.io.InputDecorator)` for more details. 119 | 120 | ####xmlTextElementName 121 | Alternate "virtual name" to use for XML CDATA segments; that is, text values. Optional property and defaults to `null` (empty String, "", is used as the virtual name). See `com.fasterxml.jackson.dataformat.xml.JacksonXmlModule#setXMLTextElementName(java.lang.String)` for more details. 122 | 123 | ###Batch Properties for `xmlItemWriter` Only 124 | In addition to the above common properties, `xmlItemWriter` also supports the following batch properties in job xml: 125 | 126 | ####writeMode 127 | Instructs this class, when the target XML resource already exists, whether to append to, or overwrite the existing resource, or fail. Valid values are `"append"`, `"overwrite"`, and `"failIfExists"`. Optional property, and defaults to append. 128 | 129 | ####defaultUseWrapper 130 | Whether use wrapper for indexed (List, array) properties or not, if there are no explicit annotations. Optional property, valid values are `"true"` and `"false"`, and defaults to `"true"`. See `com.fasterxml.jackson.dataformat.xml.JacksonXmlModule#setDefaultUseWrapper(boolean)`, and ` com.fasterxml.jackson.dataformat.xml.annotation.JacksonXmlElementWrapper` for more details. 131 | 132 | ####rootElementName 133 | Local name of the output XML root element. Required property. See `javax.xml.stream.XMLStreamWriter#writeStartElement` for more details. 134 | 135 | ####rootElementPrefix 136 | The prefix of the XML root element tag. Optional property and defaults to `null`. See `javax.xml.stream.XMLStreamWriter#writeStartElement(java.lang.String, java.lang.String, java.lang.String)` for more details. 137 | 138 | ####rootElementNamespaceURI 139 | The namespaceURI of the prefix to use. Optional property and defaults to `null`. See `javax.xml.stream.XMLStreamWriter#writeStartElement(java.lang.String, java.lang.String, java.lang.String)`, and `javax.xml.stream.XMLStreamWriter#writeStartElement(java.lang.String, java.lang.String)` for more details. 140 | 141 | ####prettyPrinter 142 | `java.lang.Class` 143 | 144 | The fully-qualified class name of `com.fasterxml.jackson.core.PrettyPrinter`, which implements pretty printer functionality, such as indentation. Optional property and defaults to `null` (default pretty printer is used). See `com.fasterxml.jackson.core.PrettyPrinter`, and `com.fasterxml.jackson.dataformat.xml.ser.ToXmlGenerator#setPrettyPrinter(com.fasterxml.jackson.core.PrettyPrinter)` for more details. 145 | 146 | ####outputDecorator 147 | `java.lang.Class` 148 | 149 | The fully-qualified class name of `com.fasterxml.jackson.core.io.OutputDecorator`, which can be used to decorate output destinations. Optional property and defaults to null. See `com.fasterxml.jackson.core.io.OutputDecorator`, and `com.fasterxml.jackson.core.JsonFactory#setOutputDecorator(com.fasterxml.jackson.core.io.OutputDecorator)` for more details. 150 | 151 | -------------------------------------------------------------------------------- /mongodb_itemreader_and_itemwriter/README.md: -------------------------------------------------------------------------------- 1 | # MongoDB ItemReader and ItemWriter 2 | 3 | jberet-support module includes `mongoItemReader` and `mongoItemWriter` for interacting with MongoDB NoSQL database. `mongoItemReader` reads data from MongoDB and binds to a custom POJO bean provided by the batch application. `mongoItemWriter` writes instances of the custom POJO bean type into MongoDB. 4 | 5 | The following dependencies are required for `mongoItemReader` and `mongoItemWriter`: 6 | 7 | ```xml 8 | 9 | org.mongodb 10 | mongo-java-driver 11 | ${version.org.mongodb} 12 | 13 | 14 | 15 | 16 | org.mongojack 17 | mongojack 18 | ${version.org.mongojack} 19 | 20 | 21 | javax.persistence 22 | persistence-api 23 | 24 | 25 | 26 | 27 | 28 | com.fasterxml.jackson.core 29 | jackson-core 30 | ${version.com.fasterxml.jackson} 31 | 32 | 33 | com.fasterxml.jackson.core 34 | jackson-databind 35 | ${version.com.fasterxml.jackson} 36 | 37 | 38 | com.fasterxml.jackson.core 39 | jackson-annotations 40 | ${version.com.fasterxml.jackson} 41 | 42 | 43 | 44 | 45 | 46 | com.fasterxml.jackson.module 47 | jackson-module-jaxb-annotations 48 | ${version.com.fasterxml.jackson} 49 | 50 | ``` 51 | 52 | ##Configure `mongoItemReader` and `mongoItemWriter` in Job XML 53 | `mongoItemReader` and `mongoItemWriter` are configured through `` or `` batch properties in job xml. All properties are of type `String`, unless noted otherwise. The following is an example job xml that references `mongoItemReader` and `mongoItemWriter`: 54 | 55 | ```xml 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | ``` 84 | 85 | ###Batch Properties for Both `mongoItemReader` and `mongoItemWriter` 86 | 87 | ####beanType 88 | `java.lang.Class` 89 | 90 | For `mongoItemReader`, it's the java type that each data item should be converted to; for `mongoItemWriter`, it's the java type for each incoming data item. Required property, and valid values are any data-representing bean class, for example, 91 | 92 | * `org.jberet.support.io.StockTrade` 93 | * `org.jberet.support.io.Person` 94 | * `my.own.custom.ItemBean` 95 | 96 | 97 | ####mongoClientLookup 98 | JNDI lookup name for `com.mongodb.MongoClient`. Optional property and defaults to `null`. When this property is specified, its value will be used to look up an instance of `com.mongodb.MongoClient`, which is typically created and administrated externally (e.g., inside application server). Otherwise, a new instance of `com.mongodb.MongoClient` will be created instead of lookup. See `org.jberet.support.io.MongoClientObjectFactory` and `com.mongodb.MongoClient` for more details. 99 | 100 | ####uri 101 | The Mongo client URI. See `com.mongodb.MongoClientURI` docs for syntax and details. Optional property. When this property is present, it should encompass all information necessary to establish a client connection. When this property is not present, other properties (e.g., `host`, `database`, `user`, `password`, `options`, and `collection`) should be specified to satisfy `com.mongodb.MongoClientURI` requirement. 102 | 103 | ####host 104 | Host and optional port information for creating `com.mongodb.MongoClientURI`. It can be single host and port, or multiple host and port specification in the format `host1[:port1][,host2[:port2],...[,hostN[:portN]]]`. See `com.mongodb.MongoClientURI`, `MongoClientObjectFactory.createMongoClientURI` for more details. 105 | 106 | ####database 107 | MongoDB database name, e.g., testData. Optional property and defaults to `null`. See `com.mongodb.MongoClientURI`, `MongoClientObjectFactory.createMongoClientURI` for more details. 108 | 109 | ####user 110 | MongoDB username. Optional property and defaults to `null`. 111 | 112 | ####password 113 | MongoDB password. Optional property and defaults to `null`. 114 | 115 | ####options 116 | MongoDB client options, e.g., `safe=true&wtimeout=1000`. Optional property and defaults to `null`. See `com.mongodb.MongoClientURI` for more details. 117 | 118 | ####collection 119 | MongoDB collection name, e.g., `movies`. Optional property and defaults to `null`. 120 | 121 | 122 | ###Batch Properties for `mongoItemReader` Only 123 | In addition to the above common properties, `mongoItemReader` also supports the following batch properties in job xml: 124 | 125 | ####skipBeanValidation 126 | `boolean` 127 | 128 | Indicates whether the current batch reader will invoke Bean Validation API to validate the incoming data POJO. Optional property and defaults to `false`, i.e., the reader will validate data POJO bean where appropriate. 129 | 130 | ####criteria 131 | Query criteria or conditions, which identify the documents that MongoDB returns to the client. Its value is a JSON string. Optional property and defaults to `null`. For example, 132 | 133 | `{ age: { $gt: 18 } }` 134 | 135 | ####projection 136 | Specifies the fields from the matching documents to return. Its value is a JSON string. Optional property and defaults to `null`. For example, 137 | 138 | `{ name: 1, address: 1}` 139 | 140 | ####limit 141 | `int` 142 | 143 | Modifies the query to limit the number of matching documents to return to the client. Optional property and defaults to `null` (limit is not set). 144 | 145 | ####batchSize 146 | `int` 147 | 148 | Limits the number of elements returned in one batch. A cursor typically fetches a batch of result objects and store them locally. If `batchSize` is positive, it represents the size of each batch of objects retrieved. It can be adjusted to optimize performance and limit data transfer. If `batchSize` is negative, it will limit of number objects returned, that fit within the max batch size limit (usually 4MB), and cursor will be closed. For example if batchSize is `-10`, then the server will return a maximum of `10` documents and as many as can fit in 4MB, then close the cursor. 149 | 150 | Note that this feature is different from `limit()` in that documents must fit within a maximum size, and it removes the need to send a request to close the cursor server-side. The batch size can be changed even after a cursor is iterated, in which case the setting will apply on the next batch retrieval. 151 | 152 | See `com.mongodb.DBCursor#batchSize(int)` for more details. 153 | 154 | ####sort 155 | Specifies how to sort the cursor's elements. Optional property and defaults to `null`. Its value is a JSON string, for example, 156 | 157 | `{ age: 1 }` 158 | 159 | ####skip 160 | `int` 161 | 162 | Specifies the number of elements to discard at the beginning of the cursor. Optional property and defaults to `0` (do not discard any elements). See `com.mongodb.DBCursor#skip(int)` for more details. 163 | 164 | ###Batch Properties for `mongoItemWriter` Only 165 | None 166 | -------------------------------------------------------------------------------- /generate_reports_with_jasper_reports/README.md: -------------------------------------------------------------------------------- 1 | # Generate Reports with Jasper Reports 2 | jberet-support module includes a batchlet that generates reports with Jasper Reports. Batch applications can reference this batchlet by name `jasperReportsBatchlet` to produce all types of reports supported by Jasper Reports. See [org.jberet.support.io.JasperReportsBatchlet javadoc](http://docs.jboss.org/jberet/) for more information. 3 | 4 | ##Dependencies 5 | In addition to the dependencies for batch applications (see Chapter Set up JBeret), applications need the following compile dependencies to use `jasperReportsBatchlet` 6 | 7 | ```xml 8 | 9 | net.sf.jasperreports 10 | jasperreports 11 | 12 | ``` 13 | 14 | ##Configure `jasperReportsBatchlet` in job xml 15 | The following is an example how to reference and configure `jasperReportsBatchlet` in job xml. Details on each batch property is explained in the next section. 16 | ```xml 17 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | ``` 37 | 38 | ##Batch Configuration Properties 39 | Configuration of `jasperReportsBatchlet` is done through either batch properties in job xml and `@BatchProperty` injections, or through CDI injections of objects created and configured by other parts of the application. Batch properties generally have higher precedence than CDI-injected counterparts. 40 | 41 | The Jasper Reports data source is configured through either `resource` property, or `jrDataSourceInstance` injection. Directly passing a `java.sql.Connection` instance to this class is not supported. For JDBC data source, The application should instead inject `net.sf.jasperreports.engine.JRResultSetDataSource` into `jrDataSourceInstance`. 42 | 43 | Report can be saved to a file, or directed to a `java.io.OutputStream`. 44 | 45 | This class supports all output report formats implemented in Jasper Reports. If `exporterInstance` field is injected with an instance of `net.sf.jasperreports.export.Exporter`, it will be used to export and generate the final report. Otherwise, if `outputType` batch property is set to `pdf`, `html`, or `jrprint`, a PDF, HTML or Jasper jrprint report is generated respectively. 46 | 47 | `jasperReportsBatchlet` can have the following batch properties in job xml. All properties are of type java.lang.String, unless otherwise noted. 48 | ####resource 49 | The resource that provides the data source for generating report. Optional property, and defaults to null. If specified, it may be a URL, a file path, or any resource path that can be loaded by the current application class loader. If this property is not specified, the application should inject appropriate `JRDataSource` into `jrDataSourceInstance`. If neither `resource` nor `jrDataSourceInstance` is specified, an `net.sf.jasperreports.engine.JREmptyDataSource` is used. 50 | 51 | `JRDataSource` injection allows for more flexible instantiation and configuration, such as setting `locale`, `datePattern`, `numberPattern`, `timeZone`, `recordDelimiter`, `useFirstRowAsHeader`, `columnNames`, `fieldDelimiter`, etc, before making the instance available to this class. 52 | 53 | This property has higher precedence than `jrDataSourceInstance` injection. 54 | 55 | ####recordDelimiter 56 | If `resource` is specified, and is a csv resource, this property specifies the delimiter between records, typically new line character (CR/LF). Optional property. See `net.sf.jasperreports.engine.data.JRCsvDataSource` for details. 57 | 58 | ####useFirstRowAsHeader 59 | If `resource` is specified, and is a csv, xls, or xlsx resource, this property specifies whether to use the first row as header. Optional property and valid values are "true" and "false". See `net.sf.jasperreports.engine.data.JRCsvDataSource` or `net.sf.jasperreports.engine.data.AbstractXlsDataSource` for details. 60 | 61 | ####fieldDelimiter 62 | If `resource` is specified, and is a CSV resource, this property specifies the field or column delimiter. Optional property. See `net.sf.jasperreports.engine.data.JRCsvDataSource` for details. 63 | 64 | ####columnNames 65 | `java.lang.String[]` 66 | 67 | If `resourc`e is specified, and is a csv, xls, or xlsx resource, this property specifies an array of strings representing column names matching field names in the report template. Optional property. See `net.sf.jasperreports.engine.data.JRCsvDataSource` or `net.sf.jasperreports.engine.data.AbstractXlsDataSourcefor` details. 68 | 69 | ####datePattern 70 | If `resource` is specified, this property specifies the date pattern string value. Optional property. See `net.sf.jasperreports.engine.data.JRAbstractTextDataSource#setDatePattern(java.lang.String)` for details. 71 | 72 | ####numberPattern 73 | If `resource` is specified, this property specifies the number pattern string value. Optional property. See `net.sf.jasperreports.engine.data.JRAbstractTextDataSource#setNumberPattern(java.lang.String)` for details. 74 | 75 | ####timeZone 76 | If `resource` is specified, this property specifies the time zone string value. Optional property. See `net.sf.jasperreports.engine.data.JRAbstractTextDataSource#setTimeZone(java.lang.String)` for details. 77 | 78 | ####locale 79 | If `resource` is specified, this property specifies the locale string value. Optional property. See `net.sf.jasperreports.engine.data.JRAbstractTextDataSource#setLocale(java.lang.String)` for details. 80 | 81 | ####charset 82 | If `resource` is specified, and is a csv resource, this property specifies the charset name for reading the csv resource. Optional property. See `net.sf.jasperreports.engine.data.JRCsvDataSource#JRCsvDataSource(java.io.File, java.lang.String)` for detail. 83 | 84 | ####template 85 | Resource path of the compiled Jasper Reports template (*.jasper file). Required property. It may be a URL, a file path, or any resource path that can be loaded by the current application class loader. 86 | 87 | ####outputType 88 | The format of report output. Optional property and defaults to "pdf". Valid values are: 89 | 90 | * pdf 91 | * html 92 | * jrprint 93 | 94 | If other formats are desired, the application should inject into `exporterInstance` with a properly configured instance of `net.sf.jasperreports.export.Exporter`. 95 | 96 | ####outputFile 97 | The file path of the generated report. Optional property and defaults to null. When this property is not specified, the application should inject a `java.io.OutputStream` into `outputStreamInstance`. For instance, in order to stream the report to servlet response, a `javax.servlet.ServletOutputStream` can be injected into `outputStreamInstance`. 98 | 99 | This property has higher precedence than `outputStreamInstance` injection. 100 | 101 | ####reportParameters 102 | `java.util.Map` 103 | 104 | Report parameters for generating the report. Optional property and defaults to null. This property can be used to specify string-based key-value pairs as report parameters. For more complex report parameters with object types, use injection into `reportParametersInstance`. 105 | 106 | This property has higher precedence than `reportParametersInstance` injection. 107 | 108 | ###Dependency Injection Fields 109 | In addition to the above batch properties configured in job.xml, some `jasperReportsBatchlet` configurations can be alternatively specified with CDI injection fields: 110 | 111 | ####outputStreamInstance 112 | `javax.enterprise.inject.Instance` 113 | 114 | Optional injection of report output stream, which allows for more control over the output stream creation, sharing, and configuration. The injected `OutputStream` is closed at the end of `process()` method. 115 | 116 | ####jrDataSourceInstance 117 | `javax.enterprise.inject.Instance` 118 | 119 | Optional injection of Jasper Reports `net.sf.jasperreports.engine.JRDataSource`, which allows for more control over the `JRDataSource` creation and configuration. 120 | 121 | ####reportParametersInstance 122 | `javax.enterprise.inject.Instance>` 123 | 124 | Optional injection of Jasper Reports report parameters, which allows for more complex, non-string values. 125 | 126 | ####exporterInstance 127 | `javax.enterprise.inject.Instance` 128 | 129 | Optional injection of an implementation of Jasper Reports `net.sf.jasperreports.export.Exporter`. The injected instance should have been properly configured, including appropriate `net.sf.jasperreports.export.ExporterOutput`. `net.sf.jasperreports.export.ExporterInput` will be set to a `net.sf.jasperreports.engine.JasperPrint` by this class. 130 | 131 | Some built-in implementations of `net.sf.jasperreports.export.ExporterOutput`: 132 | 133 | * `net.sf.jasperreports.engine.export.JRPdfExporter` 134 | * `net.sf.jasperreports.engine.export.HtmlExporter` 135 | * `net.sf.jasperreports.engine.export.ooxml.JRDocxExporter` 136 | * `net.sf.jasperreports.engine.export.ooxml.JRPptxExporter` 137 | * `net.sf.jasperreports.engine.export.JRXlsExporter` 138 | * `net.sf.jasperreports.engine.export.ooxml.JRXlsxExporter` 139 | * `net.sf.jasperreports.engine.export.JRTextExporter` 140 | * `net.sf.jasperreports.engine.export.JRRtfExporter` 141 | * `net.sf.jasperreports.engine.export.JRXmlExporter` 142 | * `net.sf.jasperreports.engine.export.JRCsvExporter` 143 | * `net.sf.jasperreports.engine.export.JsonExporter` 144 | * `net.sf.jasperreports.engine.export.oasis.JROdsExporter` 145 | * `net.sf.jasperreports.engine.export.oasis.JROdtExporter` 146 | 147 | 148 | 149 | 150 | 151 | 152 | -------------------------------------------------------------------------------- /read_and_write_csv_data_with_super_csv/README.md: -------------------------------------------------------------------------------- 1 | #Read and Write CSV Data with super-csv 2 | jberet-support module contains `csvItemReader` and `csvItemWriter` that reads and writes CSV (Comma-Separated Values) resources respectively. Batch applications can reference them by name `csvItemReader` and `csvItemWriter` in job xml. They can also be configured to handle data with other delimiters such as tab, vertical bar, etc. For fixed-length flat file, see Chapter BeanIO ItemReader and ItemWriter. 3 | 4 | The following dependency is required by `csvItemReader` and `csvItemWriter`: 5 | 6 | ```xml 7 | 8 | net.sf.supercsv 9 | super-csv 10 | ${version.net.sf.supercsv} 11 | 12 | ``` 13 | 14 | jberet-support delegates most of the CSV data read, write and processing to [supercsv](http://supercsv.sourceforge.net/), and therefore the configuration of `csvItemReader` and `csvItemWriter` mirrors that of supercsv. 15 | 16 | Besides `csvItemReader` and `csvItemWriter`, JBeret also offers other options for dealing with CSV data: 17 | * write batch reader, processor or writer in script languages, which may have built-in support or libraries for CSV data format. For more details, refer to chapter Develop Batch Artifacts in Script Languages. 18 | * use `beanIOItemReader` and `beanIOItemWriter`, which handles common data formats such as CSV, XML, JSON. See Chapter BeanIO ItemReader and ItemWriter for details. 19 | 20 | ##Configure `csvItemReader` and `csvItemWriter` in job xml 21 | The following is a sample job xml that references `csvItemReader` and `csvItemWriter` to read and write CSV data. Each batch property will be explained in the next section. Javadoc of [CsvItemReader](http://docs.jboss.org/jberet/latest/javadoc/jberet-support/org/jberet/support/io/CsvItemReader.html) and [CsvItemWriter](http://docs.jboss.org/jberet/latest/javadoc/jberet-support/org/jberet/support/io/BeanIOItemWriter.html) also contains details for each batch configuration property. 22 | ```xml 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 42 | 43 | 44 | 45 | 46 | 47 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | ``` 64 | 65 | ##Batch Configuration Properties for Both `csvItemReader` and `csvItemWriter` 66 | 67 | ###resource 68 | The resource to read from (for batch readers), or write to (for batch writers). 69 | 70 | ###nameMapping 71 | `java.lang.String[]` 72 | 73 | Specify the bean fields or map keys corresponding to CSV columns in the same order. Not used if `beanType` property is set to `java.util.List`. If the CSV column names exactly match bean fields or map keys, then no need to specify this property. If the CSV column names are missing or differ from bean fields or map keys, then this property is required. An example of `nameMapping` value: 74 | 75 | `"number, gender, title, givenName, middleInitial, surname"` 76 | 77 | ###beanType 78 | `java.lang.Class` 79 | 80 | Specifies a fully-qualified class or interface name that maps to a row of the source CSV file. For example, 81 | 82 | * `java.util.List` 83 | * `java.util.Map` 84 | * `org.jberet.support.io.Person` 85 | * `my.own.BeanType` 86 | 87 | ###preference 88 | Specifies one of the 4 predefined CSV preferences: 89 | 90 | * `STANDARD_PREFERENCE` 91 | * `EXCEL_PREFERENCE` 92 | * `EXCEL_NORTH_EUROPE_PREFERENCE` 93 | * `TAB_PREFERENCE` 94 | 95 | ###quoteChar 96 | The quote character (used when a cell contains special characters, such as the delimiter char, a quote char, or spans multiple lines). See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). The default quoteChar is double quote (`"`). If " is present in the CSV data cells, specify quoteChar to some other characters, e.g., `|`. 97 | 98 | ###delimiterChar 99 | The delimiter character (separates each cell in a row). See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). 100 | 101 | ###endOfLineSymbols 102 | The end of line symbols to use when writing (Windows, Mac and Linux style line breaks are all supported when reading, so this preference won't be used at all for reading). See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). See CSV Preferences.). 103 | 104 | ###surroundingSpacesNeedQuotes 105 | Whether spaces surrounding a cell need quotes in order to be preserved (see below). The default value is false (quotes aren't required). See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). The default value is false (quotes aren't required). See CSV Preferences.). 106 | 107 | ###commentMatcher 108 | Specifies a CommentMatcher for reading CSV resource. The CommentMatcher determines whether a line should be considered a comment. See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). For example, 109 | 110 | * `"startsWith #"` 111 | * `"matches 'regexp'"` 112 | * `"my.own.CommentMatcherImpl"` 113 | 114 | ###encoder 115 | Specifies a custom encoder when writing CSV. For example, 116 | * `default` 117 | * `select 1, 2, 3` 118 | * `select true, true, false` 119 | * `column 1, 2, 3` 120 | * `column true, true, false` 121 | * `my.own.MyCsvEncoder` 122 | 123 | See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). 124 | 125 | ###quoteMode 126 | Allows you to enable surrounding quotes for writing (if a column wouldn't normally be quoted because it doesn't contain special characters). For example, 127 | 128 | * `default` 129 | * `always` 130 | * `select 1, 2, 3` 131 | * `select true, true, false` 132 | * `column 1, 2, 3` 133 | * `column true, true, false` 134 | * `my.own.MyQuoteMode` 135 | 136 | See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). 137 | 138 | ###cellProcessors 139 | Specifies a list of cell processors, one for each column. See [Super CSV docs](http://supercsv.sourceforge.net/cell_processors.html) for supported cell processor types. The rules and syntax are as follows: 140 | 141 | * The size of the resultant list must equal to the number of CSV columns. 142 | * Cell processors appear in the same order as CSV columns. 143 | * If no cell processor is needed for a column, enter `null`. 144 | * Each column may have `null`, `1`, `2`, or multiple cell processors, separated by comma (`,`) 145 | * Cell processors for different columns must be separated with semi-colon (`;`). 146 | * Cell processors may contain parameters enclosed in parenthesis, and multiple parameters are separated with comma (`,`). 147 | string literals in cell processor parameters must be enclosed within single quotes, e.g., `'xxx'` 148 | 149 | For example, to specify cell processors for 5-column CSV: 150 | ```xml 151 | 158 | ``` 159 | 160 | ###charset 161 | The name of the character set to be used for reading and writing data, e.g., UTF-8. This property is optional, and if not set, the platform default charset is used. 162 | 163 | 164 | ##Batch Configuration Properties for `csvItemReader` Only 165 | In addition to the common properties listed above, `csvItemReader` also supports the following batch properties: 166 | 167 | ###skipBeanValidation 168 | `boolean` 169 | 170 | Indicates whether the current batch reader will invoke Bean Validation API to validate the incoming data POJO. Optional property and defaults to false, i.e., the reader will validate data POJO bean where appropriate. 171 | 172 | ###start 173 | `int` 174 | 175 | Specifies the start position (a positive integer starting from `1`) to read the data. If reading from the beginning of the input CSV, there is no need to specify this property. 176 | 177 | ###end 178 | `int` 179 | 180 | Specify the end position in the data set (inclusive). Optional property, and defaults to `Integer.MAX_VALUE`. If reading till the end of the input CSV, there is no need to specify this property. 181 | 182 | ###headerless 183 | `boolean` 184 | 185 | Indicates that the input CSV resource does not contain header row. Optional property, valid values are `true` or `false`, and the default is `false`. 186 | 187 | ##Batch Configuration Properties for `csvItemWriter` Only 188 | In addition to the common properties listed above, `csvItemWriter` also supports the following batch properties: 189 | 190 | ###header 191 | `java.lang.String[]` 192 | 193 | Specifies the CSV header row to write out. 194 | 195 | ###writeComments 196 | 197 | Specifies the complete comment line that can be recognized by any tools or programs intended to read the current CSV output. The comments should already include the required comment-defining characters or regular expressions. The value of this property will be written out as a comment line verbatim as the first line. 198 | 199 | ###writeMode 200 | Instructs `csvItemWriter`, when the target CSV resource already exists, whether to append to, or overwrite the existing resource, or fail. Valid values are: 201 | * `append` (default) 202 | * `overwrite` 203 | * `failIfExists` 204 | 205 | 206 | -------------------------------------------------------------------------------- /hornetqitemreader_and_hornetqitemwriter/README.md: -------------------------------------------------------------------------------- 1 | # HornetQItemReader and HornetQItemWriter 2 | 3 | jberet-support module includes `hornetQItemReader` and `hornetQItemWriter` for reading messages from and writing messages to HornetQ server. They can be used by batch applications that wish to directly work with HornetQ API instead of the standard JMS API. 4 | 5 | `hornetQItemReader` handles the following types of HornetQ messages: 6 | * If `beanType` batch property is set to `org.hornetq.api.core.client.ClientMessage`, the incoming message is immediately returned as is from `readItem()`. 7 | * If the message type is `org.hornetq.api.core.client.ClientMessage#TEXT_TYPE`, the string text is returned from `readItem()`. 8 | * Otherwise, a byte array is retrieved from the message body buffer, deserialize to an object, and returned from `readItem()`. 9 | 10 | This reader ends when either of the following occurs: 11 | 12 | * `receiveTimeout` (in milliseconds) has elapsed when trying to receive a message from the destination; 13 | * the size of the incoming message body is 0; 14 | 15 | `hornetQItemWriter` can send the following HornetQ message types: 16 | 17 | * If the data item is of type `java.lang.String`, a `org.hornetq.api.core.client.ClientMessage#TEXT_TYPE` message is created with the text content in the data item, and sent. 18 | * Else if the data is of type `org.hornetq.api.core.client.ClientMessage`, it is sent as is. 19 | * Else an `org.hornetq.api.core.client.ClientMessage#OBJECT_TYPE` message is created with the data item object, and sent. 20 | 21 | `durableMessage` property can be configured to send either durable or non-durable (default) messages. 22 | 23 | ##Dependencies for `hornetQItemReader` and `hornetQItemWriter` 24 | ```xml 25 | 26 | 27 | org.hornetq 28 | hornetq-core-client 29 | 30 | 31 | org.hornetq 32 | hornetq-commons 33 | 34 | 35 | io.netty 36 | netty-all 37 | 38 | 39 | 40 | 41 | org.hornetq 42 | hornetq-server 43 | 44 | ``` 45 | 46 | ##Batch Configuration Properties in Job XML 47 | 48 | `hornetQItemReader` and `hornetQItemWriter` are configured through `` or `` batch properties in job xml, and CDI field injections into `org.jberet.support.io.HornetQItemReader` and `org.jberet.support.io.HornetQItemWriter`. All properties are of type `String`, unless noted otherwise. The following is an example job xml that references `hornetQItemReader` and `hornetQItemWriter`: 49 | 50 | ```xml 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | 99 | 100 | 101 | ``` 102 | 103 | ###Batch Properties for Both `hornetQItemReader` and `hornetQItemWriter` 104 | ####connectorFactoryParams 105 | `java.util.Map` 106 | 107 | Key-value pairs to identify and configure HornetQ `org.hornetq.api.core.TransportConfiguration`, which is used to create HornetQ `ServerLocator`. Optional property and defaults to `null`. When this property is present, it will be used to create HornetQ `ServerLocator`, and the injection fields `serverLocatorInstance` and `sessionFactoryInstance` will be ignored. Valid keys and values are: 108 | 109 | * `factory-class`, the fully-qualified class name of a HornetQ connector factory, required if this property is present; 110 | * any param keys and values appropriate for the above-named HornetQ connector factory class 111 | 112 | An example of this property in job xml: 113 | 114 | ```xml 115 | 116 | ``` 117 | 118 | ####serverLocatorParams 119 | `java.util.Map` 120 | 121 | Key-value pairs to configure HornetQ `ServerLocator`. Optional property and defaults to `null`. Valid keys are: 122 | 123 | * HA: `true` or `false` (default), `true` if the `ServerLocator` receives topology updates from the cluster 124 | * Properties in `ServerLocator` class that have corresponding setter method, starting with either upper or lower case character 125 | 126 | See the current version of HornetQ `ServerLocator` javadoc for supported keys and values, e.g., [ServerLocator 2.4 Javadoc](http://docs.jboss.org/hornetq/2.4.0.Final/docs/api/hornetq-client/org/hornetq/api/core/client/ServerLocator.html) 127 | 128 | An example of this property in job xml: 129 | 130 | ```xml 131 | 132 | ``` 133 | 134 | ####queueParams 135 | `java.util.Map` 136 | 137 | Key-value pairs to identify and configure the target HornetQ queue. Required property. The following keys are supported: 138 | 139 | * `address`, required 140 | * `durable`, optional 141 | * `filter`, optional 142 | * `name`, optional 143 | * `shared`, optional 144 | * `temporary`, optional 145 | 146 | An example of `queueParams` property in job xml: 147 | 148 | ```xml 149 | 150 | ``` 151 | 152 | ####sendAcknowledgementHandler 153 | `java.lang.Class` 154 | 155 | The fully-qualified name of a class that implements `org.hornetq.api.core.client.SendAcknowledgementHandler`. A `SendAcknowledgementHandler` notifies a client when an message sent asynchronously has been received by the server. See current version of HornetQ documentation for details, e.g., [SendAcknowledgementHandler 2.4 Javadoc](https://docs.jboss.org/hornetq/2.4.0.Final/docs/api/hornetq-client/org/hornetq/api/core/client/SendAcknowledgementHandler.html) 156 | 157 | An example `sendAcknowledgementHandler` property in job xml: 158 | 159 | ```xml 160 | 161 | ``` 162 | 163 | ###CDI Field Injections for Both `hornetQItemReader` and `hornetQItemWriter` 164 | 165 | ####serverLocatorInstance 166 | `javax.enterprise.inject.Instance` 167 | 168 | This field holds an optional injection of HornetQ `ServerLocator`. When `connectorFactoryParams` is not specified, and `sessionFactoryInstance` is not satisfied, this field will be queried to obtain an instance of HornetQ `ServerLocator`. The application may implement a `javax.enterprise.inject.Produces` method to satisfy this dependency injection. 169 | 170 | ####sessionFactoryInstance 171 | `javax.enterprise.inject.Instance` 172 | 173 | This field holds an optional injection of HornetQ `ClientSessionFactory`. If this injection is satisfied, `serverLocatorInstance` will be ignored. The application may implement a `javax.enterprise.inject.Produces` method to satisfy this dependency injection. 174 | 175 | 176 | ###Batch Properties for `hornetQItemReader` Only 177 | In addition to the above common properties, `hornetQItemReader` also supports the following batch properties in job xml: 178 | 179 | ####skipBeanValidation 180 | `boolean` 181 | 182 | Indicates whether the current batch reader will invoke Bean Validation API to validate the incoming data POJO. Optional property and defaults to `false`, i.e., the reader will validate data POJO bean where appropriate. 183 | 184 | ####receiveTimeout 185 | `long` 186 | 187 | The number of milliseconds a HornetQ `ClientConsumer` blocks until a message arrives. Optional property, and defaults to `0`, which means it blocks indefinitely. 188 | 189 | ####beanType 190 | `java.lang.Class` 191 | 192 | The fully-qualified class name of the data item to be returned from `readItem()` method. Optional property and defaults to `null`. If it is specified, its valid value is: 193 | 194 | * `org.hornetq.api.core.client.ClientMessage`: an incoming HornetQ message is returned as is. 195 | 196 | When this property is not specified, `readItem()` method returns an object whose actual type is determined by the incoming HornetQ message type. 197 | 198 | 199 | ###Batch Properties for `hornetQItemWriter` Only 200 | In addition to the above common properties, `hornetQItemWriter` also supports the following batch properties in job xml: 201 | 202 | ####durableMessage 203 | `boolean` 204 | 205 | Whether the message to be produced is durable or not. Optional property and defaults to `false`. Valid values are `true` and `false`. 206 | 207 | 208 | -------------------------------------------------------------------------------- /read_and_write_csv_data/README.md: -------------------------------------------------------------------------------- 1 | #Read and Write CSV Data with super-csv 2 | jberet-support module contains `csvItemReader` and `csvItemWriter` that reads and writes CSV (Comma-Separated Values) resources respectively. Batch applications can reference them by name `csvItemReader` and `csvItemWriter` in job xml. They can also be configured to handle data with other delimiters such as tab, vertical bar, etc. For fixed-length flat file, see Chapter BeanIO ItemReader and ItemWriter. 3 | 4 | The following dependency is required by `csvItemReader` and `csvItemWriter`: 5 | 6 | ```xml 7 | 8 | net.sf.supercsv 9 | super-csv 10 | ${version.net.sf.supercsv} 11 | 12 | ``` 13 | 14 | jberet-support delegates most of the CSV data read, write and processing to [supercsv](http://supercsv.sourceforge.net/), and therefore the configuration of `csvItemReader` and `csvItemWriter` mirrors that of supercsv. 15 | 16 | Besides `csvItemReader` and `csvItemWriter`, JBeret also offers other options for dealing with CSV data: 17 | * `jacksonCsvItemReader` and `JacksonCsvItemWriter` 18 | * write batch reader, processor or writer in script languages, which may have built-in support or libraries for CSV data format. For more details, refer to chapter Develop Batch Artifacts in Script Languages. 19 | * use `beanIOItemReader` and `beanIOItemWriter`, which handles common data formats such as CSV, XML, JSON. See Chapter BeanIO ItemReader and ItemWriter for details. 20 | 21 | ##Configure `csvItemReader` and `csvItemWriter` in job xml 22 | The following is a sample job xml that references `csvItemReader` and `csvItemWriter` to read and write CSV data. Each batch property will be explained in the next section. Javadoc of [CsvItemReader](http://docs.jboss.org/jberet/latest/javadoc/jberet-support/org/jberet/support/io/CsvItemReader.html) and [CsvItemWriter](http://docs.jboss.org/jberet/latest/javadoc/jberet-support/org/jberet/support/io/BeanIOItemWriter.html) also contains details for each batch configuration property. 23 | ```xml 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 43 | 44 | 45 | 46 | 47 | 48 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | ``` 65 | 66 | ##Batch Configuration Properties for Both `csvItemReader` and `csvItemWriter` 67 | 68 | ###resource 69 | The resource to read from (for batch readers), or write to (for batch writers). 70 | 71 | ###nameMapping 72 | `java.lang.String[]` 73 | 74 | Specify the bean fields or map keys corresponding to CSV columns in the same order. Not used if `beanType` property is set to `java.util.List`. If the CSV column names exactly match bean fields or map keys, then no need to specify this property. If the CSV column names are missing or differ from bean fields or map keys, then this property is required. An example of `nameMapping` value: 75 | 76 | `"number, gender, title, givenName, middleInitial, surname"` 77 | 78 | ###beanType 79 | `java.lang.Class` 80 | 81 | Specifies a fully-qualified class or interface name that maps to a row of the source CSV file. For example, 82 | 83 | * `java.util.List` 84 | * `java.util.Map` 85 | * `org.jberet.support.io.Person` 86 | * `my.own.BeanType` 87 | 88 | ###preference 89 | Specifies one of the 4 predefined CSV preferences: 90 | 91 | * `STANDARD_PREFERENCE` 92 | * `EXCEL_PREFERENCE` 93 | * `EXCEL_NORTH_EUROPE_PREFERENCE` 94 | * `TAB_PREFERENCE` 95 | 96 | ###quoteChar 97 | The quote character (used when a cell contains special characters, such as the delimiter char, a quote char, or spans multiple lines). See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). The default quoteChar is double quote (`"`). If " is present in the CSV data cells, specify quoteChar to some other characters, e.g., `|`. 98 | 99 | ###delimiterChar 100 | The delimiter character (separates each cell in a row). See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). 101 | 102 | ###endOfLineSymbols 103 | The end of line symbols to use when writing (Windows, Mac and Linux style line breaks are all supported when reading, so this preference won't be used at all for reading). See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). See CSV Preferences.). 104 | 105 | ###surroundingSpacesNeedQuotes 106 | Whether spaces surrounding a cell need quotes in order to be preserved (see below). The default value is false (quotes aren't required). See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). The default value is false (quotes aren't required). See CSV Preferences.). 107 | 108 | ###commentMatcher 109 | Specifies a CommentMatcher for reading CSV resource. The CommentMatcher determines whether a line should be considered a comment. See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). For example, 110 | 111 | * `"startsWith #"` 112 | * `"matches 'regexp'"` 113 | * `"my.own.CommentMatcherImpl"` 114 | 115 | ###encoder 116 | Specifies a custom encoder when writing CSV. For example, 117 | * `default` 118 | * `select 1, 2, 3` 119 | * `select true, true, false` 120 | * `column 1, 2, 3` 121 | * `column true, true, false` 122 | * `my.own.MyCsvEncoder` 123 | 124 | See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). 125 | 126 | ###quoteMode 127 | Allows you to enable surrounding quotes for writing (if a column wouldn't normally be quoted because it doesn't contain special characters). For example, 128 | 129 | * `default` 130 | * `always` 131 | * `select 1, 2, 3` 132 | * `select true, true, false` 133 | * `column 1, 2, 3` 134 | * `column true, true, false` 135 | * `my.own.MyQuoteMode` 136 | 137 | See [CSV Preferences](http://supercsv.sourceforge.net/preferences.html). 138 | 139 | ###cellProcessors 140 | Specifies a list of cell processors, one for each column. See [Super CSV docs](http://supercsv.sourceforge.net/cell_processors.html) for supported cell processor types. The rules and syntax are as follows: 141 | 142 | * The size of the resultant list must equal to the number of CSV columns. 143 | * Cell processors appear in the same order as CSV columns. 144 | * If no cell processor is needed for a column, enter `null`. 145 | * Each column may have `null`, `1`, `2`, or multiple cell processors, separated by comma (`,`) 146 | * Cell processors for different columns must be separated with semi-colon (`;`). 147 | * Cell processors may contain parameters enclosed in parenthesis, and multiple parameters are separated with comma (`,`). 148 | string literals in cell processor parameters must be enclosed within single quotes, e.g., `'xxx'` 149 | 150 | For example, to specify cell processors for 5-column CSV: 151 | ```xml 152 | 159 | ``` 160 | 161 | ###charset 162 | The name of the character set to be used for reading and writing data, e.g., UTF-8. This property is optional, and if not set, the platform default charset is used. 163 | 164 | 165 | ##Batch Configuration Properties for `csvItemReader` Only 166 | In addition to the common properties listed above, `csvItemReader` also supports the following batch properties: 167 | 168 | ###skipBeanValidation 169 | `boolean` 170 | 171 | Indicates whether the current batch reader will invoke Bean Validation API to validate the incoming data POJO. Optional property and defaults to false, i.e., the reader will validate data POJO bean where appropriate. 172 | 173 | ###start 174 | `int` 175 | 176 | Specifies the start position (a positive integer starting from `1`) to read the data. If reading from the beginning of the input CSV, there is no need to specify this property. 177 | 178 | ###end 179 | `int` 180 | 181 | Specify the end position in the data set (inclusive). Optional property, and defaults to `Integer.MAX_VALUE`. If reading till the end of the input CSV, there is no need to specify this property. 182 | 183 | ###headerless 184 | `boolean` 185 | 186 | Indicates that the input CSV resource does not contain header row. Optional property, valid values are `true` or `false`, and the default is `false`. 187 | 188 | ##Batch Configuration Properties for `csvItemWriter` Only 189 | In addition to the common properties listed above, `csvItemWriter` also supports the following batch properties: 190 | 191 | ###header 192 | `java.lang.String[]` 193 | 194 | Specifies the CSV header row to write out. 195 | 196 | ###writeComments 197 | 198 | Specifies the complete comment line that can be recognized by any tools or programs intended to read the current CSV output. The comments should already include the required comment-defining characters or regular expressions. The value of this property will be written out as a comment line verbatim as the first line. 199 | 200 | ###writeMode 201 | Instructs `csvItemWriter`, when the target CSV resource already exists, whether to append to, or overwrite the existing resource, or fail. Valid values are: 202 | * `append` (default) 203 | * `overwrite` 204 | * `failIfExists` 205 | 206 | 207 | -------------------------------------------------------------------------------- /read_and_write_data_through_jdbc/README.md: -------------------------------------------------------------------------------- 1 | # Read and Write Data through JDBC 2 | jberet-support module contains `jdbcItemReader` and `jdbcItemWriter` that reads from and write to database through JDBC. Batch applications can reference them by name `jdbcItemReader` and `jdbcItemWriter` in job xml. 3 | 4 | `jdbcItemReader` can read a row of data into one of three types, configured through `beanType` batch property: 5 | * `java.util.List`: populated with column value in the order as returned by the query result set 6 | * `java.util.Map`: with column name as the key 7 | * any custom POJO, e.g., `StockTrade`, `Person`, `Employee`, etc 8 | 9 | Likewise, `jdbcItemWriter` can obtain data from one of three types for database insertion, configured through `beanType` batch property: 10 | * `java.util.List`: used to populate the insert statement 11 | * `java.util.Map`: used to set parameter values of the insert statement with map key as the parameter name 12 | * any custom POJO, e.g., `StockTrade`, `Person`, `Employee`, etc 13 | 14 | When `beanType` is configured to a custom POJO bean in `jdbcItemReader` and `jdbcItemWriter`, they use jackson-databind library to perform transformation between `java.util.Map` and POJO bean. So in this case, the following jackson dependencies are needed at runtime: 15 | 16 | ```xml 17 | 18 | 19 | com.fasterxml.jackson.core 20 | jackson-core 21 | ${version.com.fasterxml.jackson} 22 | 23 | 24 | com.fasterxml.jackson.core 25 | jackson-databind 26 | ${version.com.fasterxml.jackson} 27 | 28 | 29 | com.fasterxml.jackson.core 30 | jackson-annotations 31 | ${version.com.fasterxml.jackson} 32 | 33 | 34 | 35 | 36 | 37 | com.fasterxml.jackson.module 38 | jackson-module-jaxb-annotations 39 | ${version.com.fasterxml.jackson} 40 | 41 | ``` 42 | 43 | Of course all applications should include any database-specific JDBC driver jars in the runtime classpath. 44 | 45 | ##Configure `jdbcItemReader` and `jdbcItemWriter` in job xml 46 | The following sample job xml demonstrates how to reference and configure `jdbcItemReader` and `jdbcItemWriter` in job xml. Each batch property will be explained in the next section. 47 | 48 | ```xml 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | ... 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | ... 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | ``` 86 | 87 | ##Batch Configuration Properties for Both `jdbcItemReader` and `jdbcItemWriter` 88 | ###sql 89 | The sql statement for reading data from database, or inserting data into database. It should include parameter markers that will be filled in with real data by the current batch `ItemReader` or `ItemWriter`. 90 | 91 | ###beanType 92 | `java.lang.Class` 93 | 94 | For `ItemReader`, it's the java type that each data item should be converted to; for `ItemWriter`, it's the java type for each incoming data item. In either case, the valid values are: 95 | 96 | * java.util.Map 97 | * java.util.List 98 | * a custom java type that represents data item; 99 | 100 | ###dataSourceLookup 101 | JNDI lookup name of the `javax.sql.DataSource`. Optional property, and defaults to null. If specified, it will be used to look up the target DataSource, and other database connection batch properties for this writer class will be ignored. 102 | 103 | ###url 104 | JDBC connection url 105 | 106 | ###user 107 | User name for the JDBC connection 108 | 109 | ###password 110 | Password for the JDBC connection 111 | 112 | ###properties 113 | `java.util.Map` 114 | 115 | Additional properties for the JDBC connection 116 | 117 | 118 | ##Batch Configuration Properties for `jdbcItemReader` Only 119 | In addition to the common batch properties listed above, `jdbcItemReader` may also be configured through the following batch properties: 120 | 121 | ###autoCommit 122 | `boolean` 123 | 124 | Auto-commit mode for the JDBC connection. 125 | 126 | ###skipBeanValidation 127 | `boolean` 128 | 129 | Indicates whether the current batch reader will invoke Bean Validation API to validate the incoming data POJO. Optional property and defaults to false, i.e., the reader will validate data POJO bean where appropriate. 130 | 131 | ###start 132 | `int` 133 | 134 | The row number in the `ResultSet` to start reading. It's a positive integer starting from 1. 135 | 136 | ###end 137 | `int` 138 | 139 | The row number in the `ResultSet` to end reading (inclusive). It's a positive integer starting from 1. 140 | 141 | ###columnMapping 142 | `java.lang.String[]` 143 | 144 | String keys used in target data structure for database columns. Optional property, and if not specified, it defaults to column labels from `java.sql.ResultSetMetaData`. This property should have the same length and order as `columnTypes`, if the latter is specified. 145 | 146 | For example, if `sql` is 147 | 148 | SELECT NAME, ADDRESS, AGE FROM PERSON 149 | 150 | And you want to map the data to the following form: 151 | 152 | {"fn" = "Jon", "addr" = "1 Main st", "age" = 30} 153 | 154 | then `columnMapping` should be specified as follows in job xml: 155 | 156 | "fn, addr, age" 157 | 158 | ###columnTypes 159 | `java.lang.String[]` 160 | 161 | Tells this class which `java.sql.ResultSet` getter method to call to get `ResultSet` field value. It should have the same length and order as `columnMapping`. Optional property, and if not set, this class calls `ResultSet.getObject(java.lang.String)` for all columns. For example, this property can be configured as follows in job xml: 162 | 163 | "String, String, Int" 164 | 165 | And this class will call `ResultSet.getString(java.lang.String)`, `ResultSet.getString(java.lang.String)`, and `ResultSet.getInt(java.lang.String)`. 166 | 167 | ###resultSetProperties 168 | `java.lang.String[]` 169 | 170 | The following `resultSetProperties` can be optionally configured in job xml: 171 | 172 | * `fetchSize` (use driver default) 173 | * `fetchDirection` 174 | + `FETCH_FORWARD` (default) 175 | + `FETCH_REVERSE` 176 | + `FETCH_UNKNOWN` 177 | * `resultSetType`: 178 | + `TYPE_FORWARD_ONLY` (default) 179 | + `TYPE_SCROLL_INSENSITIVE` 180 | + `TYPE_SCROLL_SENSITIVE` 181 | * `resultSetConcurrency`: 182 | + `CONCUR_READ_ONLY` (default) 183 | + `CONCUR_UPDATABLE` 184 | * `resultSetHoldability`: 185 | + `HOLD_CURSORS_OVER_COMMIT` (default) 186 | + `CLOSE_CURSORS_AT_COMMIT` 187 | 188 | See [`java.sql.ResultSet` javadoc](http://docs.oracle.com/javase/7/docs/api/java/sql/ResultSet.html) for detailed explanation of these properties. For example: 189 | 190 | 192 | 193 | ##Batch Configuration Properties for `jdbcItemWriter` Only 194 | In addition to the common batch properties listed above, `jdbcItemWriter` may also be configured through the following batch properties: 195 | 196 | ###parameterNames 197 | `java.lang.String[]` 198 | 199 | String keys used to retrieve values from incoming data and apply to SQL insert statement parameters. It should have the same length and order as SQL insert statement parameters. Optional property and if not set, it is initialized from the columns part of SQL insert statement. This property is not used when `beanType` is `java.util.List`, which assumes that incoming data is already in the same order as SQL parameters. 200 | 201 | If `beanType` is `java.util.Map`, and any of its key is different than the target table column names, `parameterNames` should be specified. For example, if an incoming data item is: 202 | 203 | {"name" = "Jon", "address" = "1 Main st", "age" = 30} 204 | 205 | And `sql` is 206 | 207 | INSERT INTO PERSON(NAME, ADDRESS, AGE) VALUES(?, ?, ?) 208 | 209 | then `parameterNames` should be specified as follows in job xml: 210 | 211 | "name, address, age" 212 | 213 | If `beanType` is custom bean type, custom mapping may be achieved with either `parameterNames`, or in bean class with annotations, e.g., JAXB or Jackson annotations. If the bean class does not contain field mapping, or the field mapping is intended for other part of the application (e.g., `ItemReader`), `parameterNames` can be used to customize mapping. 214 | 215 | ###parameterTypes 216 | `java.lang.String[]` 217 | 218 | Tells this class which `PreparedStatement` setter method to call to set insert statement parameters. It should have the same length and order as SQL insert statement parameters. Optional property, and if not set, this class calls `PreparedStatement.setObject(int, Object)` for all parameters. For example, this property can be configured as follows in job xml: 219 | 220 | "String, String, Int" 221 | 222 | And this class will call `PreparedStatement.setString(int, String)`, `PreparedStatement.setString(int, String)`, and `PreparedStatement.setInt(int, int)`. 223 | 224 | ##Batch Configuration Properties for jackson-databind library 225 | When `jdbcItemReader` or `jdbcItemWriter` uses custom POJO bean as `beanType`, `jackson-databind` performs data transformation behind the scene, and this step can be configured through the following batch configuration properties in job xml, though the defaults should suffice in most cases: 226 | 227 | * `jsonFactoryFeatures` 228 | * `mapperFeatures` 229 | * `jsonFactoryLookup` 230 | * `serializationFeatures` 231 | * `customSerializers` 232 | * `deserializationFeatures` 233 | * `customDeserializers` 234 | * `customDataTypeModules` 235 | 236 | See Chapter JsonItemReader and JsonItemWriter for more details. 237 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use the 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | 177 | END OF TERMS AND CONDITIONS 178 | 179 | APPENDIX: How to apply the Apache License to your work. 180 | 181 | To apply the Apache License to your work, attach the following 182 | boilerplate notice, with the fields enclosed by brackets "[]" 183 | replaced with your own identifying information. (Don't include 184 | the brackets!) The text should be enclosed in the appropriate 185 | comment syntax for the file format. We also recommend that a 186 | file or class name and description of purpose be included on the 187 | same "printed page" as the copyright notice for easier 188 | identification within third-party archives. 189 | 190 | Copyright [yyyy] [name of copyright owner] 191 | 192 | Licensed under the Apache License, Version 2.0 (the "License"); 193 | you may not use this file except in compliance with the License. 194 | You may obtain a copy of the License at 195 | 196 | http://www.apache.org/licenses/LICENSE-2.0 197 | 198 | Unless required by applicable law or agreed to in writing, software 199 | distributed under the License is distributed on an "AS IS" BASIS, 200 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 201 | See the License for the specific language governing permissions and 202 | limitations under the License. 203 | -------------------------------------------------------------------------------- /beanio_itemreader_and_itemwriter/README.md: -------------------------------------------------------------------------------- 1 | # BeanIO ItemReader and ItemWriter 2 | 3 | jberet-support module includes `beanIOItemReader` and `beanIOItemWriter` that employs [BeanIO](http://beanio.org/) to read and write data in various formats that are supported by BeanIO, e.g., fixed length file, CSV file, XML, etc. They also support restart, ranged reading, custom error handler, and dynamic BeanIO mapping properties. 4 | 5 | The following BeanIO dependencies are required for `beanIOItemReader` and `beanIOItemWriter`: 6 | 7 | ```xml 8 | 9 | org.beanio 10 | beanio 11 | ${version.org.beanio} 12 | compile 13 | 14 | 15 | org.ow2.asm 16 | asm 17 | ${version.org.ow2.asm} 18 | 19 | ``` 20 | 21 | ##Batch Configuration Properties in Job XML 22 | 23 | `beanIOItemReader` and `beanIOItemWriter` are configured through `` or `` batch properties in job xml. All properties are of type `String`, unless noted otherwise. The following is an example job xml that references `beanIOItemReader` and `beanIOItemWriter`: 24 | 25 | ```xml 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | ``` 56 | The following is a sample fixed length flat file (downloaded from http://star.cde.ca.gov/star2013/research_fixfileformat.aspx), and its BeanIO mapping file, referenced in job xml as `streamMapping` property: 57 | 58 | ``` 59 | 000000000000000000201304State of California 60 | 010000000000000000201305Alameda 61 | 011001700000000000201306Alameda Alameda County Office of Education 62 | 011001701098350728201309Alameda FAME Public Charter FAME Public Charter 94560 63 | 011001701126070811201309Alameda Envision Academy for Arts & T Envision Academy for Arts & T 94612 64 | 011001701184891049201309Alameda Aspire California College Prep Aspire California College Prep 94606 65 | ``` 66 | ```xml 67 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | ``` 86 | 87 | Another sample BeanIO mapping file for person data file with "|" as the delimiter: 88 | ``` 89 | Number|Gender|Title|GivenName|MiddleInitial|Surname|StreetAddress|City|State|ZipCode|Country|CountryFull|EmailAddress|Username|Password|TelephoneNumber|MothersMaiden|Birthday|CCType|CCNumber|CVV2|CCExpires|NationalID|UPS|Color|Occupation|Company|Vehicle|Domain|BloodType|Pounds|Kilograms|FeetInches|Centimeters|GUID|Latitude|Longitude 90 | 1|female|Mrs.|Susannah|R|Pemberton|1585 Court Street|Maryland Heights|MO|63141|US|United States|SusannahRPemberton@rhyta.com|Turs1956|eiGufib3th|636-898-6411|Andrews|11/5/1956|MasterCard|5112913274177177|156|9/2015|490-10-6886|1Z 4Y0 424 93 3839 606 7|Purple|Sampler|Forth & Towne|2002 MCC Smart|RentHoldings.com|O+|218.7|99.4|5' 1"|156|c738a629-5108-47b1-a168-8ab6d1438e50|38.728008|-90.37838 91 | 2|male|Mr.|Anthony|J|Quirion|4928 Carson Street|San Diego|CA|92103|US|United States|AnthonyJQuirion@rhyta.com|Quoinep|ew2Keewoh|858-856-8403|Ferguson|7/5/1979|MasterCard|5533216847446447|542|1/2015|622-80-6651|1Z W28 086 52 9104 825 3|Orange|Shampooer|JumboSports|1993 Mazda 121|PoliticalPlanners.com|B+|140.8|64.0|5' 11"|180|79b9a58b-ada9-4fab-8093-d0f17c541cf1|32.831002|-117.113794 92 | 3|male|Mr.|Shaun|P|Schwartz|1246 Kyle Street|Grand Island|NE|68801|US|United States|ShaunPSchwartz@fleckens.hu|Sivend88|saeKooca8r|308-615-8773|Guevara|10/23/1988|MasterCard|5562020815274730|454|10/2019|507-90-6312|1Z 96V V28 41 7598 869 9|White|Lease operator|Grass Roots Yard Services|2000 BMW 528|StationCard.com|O+|191.4|87.0|5' 10"|179|ebaa641c-0344-45aa-9649-b5ae134e9e61|40.897922|-98.182374 93 | 4|male|Mr.|Michael|B|Petrey|3875 Jehovah Drive|Spotsylvania|VA|22553|US|United States|MichaelBPetrey@superrito.com|Thibust|ooho3Ocoh|540-507-6775|Weaver|12/26/1972|MasterCard|5420696909140416|637|5/2016|230-80-7928|1Z 8V1 528 75 9778 835 3|Red|Telecommunications specialist|Great American Music|1995 Ford Ranger|EmploymentCars.com|B+|165.9|75.4|6' 0"|182|dbc2c6d6-d967-4908-be39-2f4b22ba2c17|38.118885|-77.706684 94 | ``` 95 | ```xml 96 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | ``` 147 | 148 | ###Batch Properties for Both `beanIOItemReader` and `beanIOItemWriter` 149 | ####resource 150 | The resource to read from (for batch readers), or write to (for batch writers). Some reader or writer implementations may choose to ignore this property and instead use other properties that are more appropritate. 151 | 152 | ####streamName 153 | Name of the BeanIO stream defined in BeanIO mapping file. It corresponds to the batch job property. 154 | 155 | ####streamMapping 156 | Location of the BeanIO mapping file, which can be a file path, a URL, or a resource loadable by the current class loader. 157 | 158 | ####streamFactoryLookup 159 | JNDI name for looking up `org.beanio.StreamFactory` when running in application server. When `streamFactoryLookup` property is specified in job xml and hence injected into batch reader or writer class, `org.beanio.StreamFactory` will be looked up with JNDI, and `streamMapping` and `mappingProperties` will be ignored. 160 | 161 | ####mappingProperties 162 | `java.util.Map` 163 | 164 | User properties that can be used for property substitution in BeanIO mapping file. When used with batch job JSL properties, they provide dynamic BeanIO mapping attributes. For example, 165 | 166 | 1. In batch job client class, set the properties values as comma-separated key-value pairs: 167 | ```java 168 | params.setProperty("mappingProperties", "zipCodeFieldName=zipCode, zipCodeFieldType=string"); 169 | ``` 170 | 171 | 2. In job xml file, make the properties available for `beanIOItemReader` or `beanIOItemWriter` via `@BatchProperty` injection: 172 | ```xml 173 | 174 | ``` 175 | 176 | 3. In BeanIO mapping file, reference the properties defined above: 177 | ```xml 178 | 179 | ``` 180 | 181 | ####charset 182 | The name of the character set to be used for reading and writing data, e.g., UTF-8. This property is optional, and if not set, the platform default charset is used. 183 | 184 | 185 | 186 | ###Batch Properties for `beanIOItemReader` Only 187 | In addition to the above common properties, `beanIOItemReader` also supports the following batch properties in job xml: 188 | 189 | ####skipBeanValidation 190 | `boolean` 191 | 192 | Indicates whether the current batch reader will invoke Bean Validation API to validate the incoming data POJO. Optional property and defaults to `false`, i.e., the reader will validate data POJO bean where appropriate. 193 | 194 | ####start 195 | `int` 196 | 197 | A positive integer indicating the start position in the input resource. It is optional and defaults to `1` (starting from the 1st data item). 198 | 199 | ####end 200 | `int` 201 | 202 | A positive integer indicating the end position in the input resource. It is optional and defaults to `Integer.MAX_VALUE`. 203 | 204 | ####errorHandler 205 | `java.lang.Class` 206 | 207 | A class implementing `BeanReaderErrorHandler` for handling exceptions thrown by a `BeanReader`. 208 | 209 | ####locale 210 | The locale name for this `beanIOItemReader` 211 | 212 | 213 | ###Batch Properties for `beanIOItemWriter` Only 214 | None 215 | 216 | -------------------------------------------------------------------------------- /batch_properties/README.md: -------------------------------------------------------------------------------- 1 | # Batch Properties 2 | ## Batch Property Scopes 3 | Batch properties are defined in job xml to configure the job. There are 3 groups of batch properties, according to their scopes: 4 | ### Job-level batch properties: 5 | 6 | To define job-level batch properties in job xml: 7 | 8 | ```xml 9 | 10 | 11 | 12 | 13 | 14 | 15 | ``` 16 | 17 | Access job-level batch properties from a batch artifact class with `javax.batch.runtime.context.JobContext#getProperties` 18 | 19 | ```java 20 | @Named 21 | public class MyBatchlet extends AbstractBatchlet { 22 | @Inject 23 | JobContext jobContext; 24 | 25 | private Properties jobProperties; 26 | 27 | @Override 28 | public String process() throws Exception { 29 | jobProperties = jobContext.getProperties(); 30 | String value1 = jobProperties.getProperty("jobProp1"); 31 | // ... 32 | return "Processed"; 33 | } 34 | } 35 | ``` 36 | 37 | Note that the properties obtained from `JobContext#getProperties` contains job-level batch properties as defined in job xml, and should not be used by application to store application data. For this purpose, `JobContext#setTransientUserData` should be used instead. 38 | 39 | ### Step-level batch properties 40 | 41 | To define step-level batch properties in job xml: 42 | ```xml 43 | 44 | 45 | 46 | 47 | 48 | 49 | ``` 50 | 51 | Access step-level batch properties from a batch artifact class with `javax.batch.runtime.context.StepContext#getProperties` 52 | 53 | ```java 54 | @Named 55 | public class MyBatchlet extends AbstractBatchlet { 56 | @Inject 57 | StepContext stepContext; 58 | 59 | private Properties stepProperties; 60 | 61 | @Override 62 | public String process() throws Exception { 63 | stepProperties = stepContext.getProperties(); 64 | String value1 = stepProperties.getProperty("stepProp1"); 65 | // ... 66 | return "Processed"; 67 | } 68 | } 69 | ``` 70 | Note that the properties obtained from `StepContext#getProperties` contains step-level batch properties as defined in job xml, and should not be used by application to store application data. For this purpose, `StepContext#setTransientUserData` or `StepContext#setPersistentUserData` should be used instead. 71 | 72 | ### Batch artifact properties 73 | 74 | To define batch artifact properties in job xml: 75 | ```xml 76 | 77 | 78 | 79 | 80 | 81 | ``` 82 | 83 | Batch artifact properties can be accessed from a batch artifact with injection. The optional attribute `javax.batch.api.BatchProperty#name` tells which batch property to inject into the annotated field. When `name` attribute is omitted, the target batch property name is the same as the field name. The following batchlet class injects the batch property named `name1` configured in the above job.xml into 2 fields: 84 | 85 | ```java 86 | @Named 87 | public class MyBatchlet extends AbstractBatchlet { 88 | @Inject 89 | @BatchProperty 90 | String name1; 91 | 92 | @Inject 93 | @BatchProperty(name = "name1") 94 | String title; 95 | 96 | //... 97 | } 98 | ``` 99 | 100 | Note that job- and step-level batch properties may not be directly injected into batch artifact classes. However, application may inject an batch artifact property that references a job- or step-level batch property via `jobProperties` substitution in job xml. For example, 101 | 102 | ```xml 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 117 | 119 | 120 | 121 | 122 | 123 | 124 | ``` 125 | 126 | ```java 127 | @Named 128 | public class MyBatchlet extends AbstractBatchlet { 129 | @Inject 130 | @BatchProperty 131 | String prop1; 132 | 133 | @Inject 134 | @BatchProperty 135 | String prop2; 136 | 137 | //... 138 | } 139 | ``` 140 | 141 | ## Batch Property Injection with @BatchProperty 142 | A batch artifact class can contain fields annotated with `@Inject` and `@BatchProperty` to inject properties defined in job xml into these fields. The injection field can be any of the following java types: 143 | 144 | * `java.lang.String` 145 | * `java.lang.StringBuilder` 146 | * `java.lang.StringBuffer` 147 | * any primitive type, and its wrapper type: 148 | * `boolean, Boolean` 149 | * `int, Integer` 150 | * `double, Double` 151 | * `long, Long` 152 | * `char, Character` 153 | * `float, Float` 154 | * `short, Short` 155 | * `byte, Byte` 156 | * `java.math.BigInteger` 157 | * `java.math.BigDecimal` 158 | * `java.net.URL` 159 | * `java.net.URI` 160 | * `java.io.File` 161 | * `java.util.jar.JarFile` 162 | * `java.util.Date` 163 | * `java.lang.Class` 164 | * `java.net.Inet4Address` 165 | * `java.net.Inet6Address` 166 | * `java.util.List, List, List` 167 | * `java.util.Set, Set, Set` 168 | * `java.util.Map, Map, Map, Map` 169 | * `java.util.logging.Logger` 170 | * `java.util.regex.Pattern` 171 | * `javax.management.ObjectName` 172 | 173 | The following array types are also supported: 174 | 175 | * `java.lang.String[]` 176 | * any primitive type, and its wrapper type: 177 | * `boolean[], Boolean[]` 178 | * `int[], Integer[]` 179 | * `double[], Double[]` 180 | * `long[], Long[]` 181 | * `char[], Character[]` 182 | * `float[], Float[]` 183 | * `short[], Short[]` 184 | * `byte[], Byte[]` 185 | * `java.math.BigInteger[]` 186 | * `java.math.BigDecimal[]` 187 | * `java.net.URL[]` 188 | * `java.net.URI[]` 189 | * `java.io.File[]` 190 | * `java.util.jar.JarFile[]` 191 | * `java.util.zip.ZipFile[]` 192 | * `java.util.Date[]` 193 | * `java.lang.Class[]` 194 | 195 | The following example injects a number into the batchlet class as various suitable types: 196 | 197 | ```xml 198 | 199 | 200 | 201 | 202 | 203 | ``` 204 | ```java 205 | @Named 206 | public class MyBatchlet extends AbstractBatchlet { 207 | @Inject 208 | @BatchProperty 209 | int number; // field name is the same as batch property name 210 | 211 | @Inject 212 | @BatchProperty (name = "number") 213 | long asLong; // use name attribute to locate batch property 214 | 215 | @Inject 216 | @BatchProperty (name = "number") 217 | Double asDouble; // inject as Double wrapper type 218 | 219 | @Inject 220 | @BatchProperty (name = "number") 221 | private String asString; // inject as private String 222 | 223 | @Inject 224 | @BatchProperty (name = "number") 225 | BigInteger asBigInteger; // inject as BigInteger 226 | 227 | @Inject 228 | @BatchProperty (name = "number") 229 | BigDecimal asBigDecimal; // inject as BigDecimal 230 | } 231 | ``` 232 | 233 | The following example injects a number sequence into the batchlet class as various arrays and collections: 234 | 235 | ```xml 236 | 237 | 238 | 239 | 240 | 241 | ``` 242 | 243 | ```java 244 | @Named 245 | public class MyBatchlet extends AbstractBatchlet { 246 | @Inject 247 | @BatchProperty 248 | int[] weekDays; 249 | 250 | @Inject 251 | @BatchProperty (name = "weekDays") 252 | Integer[] asIntegers; 253 | 254 | @Inject 255 | @BatchProperty (name = "weekDays") 256 | String[] asStrings; 257 | 258 | @Inject 259 | @BatchProperty (name = "weekDays") 260 | byte[] asBytes; 261 | 262 | @Inject 263 | @BatchProperty (name = "weekDays") 264 | BigInteger[] asBigIntegers; 265 | 266 | @Inject 267 | @BatchProperty (name = "weekDays") 268 | BigDecimal[] asBigDecimals; 269 | 270 | @Inject 271 | @BatchProperty (name = "weekDays") 272 | List asList 273 | 274 | @Inject 275 | @BatchProperty (name = "weekDays") 276 | List asListString; 277 | 278 | @Inject 279 | @BatchProperty (name = "weekDays") 280 | Set asSet; 281 | 282 | @Inject 283 | @BatchProperty (name = "weekDays") 284 | Set asSetString; 285 | } 286 | ``` 287 | 288 | The following example injects key-value pairs batch property into the batchlet class as a Map: 289 | ```xml 290 | 291 | 292 | 294 | 295 | 296 | ``` 297 | 298 | ```java 299 | @Named 300 | public class MyBatchlet extends AbstractBatchlet { 301 | @Inject 302 | @BatchProperty 303 | protected Map pairs; 304 | 305 | @Inject 306 | @BatchProperty (name = "pairs") 307 | private Map asMap; 308 | } 309 | ``` 310 | 311 | The following example injects class batch property into the batchlet class: 312 | ```xml 313 | 314 | 315 | 316 | 317 | 318 | ``` 319 | 320 | ```java 321 | @Named 322 | public class MyBatchlet extends AbstractBatchlet { 323 | @Inject 324 | @BatchProperty 325 | private Class clazz; 326 | } 327 | ``` 328 | 329 | The following example injects java.util.Date batch property into the batchlet class. Its string value is converted to java.util.Date based on the current system locale. 330 | ```xml 331 | 332 | 333 | 334 | 335 | 336 | ``` 337 | 338 | ```java 339 | @Named 340 | public class MyBatchlet extends AbstractBatchlet { 341 | @Inject 342 | @BatchProperty 343 | private java.util.Date date; 344 | } 345 | ``` 346 | 347 | The following example injects URL or URI batch property into the batchlet class: 348 | ```xml 349 | 350 | 351 | 352 | 353 | 354 | ``` 355 | 356 | ```java 357 | @Named 358 | public class MyBatchlet extends AbstractBatchlet { 359 | @Inject 360 | @BatchProperty 361 | private URL url; 362 | 363 | @Inject 364 | @BatchProperty (name = "url") 365 | private URI uri; 366 | } 367 | ``` 368 | 369 | The following example injects File or JarFile batch property into the batchlet class: 370 | ```xml 371 | 372 | 373 | 375 | 376 | 377 | ``` 378 | 379 | ```java 380 | @Named 381 | public class MyBatchlet extends AbstractBatchlet { 382 | @Inject 383 | @BatchProperty 384 | private File file; 385 | 386 | @Inject 387 | @BatchProperty (name = "file") 388 | private JarFile asJarFile 389 | } 390 | ``` 391 | 392 | Sometimes it's convenient to assign a default value to a field in artifact java class, whether the target batch property is defined in job xml or not. If the target property is resolved to a valid value, it is injected into that field; otherwise, no value is injected and the default field value is preserved. For example, 393 | 394 | ```java 395 | /** 396 | * Comment character. If commentChar batch property is not specified in job xml, 397 | * use the default value '#'. 398 | */ 399 | @Inject 400 | @BatchProperty 401 | private char commentChar = '#'; 402 | ``` 403 | 404 | ## Comparison of Various Batch Properties and Parameters 405 | 406 | | | What | API Access | JSL Access | @BatchProperty Injectable? | 407 | | -- | -- | -- | -- | -- | 408 | | Job Parameters | Props when starting or restarting a job | JobExecution.getJobParameters() | #{jobParameters['x']}| No | 409 | | Job-level Properties | Props element in job.xml directly within element | JobContext.getProperties() | #{jobProperties['x']} | No | 410 | | Step-level Properties | Props element in job.xml directly within step element | StepContext.getProperties() | #{jobProperties['x']} | No | 411 | | Batch Artifact Properties | Props directly within batch artifact in job.xml | @Inject @BatchProperty | #{jobParameters['x']} | Yes | 412 | | Partition Plan Properties | Props element with partition plan element in job.xml | None | #{partitionPlan['x']} | No | 413 | | Java System Properties | Java system properties | System.getProperty(), System.getProperties() | #{systemProperties['x']} | No | 414 | -------------------------------------------------------------------------------- /read_and_write_csv_data_with_jackson-dataformat-csv/README.md: -------------------------------------------------------------------------------- 1 | # Read and Write CSV Data with jackson-dataformat-csv 2 | 3 | In addition to item reader and writer based on super-csv library, jberet-support also contains `jacksonCsvItemReader` and `jacksonCsvItemWriter`, which implement reading from and writing to CSV using jackson-dataformat-csv library. This offers a convenient alternative especially for those applications that already depend on jackson family of libraries. 4 | 5 | The following dependency is required for `jacksonCsvItemReader` and `jacksonCsvItemWriter`: 6 | 7 | ```xml 8 | 9 | com.fasterxml.jackson.core 10 | jackson-core 11 | 12 | 13 | com.fasterxml.jackson.core 14 | jackson-databind 15 | 16 | 17 | com.fasterxml.jackson.core 18 | jackson-annotations 19 | 20 | 21 | com.fasterxml.jackson.module 22 | jackson-module-jaxb-annotations 23 | 24 | 25 | com.fasterxml.jackson.dataformat 26 | jackson-dataformat-csv 27 | 28 | ``` 29 | 30 | ## Configure `jacksonCsvItemReader` and `jacksonCsvItemWriter` in job xml 31 | 32 | The following is a partial example job xml defining `jacksonCsvItemReader` and `jacksonCsvItemWriter` with selected batch properties: 33 | 34 | ```xml 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | ``` 61 | 62 | ##Batch Configuration Properties for Both `jacksonCsvItemReader` and `jacksonCsvItemWriter` 63 | 64 | ###resource 65 | The resource to read from (for batch readers), or write to (for batch writers). 66 | 67 | ###beanType 68 | `java.lang.Class` 69 | 70 | Specifies a fully-qualified class or interface name that maps to a row of the source CSV file. For example, 71 | 72 | * a custom java type that represents data item and serves as the CSV schema class 73 | * `java.util.Map` 74 | * `java.util.List` 75 | * `java.lang.String[]` 76 | * `com.fasterxml.jackson.databind.JsonNode` 77 | 78 | When using `java.util.List` or `java.lang.String[]` for reading, it is deemed raw access, and CSV schema will not be configured and any schema-related properties are ignored. Specifically, CSV header and comment lines are read as raw access content. 79 | 80 | ###columns 81 | 82 | Specifies CSV schema in one of the 2 ways: 83 | * columns = "<fully-qualified class name>": 84 | 85 | CSV schema is defined in the named POJO class, which typically has class-level annotation com.fasterxml.jackson.annotation.JsonPropertyOrder to define property order corresponding to CSV column order. 86 | 87 | * columns = "<comma-separated list of column names, each of which may be followed by a space and column type>": 88 | 89 | use the value to manually build CSV schema. Valid column types are defined in `com.fasterxml.jackson.dataformat.csv.CsvSchema.ColumnType`, including: 90 | * `STRING` 91 | * `STRING_OR_LITERAL` 92 | * `NUMBER` 93 | * `NUMBER_OR_STRING` 94 | * `BOOLEAN` 95 | * `ARRAY` 96 | 97 | For complete list and descriptioin, see `com.fasterxml.jackson.dataformat.csv.CsvSchema.ColumnType` javadoc. 98 | 99 | For example, 100 | 101 | `columns = "org.jberet.support.io.StockTrade"` 102 | 103 | `columns = "firstName STRING, lastName STRING, age NUMBER"` 104 | 105 | 106 | In `jacksonCsvItemReader`, if this property is not defined and `useHeader` is true (CSV input has a header), the header is used to create CSV schema. However, when `beanType` is `java.util.List` or `java.lang.String[]`, the reader is considered raw access, and all schema-related properties are ignored. 107 | 108 | This property is optional for reader and required for writer class. 109 | 110 | See Also: `com.fasterxml.jackson.dataformat.csv.CsvSchema` 111 | 112 | ###useHeader 113 | `boolean` 114 | 115 | whether the first line of physical document defines column names (true) or not (false): if enabled, parser will take first-line values to define column names; and generator will output column names as the first line. Optional property. 116 | 117 | For `jacksonCsvItemReader`, if `beanType` is `java.util.List` or `java.lang.String[]`, it is considered raw access, `useHeader` property is ignored and no CSV schema is used. 118 | 119 | valid values are true or false, and the default is false. 120 | 121 | See Also: `com.fasterxml.jackson.dataformat.csv.CsvSchema` 122 | 123 | ###quoteChar 124 | 125 | Character used for quoting values that contain quote characters or linefeeds. Optional property and defaults to " (double-quote character). 126 | 127 | See Also: `com.fasterxml.jackson.dataformat.csv.CsvSchema` 128 | 129 | ###columnSeparator 130 | 131 | Character used to separate values. 132 | 133 | Optional property and defaults to , (comma character). Other commonly used values include tab (`\t`) and pipe (`|`) 134 | 135 | See Also: `com.fasterxml.jackson.dataformat.csv.CsvSchema` 136 | 137 | 138 | ###Other Jackson Configuration 139 | The following batch properties can be used to configure jackson json objects such as `JsonFactory`, `ObjectMapper`, serialization, deserialization, and custom jackson modules. 140 | * `jsonFactoryFeatures` 141 | * `mapperFeatures` 142 | * `jsonFactoryLookup` 143 | * `serializationFeatures` 144 | * `customSerializers` 145 | * `deserializationFeatures` 146 | * `customDeserializers` 147 | * `customDataTypeModules` 148 | 149 | See Chapter JsonItemReader and JsonItemWriter for more details. 150 | 151 | 152 | ##Batch Configuration Properties for `jacksonCsvItemReader` Only 153 | In addition to the common properties listed above, `jacksonCsvItemReader` also supports the following batch properties: 154 | 155 | ###skipBeanValidation 156 | `boolean` 157 | 158 | Indicates whether the current batch reader will invoke Bean Validation API to validate the incoming data POJO. Optional property and defaults to false, i.e., the reader will validate data POJO bean where appropriate. 159 | 160 | ###start 161 | `int` 162 | 163 | Specifies the start position (a positive integer starting from `1`) to read the data. If reading from the beginning of the input CSV, there is no need to specify this property. 164 | 165 | ###end 166 | `int` 167 | 168 | Specify the end position in the data set (inclusive). Optional property, and defaults to `Integer.MAX_VALUE`. If reading till the end of the input CSV, there is no need to specify this property. 169 | 170 | ###skipFirstDataRow 171 | 172 | Whether the first data line (either first line of the document, if `useHeader`=`false`, or second, if `useHeader`=`true`) should be completely ignored by parser. Needed to support CSV-like file formats that include additional non-data content before real data begins (specifically some database dumps do this) 173 | 174 | Optional property, valid values are `true` and `false` and defaults to `false`. 175 | 176 | See Also: `com.fasterxml.jackson.dataformat.csv.CsvSchema` 177 | 178 | ###escapeChar 179 | Character, if any, used to escape values. Most commonly defined as backslash (`\`). Only used by parser; generator only uses quoting, including doubling up of quotes to indicate quote char itself. Optional protected and defaults to `null`. 180 | 181 | See Also: `com.fasterxml.jackson.dataformat.csv.CsvSchema` 182 | 183 | ###jsonParserFeatures 184 | `java.util.Map` 185 | 186 | A comma-separated list of key-value pairs that specify `com.fasterxml.jackson.core.JsonParser` features. Optional property and defaults to `null`. For example, 187 | 188 | `ALLOW_COMMENTS=true, ALLOW_YAML_COMMENTS=true, ALLOW_NUMERIC_LEADING_ZEROS=true, STRICT_DUPLICATE_DETECTION=true` 189 | 190 | See Also: `com.fasterxml.jackson.core.JsonParser.Feature` 191 | 192 | ###csvParserFeatures 193 | `java.util.Map` 194 | 195 | A comma-separated list of key-value pairs that specify `com.fasterxml.jackson.dataformat.csv.CsvParser.Feature`. Optional property and defaults to null. For example, 196 | 197 | `TRIM_SPACES=false, WRAP_AS_ARRAY=false` 198 | 199 | See Also: `com.fasterxml.jackson.dataformat.csv.CsvParser.Feature` 200 | 201 | ###deserializationProblemHandlers 202 | A comma-separated list of fully-qualified names of classes that implement `com.fasterxml.jackson.databind.deser.DeserializationProblemHandler`, which can be registered to get called when a potentially recoverable problem is encountered during deserialization process. Handlers can try to resolve the problem, throw an exception or do nothing. Optional property and defaults to `null`. For example, 203 | 204 | `org.jberet.support.io.JsonItemReaderTest$UnknownHandler, org.jberet.support.io.JsonItemReaderTest$UnknownHandler2` 205 | 206 | See Also: `com.fasterxml.jackson.databind.deser.DeserializationProblemHandler`, `com.fasterxml.jackson.databind.ObjectMapper#addHandler(com.fasterxml.jackson.databind.deser.DeserializationProblemHandler)` 207 | 208 | ###inputDecorator 209 | `java.lang.Class` 210 | 211 | Fully-qualified name of a class that extends `com.fasterxml.jackson.core.io.InputDecorator`, which can be used to decorate input sources. Typical use is to use a filter abstraction (filtered stream, reader) around original input source, and apply additional processing during read operations. Optional property and defaults to null. For example, 212 | 213 | `org.jberet.support.io.JsonItemReaderTest$NoopInputDecorator` 214 | 215 | See Also: `com.fasterxml.jackson.core.JsonFactory#setInputDecorator(com.fasterxml.jackson.core.io.InputDecorator)`, `com.fasterxml.jackson.core.io.InputDecorator` 216 | 217 | 218 | 219 | ##Batch Configuration Properties for `jacksonCsvItemWriter` Only 220 | In addition to the common properties listed above, `jacksonCsvItemWriter` also supports the following batch properties: 221 | 222 | ###nullValue 223 | 224 | When asked to write Java `null`, this String value will be used instead. Optional property and defaults to empty string. 225 | 226 | See Also `com.fasterxml.jackson.dataformat.csv.CsvSchema` 227 | 228 | 229 | ###writeMode 230 | Instructs `csvItemWriter`, when the target CSV resource already exists, whether to append to, or overwrite the existing resource, or fail. Valid values are: 231 | * `append` (default) 232 | * `overwrite` 233 | * `failIfExists` 234 | 235 | ###lineSeparator 236 | 237 | Character used to separate data rows. Only used by generator; parser accepts three standard linefeeds (`\r`, `\r\n`, `\n`). Optional protected and defaults to `\n`. 238 | 239 | See Also: `com.fasterxml.jackson.dataformat.csv.CsvSchema` 240 | 241 | ###jsonGeneratorFeatures 242 | `java.util.Map` 243 | 244 | A comma-separated list of key-value pairs that specify `com.fasterxml.jackson.core.JsonGenerator` features. Optional property and defaults to null. Keys and values must be defined in `com.fasterxml.jackson.core.JsonGenerator.Feature`. For example, 245 | 246 | `WRITE_BIGDECIMAL_AS_PLAIN=true, WRITE_NUMBERS_AS_STRINGS=true, QUOTE_NON_NUMERIC_NUMBERS=false` 247 | 248 | See Also: `com.fasterxml.jackson.core.JsonGenerator.Feature` 249 | 250 | ###csvGeneratorFeatures 251 | `java.util.Map` 252 | 253 | A comma-separated list of key-value pairs that specify `com.fasterxml.jackson.dataformat.csv.CsvGenerator.Feature`. Optional property and defaults to `null`. For example, 254 | 255 | `STRICT_CHECK_FOR_QUOTING=false, OMIT_MISSING_TAIL_COLUMNS=false, ALWAYS_QUOTE_STRINGS=false` 256 | 257 | See Also: `com.fasterxml.jackson.dataformat.csv.CsvGenerator.Feature` 258 | 259 | ###outputDecorator 260 | `java.lang.Class` 261 | 262 | Fully-qualified name of a class that implements `com.fasterxml.jackson.core.io.OutputDecorator`, which can be used to decorate output destinations. Typical use is to use a filter abstraction (filtered output stream, writer) around original output destination, and apply additional processing during write operations. Optional property and defaults to `null`. For example, 263 | 264 | `org.jberet.support.io.JsonItemReaderTest$NoopOutputDecorator` 265 | 266 | See Also: `com.fasterxml.jackson.core.io.OutputDecorator`, `com.fasterxml.jackson.core.JsonFactory#setOutputDecorator(com.fasterxml.jackson.core.io.OutputDecorator)` 267 | -------------------------------------------------------------------------------- /jsonitemreader_and_jsonitemwriter/README.md: -------------------------------------------------------------------------------- 1 | # JsonItemReader and JsonItemWriter 2 | 3 | jberet-support module contains `jsonItemReader` and `jsonItemWriter` batch artifacts for reading and writing JSON data. `jsonItemReader` reads JSON data and binds them to `java.util.Map`, `com.fasterxml.jackson.databind.JsonNode`, or any custom POJO bean provided by the batch application. 4 | 5 | `jsonItemReader` and `jsonItemWriter` leverages `jackson` library to handle JSON data parsing and binding. The following `jackson` dependencies are required: 6 | 7 | ```xml 8 | 9 | com.fasterxml.jackson.core 10 | jackson-core 11 | ${version.com.fasterxml.jackson} 12 | 13 | 14 | com.fasterxml.jackson.core 15 | jackson-databind 16 | ${version.com.fasterxml.jackson} 17 | 18 | 19 | com.fasterxml.jackson.core 20 | jackson-annotations 21 | ${version.com.fasterxml.jackson} 22 | 23 | 24 | 25 | 26 | 27 | com.fasterxml.jackson.module 28 | jackson-module-jaxb-annotations 29 | ${version.com.fasterxml.jackson} 30 | 31 | ``` 32 | 33 | ##Batch Configuration Properties in Job XML 34 | 35 | `jsonItemReader` and `jsonItemWriter` are configured through `` or `` batch properties in job xml. All properties are of type `String`, unless noted otherwise. The following is an example job xml that references `jsonItemReader` and `jsonItemWriter`: 36 | 37 | ```xml 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | ``` 69 | 70 | ###Batch Properties for Both `jsonItemReader` and `jsonItemWriter` 71 | 72 | ####resource 73 | 74 | The resource to read from (for batch readers), or write to (for batch writers). 75 | 76 | ####jsonFactoryFeatures 77 | A comma-separated list of key-value pairs that specify `com.fasterxml.jackson.core.JsonFactory.Feature`s. Optional property and defaults to `null`. Only keys and values defined in `com.fasterxml.jackson.core.JsonFactory.Feature` are accepted. For example, 78 | 79 | `INTERN_FIELD_NAMES=false, CANONICALIZE_FIELD_NAMES=false` 80 | 81 | See `com.fasterxml.jackson.core.JsonFactory.Feature`, and `org.jberet.support.io.NoMappingJsonFactoryObjectFactory.configureJsonFactoryFeatures(com.fasterxml.jackson.core.JsonFactory, java.lang.String)` for more details. 82 | 83 | ####mapperFeatures 84 | A comma-separated list of key-value pairs that specify `com.fasterxml.jackson.databind.ObjectMapper` features. Optional property and defaults to `nul`l. Only keys and values defined in `com.fasterxml.jackson.databind.MapperFeature` are accepted. For example, 85 | 86 | `USE_ANNOTATIONS=false, AUTO_DETECT_FIELDS=false, AUTO_DETECT_GETTERS=true` 87 | 88 | See `com.fasterxml.jackson.databind.MapperFeature`, and `org.jberet.support.io.MappingJsonFactoryObjectFactory.configureMapperFeatures(com.fasterxml.jackson.databind.ObjectMapper, java.lang.String)` for more details. 89 | 90 | ####jsonFactoryLookup 91 | JNDI lookup name for `com.fasterxml.jackson.core.JsonFactory`, which is used for constructing `com.fasterxml.jackson.core.JsonParser` in `jsonItemReader` and `com.fasterxml.jackson.core.JsonGenerator` in `jsonItemWriter`. Optional property and defaults to `null`. When this property is specified, its value is used to look up an instance of `com.fasterxml.jackson.core.JsonFactory`, which is typically created and administrated externally (e.g., inside application server). Otherwise, a new instance of `com.fasterxml.jackson.core.JsonFactory` is created instead of lookup. 92 | 93 | ####serializationFeatures 94 | A comma-separated list of key-value pairs that specify `com.fasterxml.jackson.databind.SerializationFeature`s. Optional property and defaults to `null`. Only keys and values defined in `com.fasterxml.jackson.databind.SerializationFeature` are accepted. For example, 95 | 96 | `WRAP_ROOT_VALUE=true, INDENT_OUTPUT=true, FAIL_ON_EMPTY_BEANS=false` 97 | 98 | See `com.fasterxml.jackson.databind.SerializationFeature`, and `org.jberet.support.io.MappingJsonFactoryObjectFactory.configureSerializationFeatures(com.fasterxml.jackson.databind.ObjectMapper, java.lang.String) ` for more details. 99 | 100 | ####customSerializers 101 | A comma-separated list of fully-qualified name of classes that implement `com.fasterxml.jackson.databind.JsonSerializer`, which can serialize Objects of arbitrary types into JSON. For example, 102 | 103 | `org.jberet.support.io.JsonItemReaderTest$JsonSerializer, org.jberet.support.io.JsonItemReaderTest$JsonSerializer2` 104 | 105 | See `com.fasterxml.jackson.databind.JsonSerializer`, and `org.jberet.support.io.MappingJsonFactoryObjectFactory.configureCustomSerializersAndDeserializers(com.fasterxml.jackson.databind.ObjectMapper, java.lang.String, java.lang.String, java.lang.ClassLoader)` for more details. 106 | 107 | ####deserializationFeatures 108 | A comma-separated list of key-value pairs that specify `com.fasterxml.jackson.databind.DeserializationFeature`s. Optional property and defaults to `null`. Only keys and values defined in `com.fasterxml.jackson.databind.DeserializationFeature` are accepted. For example, 109 | 110 | `USE_BIG_DECIMAL_FOR_FLOATS=true, USE_BIG_INTEGER_FOR_INTS=true, USE_JAVA_ARRAY_FOR_JSON_ARRAY=true` 111 | 112 | See `com.fasterxml.jackson.databind.DeserializationFeature`, and `org.jberet.support.io.MappingJsonFactoryObjectFactory.configureDeserializationFeatures(com.fasterxml.jackson.databind.ObjectMapper, java.lang.String)` for more details. 113 | 114 | ####customDeserializers 115 | A comma-separated list of fully-qualified name of classes that implement `com.fasterxml.jackson.databind.JsonDeserializer`, which can deserialize Objects of arbitrary types from JSON. For example, 116 | 117 | `org.jberet.support.io.JsonItemReaderTest$JsonDeserializer, org.jberet.support.io.JsonItemReaderTest$JsonDeserializer2` 118 | 119 | See `com.fasterxml.jackson.databind.JsonDeserializer`, and `org.jberet.support.io.MappingJsonFactoryObjectFactory.configureCustomSerializersAndDeserializers(com.fasterxml.jackson.databind.ObjectMapper, java.lang.String, java.lang.String, java.lang.ClassLoader)` for more details. 120 | 121 | ####customDataTypeModules 122 | A comma-separated list of Jackson datatype module type ids that extend `com.fasterxml.jackson.databind.Module`. These modules will be registered with `objectMapper`. For example, 123 | 124 | `com.fasterxml.jackson.datatype.joda.JodaModule, com.fasterxml.jackson.datatype.jsr353.JSR353Module, com.fasterxml.jackson.datatype.jsr310.JSR310Module, com.fasterxml.jackson.module.afterburner.AfterburnerModule` 125 | 126 | 127 | ###Batch Properties for `jsonItemReader` Only 128 | In addition to the above common properties, `jsonItemReader` also supports the following batch properties in job xml: 129 | 130 | ####skipBeanValidation 131 | `boolean` 132 | 133 | Indicates whether the current batch reader will invoke Bean Validation API to validate the incoming data POJO. Optional property and defaults to `false`, i.e., the reader will validate data POJO bean where appropriate. 134 | 135 | ####beanType 136 | `java.lang.Class` 137 | 138 | The bean type that represents individual data item in the source Json `resource`. Required property, and valid values are: 139 | ` 140 | * any custom bean type, for example `org.jberet.support.io.StockTrade` 141 | * `java.util.Map` 142 | * `com.fasterxml.jackson.databind.JsonNode` 143 | 144 | ####start 145 | `int` 146 | 147 | Specifies the start position (a positive integer starting from `1`) to read the data. If reading from the beginning of the input Json resource, there is no need to specify this property. 148 | 149 | ####end 150 | `int` 151 | 152 | Specify the end position in the data set (inclusive). Optional property, and defaults to `Integer.MAX_VALUE`. If reading till the end of the input Json resource, there is no need to specify this property. 153 | 154 | ####jsonParserFeatures 155 | `java.util.Map` 156 | 157 | A comma-separated list of key-value pairs that specify `com.fasterxml.jackson.core.JsonParser` features. Optional property and defaults to `null`. For example, 158 | 159 | `ALLOW_COMMENTS=true, ALLOW_YAML_COMMENTS=true, ALLOW_NUMERIC_LEADING_ZEROS=true, STRICT_DUPLICATE_DETECTION=true` 160 | 161 | See `com.fasterxml.jackson.core.JsonParser.Feature` for more details. 162 | 163 | ####deserializationProblemHandlers 164 | A comma-separated list of fully-qualified names of classes that implement `com.fasterxml.jackson.databind.deser.DeserializationProblemHandler`, which can be registered to get called when a potentially recoverable problem is encountered during deserialization process. Handlers can try to resolve the problem, throw an exception or do nothing. Optional property and defaults to `null`. For example, 165 | 166 | `org.jberet.support.io.JsonItemReaderTest$UnknownHandler, org.jberet.support.io.JsonItemReaderTest$UnknownHandler2` 167 | 168 | See `com.fasterxml.jackson.databind.deser.DeserializationProblemHandler`, `com.fasterxml.jackson.databind.ObjectMapper#addHandler(com.fasterxml.jackson.databind.deser.DeserializationProblemHandler)`, `org.jberet.support.io.MappingJsonFactoryObjectFactory.configureDeserializationProblemHandlers(com.fasterxml.jackson.databind.ObjectMapper, java.lang.String, java.lang.ClassLoader)` for more details. 169 | 170 | ####inputDecorator 171 | `java.lang.Class` 172 | 173 | Fully-qualified name of a class that extends `com.fasterxml.jackson.core.io.InputDecorator`, which can be used to decorate input sources. Typical use is to use a filter abstraction (filtered stream, reader) around original input source, and apply additional processing during read operations. Optional property and defaults to `null`. For example, 174 | 175 | `org.jberet.support.io.JsonItemReaderTest$NoopInputDecorator` 176 | 177 | See `com.fasterxml.jackson.core.JsonFactory#setInputDecorator(com.fasterxml.jackson.core.io.InputDecorator)`, and `com.fasterxml.jackson.core.io.InputDecorator` for more details. 178 | 179 | 180 | ###Batch Properties for `jsonItemWriter` Only 181 | In addition to the above common properties, `jsonItemWriter` also supports the following batch properties in job xml: 182 | 183 | ####writeMode 184 | Instructs this class, when the target Json resource already exists, whether to append to, or overwrite the existing resource, or fail. Valid values are `append`, `overwrite`, and `failIfExists`. Optional property, and defaults to `append`. 185 | 186 | ####jsonGeneratorFeatures 187 | `java.util.Map` 188 | 189 | A comma-separated list of key-value pairs that specify `com.fasterxml.jackson.core.JsonGenerator` features. Optional property and defaults to `null`. Keys and values must be defined in `com.fasterxml.jackson.core.JsonGenerator.Feature`. For example, 190 | 191 | `WRITE_BIGDECIMAL_AS_PLAIN=true, WRITE_NUMBERS_AS_STRINGS=true, QUOTE_NON_NUMERIC_NUMBERS=false` 192 | 193 | See `com.fasterxml.jackson.core.JsonGenerator.Feature` for more details. 194 | 195 | ####prettyPrinter 196 | `java.lang.Class` 197 | 198 | Fully-qualified name of a class that implements `com.fasterxml.jackson.core.PrettyPrinter`, which implements pretty printer functionality, such as indentation. Optional property and defaults to `null` (the system default pretty printer is used). For example, 199 | 200 | `com.fasterxml.jackson.core.util.MinimalPrettyPrinter` 201 | 202 | See `com.fasterxml.jackson.core.PrettyPrinter`, and `com.fasterxml.jackson.core.JsonGenerator#setPrettyPrinter(com.fasterxml.jackson.core.PrettyPrinter)` for more details. 203 | 204 | ####outputDecorator 205 | `java.lang.Class` 206 | 207 | Fully-qualified name of a class that implements `com.fasterxml.jackson.core.io.OutputDecorator`, which can be used to decorate output destinations. Typical use is to use a filter abstraction (filtered output stream, writer) around original output destination, and apply additional processing during write operations. Optional property and defaults to `null`. For example, 208 | 209 | `org.jberet.support.io.JsonItemReaderTest$NoopOutputDecorator` 210 | 211 | See `com.fasterxml.jackson.core.io.OutputDecorator`, and `com.fasterxml.jackson.core.JsonFactory#setOutputDecorator(com.fasterxml.jackson.core.io.OutputDecorator)` for more details. 212 | -------------------------------------------------------------------------------- /excel_itemreaders_and_itemwriters/README.md: -------------------------------------------------------------------------------- 1 | # Excel ItemReaders and ItemWriters 2 | 3 | jberet-module includes `excelEventItemReader`, `excelUserModelItemReader`, `excelUserModelItemWriter`, `excelStreamingItemReader`, `excelStreamingItemWriter` for working with Excel files, including both binary Excel files (.xls) and OOXML Excel files (.xlsx). They are all based on Apache POI library for low-level Excel operations. 4 | 5 | | Name | Excel File Type | Underlying POI or XML API | Benefits | 6 | | -- | -- | -- | 7 | | `excelEventItemReader` | .xls | POI event model API | small memory footprint | 8 | | `excelUserModelItemReader` | .xls & .xlsx | POI user model API | simple API for in-memory access | 9 | | `excelUserModelItemWriter` | .xls & .xlsx | POI user model API | simple API for in-memory content generation | 10 | | `excelStreamingItemReader` | .xlsx | POI XSSF streaming reader API & StAX | stream read large data set | 11 | | `excelStreamingItemWriter` | .xlsx | POI SXSSF (buffered streaming) API | write large data set | 12 | 13 | 14 | The following dependencies are required: 15 | ```xml 16 | 17 | org.apache.poi 18 | poi 19 | ${version.org.apache.poi} 20 | compile 21 | 22 | 23 | commons-codec 24 | commons-codec 25 | ${version.commons-codec} 26 | 27 | 28 | 29 | 30 | org.apache.poi 31 | poi-ooxml 32 | ${version.org.apache.poi} 33 | 34 | 35 | org.apache.poi 36 | poi-ooxml-schemas 37 | ${version.org.apache.poi} 38 | 39 | 40 | org.apache.xmlbeans 41 | xmlbeans 42 | ${version.org.apache.xmlbeans} 43 | 44 | 45 | stax 46 | stax-api 47 | 48 | 49 | 50 | 51 | dom4j 52 | dom4j 53 | ${version.dom4j} 54 | 55 | 56 | xml-apis 57 | xml-apis 58 | 59 | 60 | 61 | ``` 62 | ##Batch Configuration in Job XML 63 | `excelEventItemReader`, `excelUserModelItemReader`, `excelUserModelItemWriter`, `excelStreamingItemReader`, `excelStreamingItemWriter` are configured through `` or `` batch properties in job xml. All properties are of type `String`, unless noted otherwise. The following is example job xml files that reference these readers and writers: 64 | 65 | ###`excelEventItemReader` 66 | ```xml 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95 | 96 | 97 | 98 | ``` 99 | ###`excelUserModelItemReader` 100 | ```xml 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | ``` 127 | 128 | ###`excelUserModelItemWriter` 129 | ```xml 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | ``` 157 | 158 | ###`excelStreamingItemReader` 159 | ```xml 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | ``` 173 | 174 | ###`excelStreamingItemWriter` 175 | ```xml 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | ``` 204 | 205 | --------------------- 206 | 207 | ###Batch Properties for All Excel Readers and Writers 208 | 209 | ####resource 210 | The resource to read from (for batch readers), or write to (for batch writers). 211 | 212 | ####beanType 213 | `java.lang.Class` 214 | 215 | ####header 216 | `java.lang.String[]` 217 | 218 | Specifies the header as an ordered string array. For reader, header information must be specified with either this property or `ExcelUserModelItemReader.headerRow` property. This property is typically specified when there is no header row in the Excel file. For example, 219 | 220 | `"id, name, age"` specifies 1st column is `id`, 2nd column is `name` and 3rd column is `age`. 221 | 222 | This is a required property for writer. 223 | 224 | ####sheetName 225 | The optional name of the target sheet. When specified for a reader, it has higher precedence over `ExcelUserModelItemReader.sheetIndex` 226 | 227 | ####additional jackson-related batch properties 228 | All Excel readers and writers use `jackson-databind` for data transformation behind the scene, and this step can be configured through the following batch configuration properties in job xml, though the defaults should suffice in most cases: 229 | 230 | * `jsonFactoryFeatures` 231 | * `mapperFeatures` 232 | * `jsonFactoryLookup` 233 | * `serializationFeatures` 234 | * `customSerializers` 235 | * `deserializationFeatures` 236 | * `customDeserializers` 237 | * `customDataTypeModules` 238 | 239 | See Chapter JsonItemReader and JsonItemWriter for more details. 240 | 241 | ----------------- 242 | 243 | ###Batch Properties for `excelUserModelItemReader` 244 | In additional to the shared batch properties listed above, `excelUserModelItemReader` can also be configured through the following batch properties: 245 | 246 | ####start 247 | `int` 248 | 249 | A positive integer indicating the start position in the input resource. It is optional and defaults to `0` (starting from the 1st data item). If a header row is present, the start point should be after the header row. 250 | 251 | ####end 252 | `int` 253 | 254 | A positive integer indicating the end position in the input resource. It is optional and defaults to `Integer.MAX_VALUE`. 255 | 256 | ####sheetIndex 257 | `int` 258 | 259 | The index (`0`-based) of the target sheet to read, defaults to `0`. 260 | 261 | ####headerRow 262 | `java.lang.Integer` 263 | 264 | The physical row number of the header. 265 | 266 | ------------- 267 | 268 | ###Batch Properties for `excelEventItemReader` 269 | ####All the shared batch properties for all Excel readers and writers 270 | 271 | ####All the batch properties for `excelUserModelItemReader` 272 | 273 | ####queueCapacity 274 | `int` 275 | 276 | the capacity of the queue used by `org.apache.poi.hssf.eventusermodel.HSSFListener` to hold pre-fetched data rows. Optional property and defaults to `ExcelEventItemReader.MAX_WORKSHEET_ROWS` (`65536`). 277 | 278 | -------------- 279 | 280 | ###Batch Properties for `excelStreamingItemReader` 281 | ####All the shared batch properties for all Excel readers and writers 282 | 283 | ####All the batch properties for `excelUserModelItemReader` 284 | 285 | -------------- 286 | 287 | ###Batch Properties for `excelUserModelItemWriter` 288 | In addition to the shared batch properties for all Excel readers and writers, `excelUserModelItemWriter` also support the following batch properties: 289 | 290 | ####writeMode 291 | Valid `writeMode` for this writer class is `overwrite` and `failIfExists`. 292 | 293 | ####templateResource 294 | The resource of an existing Excel file or template file to be used as a base for generating output Excel. Its format is similar to `resource` batch property. 295 | 296 | ####templateSheetName 297 | The sheet name in the template file to be used for generating output Excel. If `templateResource` is specified but this property is not specified, `templateSheetIndex` is used instead. 298 | 299 | ####templateSheetIndex 300 | `int` 301 | 302 | The sheet index (`0`-based) in the template file to be used for generating output Excel. 303 | 304 | ####templateHeaderRow 305 | `java.lang.Integer` 306 | 307 | The row number (`0`-based) of the header in the template sheet. If `header` property is provided in job xml file, then this property is ignored. Otherwise, it is used to retrieve header values. 308 | 309 | ------------ 310 | 311 | ###Batch Properties for `excelStreamingItemWriter` 312 | ####All the shared batch properties for all Excel readers and writers 313 | 314 | ####All the batch properties for `excelUserModelItemWriter` 315 | 316 | ####compressTempFiles 317 | `java.lang.Boolean` 318 | 319 | Whether to compress the temp files in the course of generating Excel file, defaults to `false`. 320 | 321 | 322 | -------------------------------------------------------------------------------- /develop_batch_artifacts_in_script_languages/README.md: -------------------------------------------------------------------------------- 1 | # Develop Batch Artifacts in Script Languages 2 | Whether it's for data ETL or quick testing, script language offers a valuable alternative to Java in developing batch applications. JBeret supports writing `Batchlet`, `ItemReader`, `ItemProcessor` and `ItemWriter` in popular script languages. A job xml may reference an external script resource, or directly include script as CDATA or PCDATA. 3 | 4 | JBeret relies on JSR-223 Scripting for the Java™ Platform (available in Java SE) standard API to run script batch artifacts. Each script language used by a batch application requires its JSR-223-compliant script engine as runtime dependency. The following table lists some common script languages and script engines: 5 | 6 | | Script Language | Script Engine | Obtain from | Impl javax.script.Invocable?| Suitable for | 7 | | -- | -- | -- | -- | -- | 8 | | JavaScript | Mozilla Rhino | included in Oracle JDK 5, 6 & 7 | Yes | reader, processor, writer & batchlet | 9 | | JavaScript | Oracle Nashorn | included in Oracle JDK 8 | Yes | reader, processor, writer & batchlet | 10 | | Groovy | org.codehaus.groovy:groovy-jsr223 | Maven Central | Yes | reader, processor, writer & batchlet | 11 | | Jython / Python | org.python:jython | Maven Central | Yes | reader, processor, writer & batchlet | 12 | | JRuby / Ruby | org.jruby:jruby | Maven Central | Yes | reader, processor, writer & batchlet | 13 | | Scala | org.scala-lang:scala-compiler | Maven Central | No | processor & batchlet | 14 | | PHP | com.caucho:resin-quercus | caucho-repository http://caucho.com/m2/ | No | processor & batchlet | 15 | | R (Renjin) | org.renjin:renjin-script-engine | http://nexus.bedatadriven.com/content/groups/public | Yes | processor & batchlet | 16 | 17 | The following XML snippet includes all the above script engine dependencies. A batch application should only include what is really needed at runtime. 18 | 19 | ```xml 20 | 21 | 22 | org.codehaus.groovy 23 | groovy-jsr223 24 | 25 | 26 | org.codehaus.groovy 27 | groovy 28 | 29 | 30 | 31 | org.jruby 32 | jruby 33 | 34 | 35 | 36 | org.python 37 | jython 38 | 39 | 40 | 41 | org.scala-lang 42 | scala-compiler 43 | 44 | 45 | 46 | com.caucho 47 | resin-quercus 48 | 49 | 50 | 51 | org.renjin 52 | renjin-script-engine 53 | 54 | 55 | com.google.guava 56 | guava 57 | 58 | 59 | 60 | 61 | 62 | caucho-repository 63 | http://caucho.com/m2/ 64 | 65 | 66 | bedatadriven 67 | bedatadriven public repo 68 | http://nexus.bedatadriven.com/content/groups/public/ 69 | 70 | 71 | ``` 72 | 73 | ##Write `Batchlet`, `ItemReader`, `ItemProcessor` or `ItemWriter` in Scripts 74 | Batch artifacts written in script languages must follow the following rules: 75 | * Rules for `ItemReader` script: 76 | 77 | * The script engine must implement `javax.script.Invocable` so JBeret can invoke various methods defined in `ItemReader` interface. 78 | * An `ItemReader` script must implement `readItem` function, or another function mapped to `readItem` method. Other methods from `ItemReader` interface (`open`, `close` & `checkpointInfo`) may be optionally implemented. 79 | 80 | * Rules for `ItemProcessor` script: 81 | 82 | * If the script engine implements `javax.script.Invocable`, and the `ItemProcessor` script implements `processItem` function, or another function mapped to `processItem` method, that function will be invoked. 83 | * Otherwise, the script content is evaluated to fulfill `processItem` method. 84 | 85 | * Rules for `ItemWriter` script: 86 | * The script engine must implement `javax.script.Invocable` so JBeret can invoke various methods defined in `ItemWriter` interface. 87 | * An `ItemWriter` script must implement `writeItems` function, or another function mapped to `writeItems` method. Other methods from `ItemWriter` interface (`open`, `close` & `checkpointInfo`) may be optionally implemented. 88 | 89 | * Rules for `Batchlet` script: 90 | * `process` method requirement: 91 | * If the script engine implements `javax.script.Invocable`, and the `Batchlet` script implements `process` function, or another function mapped to `process` method, that function will be invoked to fulfill `process` method. 92 | * Otherwise, the script content is evaluated to fulfill `process` method. 93 | * `stop` method requirement: 94 | * If the script engine implements `javax.script.Invocable`, and the `Batchlet` script implements `stop` function, or another function mapped to `stop` method, that function will be invoked to fulfill `stop` method. 95 | * Otherwise, nothing is done. 96 | 97 | ###Method-to-function Mapping 98 | By default, script function names are the same as method names in batch API interfaces. In certain cases, custom names may be needed to avoid naming conflict, or to follow different naming convention. This can be achieved with `methodMapping` property under ``, ``, ``, or `` element in job xml. 99 | 100 | The property value is a comma-separated list of key-value pairs, with the key as method name in batch API interfaces, and value as the function name in script. 101 | 102 | ```xml 103 | 104 | ``` 105 | 106 | Note that batch API method names like `open` or `close` may be reserved words or built-in functioins in certain script languages. If so, they must be mapped to different function names. 107 | 108 | ###Access Batch Objects from Script 109 | JBeret exposes the following batch objects to the script in the scope of `javax.script.ScriptContext#ENGINE_SCOPE`, so the script application can access information and interact with batch runtime. 110 | 111 | | Name | Description | Type | Example (syntax varies) | 112 | | -- | -- | -- | -- | 113 | | `jobContext` | `JobContext` of the current job execution | `javax.batch.runtime.context.JobContext` | `jobContext.getJobName()`, `jobContext.setExitStatus('xxx')` | 114 | | `stepContext` | `StepContext` of the current step execution | `javax.batch.runtime.context.StepContext` | `stepContext.getStepName()` | 115 | | `batchProperties` | properties specified in job xml under the current batch artifact | `java.util.Properties` | `batchProperties.get('testName')` | 116 | 117 | 118 | ##Configure Script in Job XML 119 | Script may be configured in job xml either as a direct sub-element, or as an external reference. To achieve that, a ` 159 | 160 | 161 | 162 | ``` 163 | 164 | The script content may be CDATA or element text, and CDATA should be the preferred method to avoid issues with special characters. 165 | 166 | Note that in certian script languages (e.g., Python), indentation is significant. So any inline script content should be indented from the left-most column, regardless of XML indentation. 167 | 168 | Inline script is convenient for short, or even no-op batch artifact, for example, 169 | 170 | ```xml 171 | 172 | 175 | 176 | ``` 177 | 178 | ###External Script Resource Referenced by Job XML 179 | To reference an external script file or resource, simply specify `src` attribute, and optionally `type` attribute. 180 | 181 | ```xml 182 | 183 | 184 | 185 | 186 | 187 | 188 | 231 | 232 | 233 | 234 | 235 | 236 | 243 | 244 | 245 | 246 | 247 | 248 | ``` 249 | ----------------------- 250 | __item-reader.py:__ 251 | 252 | ```python 253 | rows = [] 254 | position = 0 255 | 256 | def openBatchReader(checkpoint): 257 | global rows 258 | resource = batchProperties.get("resource") 259 | 260 | f = open(resource, 'rb') 261 | try: 262 | for row in f.readlines(): 263 | columnValues = row.split(",") 264 | rows.append(columnValues) 265 | finally: 266 | f.close() 267 | 268 | if (checkpoint is None): 269 | position = checkpoint 270 | 271 | 272 | def checkpointInfo(): 273 | return position 274 | 275 | 276 | def readItem(): 277 | global position 278 | if (position >= len(rows)): 279 | return None 280 | 281 | item = rows[position] 282 | position += 1; 283 | return item; 284 | 285 | ``` 286 | 287 | ###External Groovy Reader, Inline Processor and Writer: 288 | 289 | ```xml 290 | 291 | 292 | 293 | 294 | 295 | 296 | 297 | 314 | 315 | 316 | 323 | 324 | 325 | 326 | 327 | 328 | ``` 329 | ----------------------------- 330 | __ItemReader.groovy__ 331 | 332 | ```groovy 333 | package groovy 334 | 335 | import groovy.transform.Field 336 | 337 | @Field List rows; 338 | @Field int position = 0; 339 | 340 | def open(checkpoint) { 341 | String resourcePath = batchProperties.get("resource"); 342 | InputStream inputFile = this.class.getClassLoader().getResourceAsStream(resourcePath); 343 | 344 | String[] lines = inputFile.text.split('\n'); 345 | rows = lines.collect { it.split(',') }; 346 | inputFile.close(); 347 | 348 | if (checkpoint != null) { 349 | position = checkpoint; 350 | } 351 | println("ItemReader.groovy open, rows: " + rows); 352 | } 353 | 354 | def checkpointInfo() { 355 | return position; 356 | } 357 | 358 | def readItem() { 359 | if (position >= rows.size()) { 360 | return null; 361 | } 362 | return rows.get(position++); 363 | } 364 | ``` 365 | 366 | ###External Ruby Reader, Inline Processor and Writer: 367 | ```xml 368 | 369 | 370 | 371 | 372 | 373 | 374 | 375 | 380 | 381 | 382 | 400 | 401 | 402 | 403 | 404 | 405 | 421 | 422 | 423 | 424 | 425 | 426 | ``` 427 | -------------------- 428 | __item-reader.rb__ 429 | 430 | ```ruby 431 | require 'csv' 432 | 433 | $rows = [] 434 | $position = 0 435 | 436 | def openBatchReader(checkpoint) 437 | resource = $batchProperties.get("resource") 438 | puts(resource) 439 | $rows = CSV.read(resource) 440 | 441 | if checkpoint != nil 442 | $position = checkpoint 443 | end 444 | 445 | puts($rows) 446 | end 447 | 448 | def closeBatchReader() 449 | puts('In reader close') 450 | end 451 | 452 | 453 | def checkpointInfo() 454 | return $position 455 | end 456 | 457 | 458 | def readItem() 459 | if $position >= $rows.length 460 | return nil 461 | end 462 | 463 | item = $rows[$position] 464 | $position += 1 465 | return item 466 | end 467 | ``` 468 | ###Inline Scala Batchlet: 469 | 470 | ```xml 471 | 472 | 473 | 474 | 475 | 476 | 477 | 492 | 493 | 494 | 495 | ``` 496 | 497 | ###Inline R (Renjin) Batchlet: 498 | 499 | ```xml 500 | 501 | 502 | 504 | ]> 505 | 506 | 507 | 508 | 509 | &batchlet-properties-segment; 510 | 525 | 526 | 527 | 528 | ``` 529 | 530 | --------------------------------------------------------------------------------