Batch Processing with Spring Cloud Data Flow

Azure Container Apps is a fully managed serverless container service that enables you to build and deploy modern, cloud-native Java applications and microservices at scale. It offers a simplified developer experience while providing the flexibility and portability of containers.

Of course, Azure Container Apps has really solid support for our ecosystem, from a number of build options, managed Java components, native metrics, dynamic logger, and quite a bit more.

To learn more about Java features on Azure Container Apps, visit the documentation page.

You can also ask questions and leave feedback on the Azure Container Apps GitHub page.

Of course, Azure Container Apps has really solid support for our ecosystem, from a number of build options, managed Java components, native metrics, dynamic logger, and quite a bit more.

To learn more about Java features on Azure Container Apps, you can get started over on the documentation page.

And, you can also ask questions and leave feedback on the Azure Container Apps GitHub page.

Modern software architecture is often broken. Slow delivery leads to missed opportunities, innovation is stalled due to architectural complexities, and engineering resources are exceedingly expensive.

Orkes is the leading workflow orchestration platform built to enable teams to transform the way they develop, connect, and deploy applications, microservices, AI agents, and more.

With Orkes Conductor managed through Orkes Cloud, developers can focus on building mission critical applications without worrying about infrastructure maintenance to meet goals and, simply put, taking new products live faster and reducing total cost of ownership.

Try a 14-Day Free Trial of Orkes Conductor today.

Orkes is the leading workflow orchestration platform built to enable teams to transform the way they develop, connect, and deploy applications, microservices, AI agents, and more.

Try a 14-Day Free Trial of Orkes Conductor today.

End-to-end testing is a very useful method to make sure that your application works as intended. This highlights issues in the overall functionality of the software, that the unit and integration test stages may miss.

Playwright is an easy-to-use, but powerful tool that automates end-to-end testing, and supports all modern browsers and platforms.

When coupled with LambdaTest (an AI-powered cloud-based test execution platform) it can be further scaled to run the Playwright scripts in parallel across 3000+ browser and device combinations:

>> Automated End-to-End Testing With Playwright

1. Overview

In the first article of the series, we introduced Spring Cloud Data Flow‘s architectural component and how to use it to create a streaming data pipeline.

As opposed to a stream pipeline, where an unbounded amount of data is processed, a batch process makes it easy to create short-lived services where tasks are executed on demand.

2. Local Data Flow Server and Shell

The Local Data Flow Server is a component that is responsible for deploying applications, while the Data Flow Shell allows us to perform DSL commands needed for interacting with a server.

In the previous article, we used Spring Initilizr to set them both up as a Spring Boot Application.

After adding the @EnableDataFlowServer annotation to the server’s main class and the @EnableDataFlowShell annotation to the shell’s main class respectively, they are ready to be launched by performing:

mvn spring-boot:run

The server will boot up on port 9393 and a shell will be ready to interact with it from the prompt.

You can refer to the previous article for the details on how to obtain and use a Local Data Flow Server and its shell client.

3. The Batch Application

As with the server and the shell, we can use Spring Initilizr to set up a root Spring Boot batch application.

After reaching the website, simply choose a Group, an Artifact name and select Cloud Task from the dependencies search box.

Once this is done, click on the Generate Project button to start downloading the Maven artifact.

The artifact comes preconfigured and with basic code. Let’s see how to edit it in order to build our batch application.

3.1. Maven Dependencies

First of all, let’s add a couple of Maven dependencies. As this is a batch application, we need to import libraries from the Spring Batch Project:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-batch</artifactId>
</dependency>

Also, as the Spring Cloud Task uses a relational database to store results of an executed task, we need to add a dependency to an RDBMS driver:

<dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
</dependency>

We’ve chosen to use the H2 in-memory database provided by Spring. This gives us a simple method of bootstrapping development. However, in a production environment, you’ll want to configure your own DataSource.

Keep in mind that artifacts’ versions will be inherited from Spring Boot’s parent pom.xml file.

3.2. Main Class

The key point to enabling desired functionality would be to add the @EnableTask and @EnableBatchProcessing annotations to the Spring Boot’s main class. This class level annotation tells Spring Cloud Task to bootstrap everything:

@EnableTask
@EnableBatchProcessing
@SpringBootApplication
public class BatchJobApplication {

    public static void main(String[] args) {
        SpringApplication.run(BatchJobApplication.class, args);
    }
}

3.3. Job Configuration

Lastly, let’s configure a job – in this case a simple print of a String to a log file:

@Configuration
public class JobConfiguration {

    private static Log logger
      = LogFactory.getLog(JobConfiguration.class);

    @Autowired
    public JobBuilderFactory jobBuilderFactory;

    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Bean
    public Job job() {
        return jobBuilderFactory.get("job")
          .start(stepBuilderFactory.get("jobStep1")
          .tasklet(new Tasklet() {
            
              @Override
              public RepeatStatus execute(StepContribution contribution, 
                ChunkContext chunkContext) throws Exception {
                
                logger.info("Job was run");
                return RepeatStatus.FINISHED;
              }
        }).build()).build();
    }
}

Details on how to configure and define a job are outside the scope of this article. For more information, you can see our Introduction to Spring Batch article.

Finally, our application is ready. Let’s install it inside our local Maven repository. To do this cd into the project’s root directory and issue the command:

mvn clean install

Now it’s time to put the application inside the Data Flow Server.

4. Registering the Application

To register the application within the App Registry we need to provide a unique name, an application type, and a URI that can be resolved to the app artifact.

Go to the Spring Cloud Data Flow Shell and issue the command from the prompt:

app register --name batch-job --type task 
  --uri maven://com.baeldung.spring.cloud:batch-job:jar:0.0.1-SNAPSHOT

5. Creating a Task

A task definition can be created using the command:

task create myjob --definition batch-job

This creates a new task with the name myjob pointing to the previously registeredbatch-job application .

A listing of the current task definitions can be obtained using the command:

task list

6. Launching a Task

To launch a task we can use the command:

task launch myjob

Once the task is launched the state of the task is stored in a relational DB. We can check the status of our task executions with the command:

task execution list

7. Reviewing the Result

In this example, the job simply prints a string in a log file. The log files are located within the directory displayed in the Data Flow Server’s log output.

To see the result we can tail the log:

tail -f PATH_TO_LOG\spring-cloud-dataflow-2385233467298102321\myjob-1472827120414\myjob
[...] --- [main] o.s.batch.core.job.SimpleStepHandler: Executing step: [jobStep1]
[...] --- [main] o.b.spring.cloud.JobConfiguration: Job was run
[...] --- [main] o.s.b.c.l.support.SimpleJobLauncher:
  Job: [SimpleJob: [name=job]] completed with the following parameters: 
    [{}] and the following status: [COMPLETED]

8. Conclusion

In this article, we have shown how to deal with batch processing through the use of Spring Cloud Data Flow.

The code backing this article is available on GitHub. Once you're logged in as a Baeldung Pro Member, start learning and coding on the project.

Of course, Azure Container Apps has really solid support for our ecosystem, from a number of build options, managed Java components, native metrics, dynamic logger, and quite a bit more.

To learn more about Java features on Azure Container Apps, visit the documentation page.

You can also ask questions and leave feedback on the Azure Container Apps GitHub page.

Orkes is the leading workflow orchestration platform built to enable teams to transform the way they develop, connect, and deploy applications, microservices, AI agents, and more.

Try a 14-Day Free Trial of Orkes Conductor today.

Orkes is the leading workflow orchestration platform built to enable teams to transform the way they develop, connect, and deploy applications, microservices, AI agents, and more.

Try a 14-Day Free Trial of Orkes Conductor today.

Of course, Azure Container Apps has really solid support for our ecosystem, from a number of build options, managed Java components, native metrics, dynamic logger, and quite a bit more.

To learn more about Java features on Azure Container Apps, visit the documentation page.

You can also ask questions and leave feedback on the Azure Container Apps GitHub page.