Beginner’s guide for building a data-intensive Spring /Spring Boot App?

I use the Spring Boot framework immensely to develop various data processing pipeline applications for batch and streaming processes. Usually, Flask is my go-to option for any application development. Even though I am a Pythonic person, I liked spring boot capabilities when comes to building data pipelines using spark or Kafka streams.

So, Initially, I always used to get confused or lost whenever reading the code in GitHub for coding style and best practices reference. Different projects use different libraries and no idea on what are those libraries used for. So I thought of consolidating these popular libraries and read through one by one.

Let's take an example use case

Develop a cloud-based microservice application to stream the data from Streaming services like Kafka and load them into a database or cloud storage. Log the application metrics and push those metrics to dev-ops analyzer tools like graphite. These stored data should be accessible through REST APIs. The application should be secure and fault-tolerant.

If we break down this requirement into high-level components, we will end up like this

1. Streaming App — Read the messages from the Kafka streaming service and process or aggregate the data and load it into the data store.
2. Monitoring — Log the application performance metrics like no of rows processed, processing time, etc.
3. Security — Store the credentials for databases, cloud services, and API tokens in the vault and retrieve whenever required through the application.
4. Metrics Dashboard — Collected metrics should be published to monitor tools like graphite.
5. REST API — Expose these processed stored data to the client or UI through the REST service.
6. Architecture Style — Since it will be microservices, have to take care of service discovery for fault tolerance, API Gateway, Circuit Breaker, etc.

Spring Boot framework made easy for us to develop such components or services and integrate seamlessly. When comes to libraries, there are various options available. Here are the most popular libraries used to build those various components. We can pick based on our choice and explore it.

Personally, I use mind mapping to get a clear big picture.

Here it is. Cool MindMap Viz built using https://coggle.it/