java - 用于消耗来自不同数据源的大量数据的体系结构设计(Architectural design for consuming big volume of data from different data sources)

Question

The current product implementation is: In the back end we are using spring- boot and for the front-end we have React\Redux app.

(当前的产品实现是：在后端，我们使用spring-boot，在前端，我们使用React \ Redux应用程序。)

Up until now, we were working with only one database (Oracle) with a small amount of data.

(到目前为止，我们只使用一个数据库（Oracle）处理少量数据。)

Lately due to business needs (more info on the client-side) we needed to pull more data form few more databases (can be Oracle and MySql) that contain a huge amount of data.

(最近，由于业务需求（客户端上的更多信息），我们需要从更多的包含大量数据的数据库（可以是Oracle和MySql）中提取更多数据。)

Doing this with regular SQL via spring boot will not work due to the amount of data.

(由于数据量大，无法通过Spring Boot使用常规SQL进行此操作。)

We cannot work in the database since they belong to external customers and we also need to combine data for analyzing.

(由于他们属于外部客户，因此我们无法在数据库中工作，我们还需要合并数据进行分析。)

To summaries – I am looking for some middleware product that can work with different data-sources, able to handle the high volume of data, capabilities to create API in order to process data that coming from different sources and finally this need to be very fast in order to view the output on the client-side (Rest\React).

(总结–我正在寻找一些中间件产品，该产品可以与不同的数据源一起使用，能够处理大量数据，具有创建API以便处理来自不同源的数据的功能，最后这需要非常快为了查看客户端上的输出（Rest \ React）。)

Will be happy to hear your stories\experience and recommendation.

(很高兴听到您的故事\经验和推荐。)

Thank you

(谢谢)

  ask by angus translate from so

Answer 1

We have recently implemented somewhat similar ask from our client.

(我们最近从客户那里实施了一些类似的要求。)

Solution is fully on AWS Cloud Native and Serverless.

(解决方案完全在AWS Cloud Native和无服务器上。)

The back end (both data consolidation from disparate sources and Data retrieval to React) can be done in several different ways.

(后端（从不同来源进行数据整合以及从数据检索到React）都可以通过几种不同的方式完成。)

Data Consolidation back end - Existing Java code can be factored into Lambda Java Functions and by enabling VPC, these can also talk to On-Prem sources of Data.

(数据整合后端-现有的Java代码可以分解为Lambda Java函数，并且通过启用VPC，它们也可以与本地数据源进行通信。)

Alternately, Batch can be used Or Databricks if a commercial tool is ready.

(或者，如果已经准备好商业工具，则可以使用Batch或Databricks。)

Data Retrieval for React - Serverless Java functions if needed to leverage existing code/logic.

(React的数据检索-如果需要利用现有代码/逻辑，则无服务器Java功能。)

Data Storage - Redshift?

(数据存储-Redshift？)

Depending on your budgets.

(根据您的预算。)