Our client is a big player on stock exchange market. They approached us to help them build an ecosystem for market data processing.
Their main business was to build a reliable and scalable solution for real-time stock data processing.
The data had to be extracted, aggregated, notification rules applied and emails sent to the stakeholders. Another requirement was that the storage ecosystem/data processors had to be located in Azure Cloud. As our customer expected fast data growth, data storage/processing engines had to be chosen appropriately.
Taking into account our customer’s requirement to avoid whenever possible custom code development, we have built streaming application based on Talend for Big Data in the scope of our work.
Per customer request, we’ve placed run-engines and orchestrators on VMs in Azure cloud. For fast market data distribution we utilized Azure Service Bus. And we have also used Apache Hadoop and Impala for data storage and analysis.