CTI’s big data architecture blueprints vary based on a company’s infrastructure and needs, but they generally will include the following components.
All big data architecture starts with your sources. This can include data from media, the cloud, the web, databases, data from real-time sources such as IoT devices, and static files generated from applications. The architecture must be designed to support it being must be classified or sourced well in order to achieve to assure its usability and relevance.
For real-time sources, we'll consider the best way to build a mechanism into your architecture to ingest that data. Data storage for the data that will be processed via big data architecture often stored in a data lake as a large unstructured database that scales easily will also be considered.
The architecture will address the need to handle both real-time data and static data. The large volume of data processed can be handled efficiently using batch processing, while real-time data needs to be processed immediately to bring value. Batch processing approaches will be considered for long-running jobs to filter, aggregate, and prepare the data for analysis.
It’s important to architect an orchestration platform and process to move the data through these various systems in some form of automation. Ingesting and transforming the data, moving it in batches and stream processes, loading it to an analytical data store, and finally deriving insights must be in a repeatable workflow.
