API Description:
This is an open API to fetch the daily time series data for IBM stock.
The API returns data in JSON format, including daily open, high, low, close prices, and volume for IBM.
The dataset includes the latest 100 trading days.
Data is updated every two to three days.
Incremental Ingestion
To ensures that business intelligence tools and dashboards display the most current data, we will build CDC pipeline enables continuous, incremental updates to data warehouses by propagating only the changes instead of reloading complete datasets. Also we will make sure to keep all historical data saved in our staging table as the API 100 days API window moving forward,
Timestamp-based CDC
We will rely on the last modified date for each record to determine the new records to be updated by comparing timestamps between source and target , the system identifies records that have changed since the previous check.
Bronze Layer
E : Extract DATA
T :Transformation : Data Parsing & Reshaping
Connecting to destination data warehouse & read staging table
L : Data Loading: into warehouse (target database.)
Silver Layer
The last stage of the pipeline is carried out with a stored procedure that updates the silver table to persist daily, cleaned, and up-to-date records.
