...
This document is a walkthrough to create and edit an Ingest Model through the Sprngy UI.
Pre-requisites:
The list of tools required to run/develop in the UI are as follows:
...
Visual Studio Code (Developers only)
...
Python 3.8.10
...
Flask 1.1.4
...
Node 14.17.6
...
npm 7.24.1
...
R 3.6.3
...
Application is created in Admin UI using the Application Configuration screens and entities and relationships are defined.
Meta Models and Ingest Models are defined.
Data is imported (either through data upload UI or using import model UI)
Manage Workloads
Manage Workloads includes running workloads and Managing workloads. It keeps track of all the workloads that ran for which layer as well as the result of it.
...
S.No. | Rule From Layer | Rule To Layer(Defaulted) |
|---|---|---|
1. | SDL (Staging Data Lake) | FDL (Fast Data Lake) |
2. | FDL (Fast Data Lake) | BDL |
3. | BDL | BAL |
...
(Business Data Lake) |
The SDL → FDL workload executes pre-built data profiling pipelines as per the preferences provided in meta and ingest model. The FDL → BDL workload executes pre-built pipelines creating business datasets based on relationships and preferences defined in the meta and ingest model.
While running workloads, SDL → FDL workloads should be run first followed by FDL → BDL workloads.
Manage Workload
It is a grid having table listing all the batches which have run previously.
...
The Grid table displays Module Name, Entity Name, To Layer, From Layer, Batch_id, Workload Status, and Workload State for all the batches that run before. It also provides an option for a re-run and deleting batch.
...