Disclaimer
Sprngy Platform Azure Engineering Guide
Release 1.0.0.1
Copyright © Sprngy. All rights reserved.
This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.
This document is intended for:
SPRINGYCORE Administrators
SPRINGYCORE Developers
SPRINGYCORE Architects
This document is a walkthrough to create and edit an Ingest Model through the Sprngy UI.
Pre-requisites:
The list of tools required to run/develop in the UI are as follows:
Visual Studio Code (Developers only)
Python 3.8.10
Flask 1.1.4
Node 14.17.6
npm 7.24.1
R 3.6.3
Sprngy Libraries
Ingest Model:
An ingest model is used to define rules for processing the entity data. Below are the processes to Create and Edit a Meta Model from the UI.
Creating an Ingest Model:
Home Screen:
The image below is the Home screen of the UI.From the side menu open ‘Meta Data Configuration' option, then select 'Ingest Model’ option.
Create Ingest:
Create Ingest is divided into two forms one for module and entity information and the second for adding processors for each data lake.
Form 1 -
Clicking on “Create/Edit Ingest Model” will navigate you to Step 1 of Ingest form. This form requires you to fill out the preEntity details and is a page with two steps. The following is the “Step 1”:
Field Name | Description | Data Type | is Required? | Validation
|
|---|---|---|---|---|
Module Name | Choose from the list of modules | String | Yes | One of the seven on the website |
Entity Name | Choose from the list of entities | String | Yes | Only appears after the module name is selected |
Entity Description | The description of the respective entity | String | Yes |
|
Once you click on “Next”, you will be sent to “Step 2” to fill out the other preEntity details. The following image is “Step 2”:
Field Name | Description | Data Type | is Required? | Validation |
|---|---|---|---|---|
Entity Business Owner Name | First name + Last name of Entity Business Owner | String | Yes |
|
Entity Business Owner Email | The Business Owner email Id | String | Yes | should have @ |
Entity Data Owner Name | The Data Owner Name of Entity | String | Yes |
|
Entity Data Owner Email | The Data Owner Email of Entity | String | Yes | should have @ |
Entity It Owner Name | The IT owner name of an entity | String | Yes |
|
Entity It Owner Email | The IT owner email of the entity | String | Yes | should have @ |
Vendor Support Name | The vendor support name of the entity.Default Non-editable to support@biganalytixs.com | String | Auto-Fill |
|
Vendor Support Email | The vendor support name of the entity.Default Non-editable to support@biganalytixs.com | String | Auto-Fill |
|
After these details have been filled out, you can hit on “Back” to go to “Step 1” or if you are done filling out your PreEntity Details you can hit on “Submit”.
Form 2:
If you hit “Continue” (in the case that the ingest model exists) or “Submit” (in the case that it does not exist), you will be taken to the second part of ingest form. This part is a two-step process wherein first we select the data lake and choose the processors from the list of available processors for that layer and then we decide if we want to stop or continue execution if a processor fails. These three steps are repeated for all the data layers in the order of SDL, FDL, and BDL. After submitting for BDL layer ingest model will be created.
Here we first select SDL as rule from layer which would autofill ‘Rule Group’, ‘Rule to Layer' and default Rule Name. You can select from the drop-down other rules for sdl-fdl layer and click 'Submit’
Field Name | Description | Data Type | Is Required? | Validation |
|---|---|---|---|---|
Rule Group | Whether the rule is used for ingestion, integration, or analytics | String | Yes | Values should be passed in ALL Caps |
Rule From Layer | The name of the source data lake layer upon which the rule is required to be run. | String | Yes | |
Rule to Layer | The name of the destination data lake layer in which the results after the rule is run is to be written. If the rule_from_layer is SDL, the rule_to_layer is FDL, and so on and so forth | String | Yes | Should be populated automatically based on the selection in rule_from_layer |
Rule Name | The name of the rule | String | Yes |
Next is defining if the workload should stop or continue if that processor fails.
After making the required changes click on ‘Submit’
This will bring you back to the layer delection screen. Here select ‘FDL’ from the drop-down of ‘Rule From Layer’, this will auto fill ‘Rule Group’, ‘Rule to Layer' and default Rule Name. You can select from the drop-down other rules for fdl-bdl layer and click 'Submit’.
Next is defining if the workload should stop or continue if that processor fails.
After making the required changes click on ‘Final Submit’. This will save the ingest model to HDFS
Editing an Ingest
From the menu, if we select “Create/Edit Ingest Model” Screen 0 will be rendered.
Screen 0:
Screen 0 takes in module name and entity name and on submitting returns back the ingest model to be edited. The next screen is the module and entity information 2-step form which was filled while creating the ingest form. It will have pre-populated fields.
We can update these values or can simply submit which will bring a selection page. If you want to edit any data layer then select that layer and click ‘Submit' or else click 'Final Submit’ to save the edited ingest model in HDFS.
From this page, we can select any layer which we want to edit. On submit will redirect to the two-step process that we filled up while creating.
It will be pre-populated with previously saved data and we can change the information and click on submit. The submit will render back to the selection page from which we can edit another layer or do a “Final Submit”. By clicking on “Final Submit” the updated object is stored in HDFS.