...
The image below is the Home screen of the UI. From the side menu open ‘Meta Data Configuration' option, then select 'Ingest Model’ option.
...
Create Ingest:
Create Ingest is divided into two forms one for module and entity information and the second for adding processors for each data lake.
Form 1 -
Clicking on “Create/Edit Ingest Model” will navigate you to Step 1 of Ingest form. This form requires you to fill out the preEntity details and is a page with two steps. The following is the “Step 1”:
...
Step 1 -
After clicking on the Ingest Model menu option below screen will be displayed.
...
To create a new Ingest Model, select the required Module name and Entity name from the drop down and click 'Submit'.
This checks the existence of a ingest model. If there is a previously created or even initiated ingest model then it will bring back that data, if creating ingest model for the very first time then provides blank fields for input.
Step 2 -
After submission, the below screen will be rendered. Here you can provide information regarding the ingest model for given entity.
...
This two step form takes information such as entity description and business owner, data owner and IT owner information working on this entity. Once done click on Submit.
Field Name | Description | Data Type | is Required? | Validation
|
|---|---|---|---|---|
Module Name | Choose from the list of modules | String | Yes | One of the seven on the website |
Entity Name | Choose from the list of entities | String | Yes | Only appears after the module name is selected |
Entity Description | The description of the respective entity | String | Yes |
|
Once you click on “Next”, you will be sent to “Step 2” to fill out the other preEntity details. The following image is “Step 2”:
...
Field Name | Description | Data Type | is Required? | Validation |
|---|---|---|---|---|
Entity Business Owner Name | First name + Last name of Entity Business Owner | String | Yes |
|
Entity Business Owner Email | The Business Owner email Id | String | Yes | should have @ |
Entity Data Owner Name | The Data Owner Name of Entity | String | Yes |
|
Entity Data Owner Email | The Data Owner Email of Entity | String | Yes | should have @ |
Entity It Owner Name | The IT owner name of an entity | String | Yes |
|
Entity It Owner Email | The IT owner email of the entity | String | Yes | should have @ |
Vendor Support Name | The vendor support name of the entity.Default Non-editable to support@biganalytixs.com | String | Auto-Fill |
|
Vendor Support Email | The vendor support name of the entity.Default Non-editable to support@biganalytixs.com | String | Auto-Fill |
|
...
After these details have been filled out, you can hit on “Back” to go to “Step 1” or if you are done filling out your PreEntity Details you can hit on “Submit”.
Form 2:
If you hit “Continue” (in the case that the ingest model exists) or “Submit” (in the case that it does not exist), you will be taken to the second part of ingest form. This part is a two-step process wherein first we select the data lake and choose the processors from the list of available processors for that layer and then we decide if we want to stop or continue execution if a processor fails. These three steps are repeated for all the data layers in the order of SDL, FDL, and BDL. After submitting for BDL layer ingest model will be created.
Here we first select SDL as rule from layer which would autofill ‘Rule Group’, ‘Rule to Layer' and default Rule Name. You can select from the drop-down other rules for sdl-fdl layer and click 'Submit’
...
...
Field Name
...
Description
...
Data Type
...
Is Required?
...
Validation
...
Rule Group
...
Whether the rule is used for ingestion, integration, or analytics
...
String
...
Yes
...
Values should be passed in ALL Caps
...
Rule From Layer
...
The name of the source data lake layer upon which the rule is required to be run.
...
String
...
Yes
...
Rule to Layer
...
The name of the destination data lake layer in which the results after the rule is run is to be written. If the rule_from_layer is SDL, the rule_to_layer is FDL, and so on and so forth
...
String
...
Yes
...
Should be populated automatically based on the selection in rule_from_layer
...
Rule Name
...
The name of the rule
...
String
...
Yes
Next is defining if the workload should stop or continue if that processor fails.
...
After making the required changes click on ‘Submit’
This will bring you back to the layer delection screen. Here select ‘FDL’ from the drop-down of ‘Rule From Layer’, this will auto fill ‘Rule Group’, ‘Rule to Layer' and default Rule Name. You can select from the drop-down other rules for fdl-bdl layer and click 'Submit’.
...
Next is defining if the workload should stop or continue if that processor fails.
...
After making the required changes click on ‘Final Submit’. This will save the ingest model to HDFS
Editing an Ingest
From the menu, if we select “Create/Edit Ingest Model” Screen 0 will be rendered.
...
Screen 0:
...
Screen 0 takes in module name and entity name and on submitting returns back the ingest model to be edited. The next screen is the module and entity information 2-step form which was filled while creating the ingest form. It will have pre-populated fields.
...
We can update these values or can simply submit which will bring a selection page. If you want to edit any data layer then select that layer and click ‘Submit' or else click 'Final Submit’ to save the edited ingest model in HDFS.
...
From this page, we can select any layer which we want to edit. On submit will redirect to the two-step process that we filled up while creating.
...
It will be pre-populated with previously saved data and we can change the information and click on submit. The submit will render back to the selection page from which we can edit another layer or do a “Final Submit”. By clicking on “Final Submit” the updated object is stored in HDFSStep 3 -
On submit, you will get a form with actions to be taken while processing data. It consist of a list of questions such as ways of reading data, remove nulls from dataset, impute the nulls with default value and so on.
...
After filling the details, click on 'Submit'. This would save the ingest model details.
Editing an Ingest
Editing a Ingest model has similar steps to creating a Ingest Model. From the side menu open ‘Meta Data Configuration' option, then select 'Ingest Model’ option.
...
Step 1 -
After clicking on the Ingest Model menu option below screen will be displayed.
...
Here select the module and entity of which ingest model you need to update and click on submit button.
This will load the information you previously stored for that module and entity. First of all there is the entity owner information that you can update. After that there will the form describing the rules to be applied while processing data.
Step 2 -
On Submitting the owner information, below screen will be loaded.
...
Here, the rules can be updated as required and the click on 'Submit'. This will save the changes made to the ingest model.
Bulk Upload Ingest Model
There is also one other way to create a ingest model, that is by uploading a CSV file. For this, you are required to have a ingest model .csv file with all the required fields. To directly upload a csv -
Home Screen:
The image below is the Home screen of the UI.
From the side menu open ‘Meta Data Configuration' option, then select 'Ingest Model’ option.
...
Step 1 -
After clicking on the Ingest Model menu option below screen will be displayed.
...
Select the module and entity for which you want to upload a ingest model. In the file upload option browse or drag and drop the ingest model csv file and click Submit.