Disclaimer

Sprngy Platform Azure Engineering Guide

Release 1.0.0.1

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

This document is intended for:

SPRNGYCORE Administrators
SPRNGYCORE Developers
SPRNGYCORE Architects

This document is a walkthrough to create and edit a Meta Model through the Sprngy UI.

Pre-requisites:

The list of tools required to run/develop in the UI are as follows:

Visual Studio Code (Developers only)
Python 3.8.10
Flask 1.1.4
Node 14.17.6
npm 7.24.1
R 3.6.3
Big Analytixs Libraries

Meta Model:

A meta model is a format that allows to captured detailed information about attributes of data sets that are to be processed. Below are the processes to Create/Edit a Meta Model from the UI.

Before Creating a meta model, it is important to remember the following:

The unique field while creating an attribute is “entity_attribute_model_name”. You will get an error if you try to create an attribute with the same name.
When the meta model is created, a field called “as_of_date” will be system generated and defaulted. You will not be able to edit or delete that field.

Home Screen:

The image below is the Home screen of the UI.

Click on the three lines (pulls up the menu) and click on the “Create/Edit Meta Model”

Screen 0:

This checks the existence of a metamodel. There are a few checks taking place:

First, the local JSON files are checked for. If it does not exist(implies that you have not left creating/editing the metamodel halfway), it will then check the HDFS for that meta model. If it exists in the HDFS, then that existing metamodel is pulled. If not, then the metamodel will be created from scratch.

On the next screen there will be fields to add attribute name (i.e. Column Name) and it’s datatype. This screen is visible only when creating the meta model for first time Add all the column names with it’s datatypes (string,numeric or integer) and click Submit. On submit a meta model will be created with default values.

The next page rendered will be of a grid with created column names. To customise or change any configurations click on edit icon of that column name.

Attributes Grid Screen:

Clicking on the “Submit” button on the Form1 page, will save your preEntity details locally and route you to the Attributes/Grid Screen. In this screen you have the list of attributes that exist in the meta model.

There are multiple functionalities that are available on this screen:

1. Add New Attribute - It navigates you to the Form2 as explained below:

Form2(AttributeForm):

You can only fill and save one attribute at a time. The following is a 5 step page to incorporate all the details required to create one attribute:

Step 1:

Field Name	Description	Data Type	Is Required?	Validation
Entity Attribute Model Name	The name of the attribute	String	Yes	No spaces
Entity Attribute Model data format	The format of the attribute - character, numeric, or integer	String	Yes	one of the three in column B
Entity Attribute Model Data Length	The length of the field	Numeric	Yes	only positive numbers
Entity Attribute Model Data Precision	if numeric then precision	Numeric	Yes	only positive numbers
Entity Attribute Unique Ind	Yes/No Indicator	(Yes or No flag)	Yes	YES/NO
Entity Attribute In Use Ind	Yes/No Indicator	(Yes or No flag)	Yes	YES/NO
Entity Attribute Description	The metadata definition of the attribute	String	Yes
Entity Attribute Sample Value	The sample value of the attribute	String	Yes	Length of field should be same as input of “Entity Attribute Model Data Length”
Entity Attribute Default Value	The default value to assign if not imputing to treat nulls	String	Yes	Length of field should be same as input of “Entity Attribute Model Data Length”

Step 2:

Field Name	Description	Data Type	Is Required?	Validation
Entity Attribute Transform Rule Inbound	The transform rule for the attribute	String	Yes
Entity Attribute Transform Rule Outbound	The transform rule for the attribute	String	Yes	by default
Entity Attribute Validation Rule	the validation rule for the attribute	String	Yes	by default NONE
Entity Attribute Validation Rule Description	the validation rule for the attribute	String	Yes	by default NONE
Entity Attribute Validation Rule Iterate by Key Role	Yes/No Indicator	String	Yes	YES/NO
Entity Attribute Quality Rule	a~b	String	Yes	Has to have “~” in the rule or “NONE” if there is no quality rule
Entity Attribute Quality Rule Description	a~b	String	Yes
Entity Attribute Quality Rule Iterate By Key Role	other than a or b	String	Yes

Step 3:

Field Name	Description	Data Type	Is Required?	Validation
Entity Attribute Impute Method	if the attribute is null, pick out an impute method. For character columns, it's the default value. For numeric nd integers, mean or median	String	Yes	one of the three in column B
Entity Attribute Compare Key Role	Yes/No Indicator	String	Yes	YES/NO
Entity Attribute Filter Key	the construct to filter records	String	Yes	default is NO
Entity Attribute Nested Key Role	Whether the given attribute should be nested or not	String	Yes	default is YES
Entity Attribute Parent Lookup Key Role	(Yes or No flag)	String	Yes	YES/NO
Entity Attribute Parent Lookup Location	The hdfs location of the parent entity for performing lookups	String	Yes	default is NONE
Entity Attribute Calendar join key Role	(Yes or No flag)	String	Yes	YES/NO

Step 4:

Field Name	Description	Data Type	Is Required?	Validation
Entity Attribute Iqm Codes Match	(Yes or No flag)	String	Yes	YES/NO
Entity Attribute Iqm Metric Match	(Yes or No flag)	String	Yes	YES/NO
Entity Attribute Iqm Iterate By Key Role	(Yes or No flag)	String	Yes	YES/NO
Entity Attribute Eda Dimension	(Yes or No flag)	String	Yes	YES/NO
Entity Attribute Eda Metric	(Yes or No flag)	String	Yes	YES/NO
Entity Attribute Eda Iterate By Key Role	(Yes or No flag)	String	Yes	YES/NO

Step 5:

Field Name	Description	Data Type	Is Required?	Validation
Entity BI Object Repository	defaults to the module name	String	Yes	NO Space
Entity BI Object Name	defaults to, The name of the attribute	String	Yes	NO Space
Entity Attribute BI Name	defaults to , The format of the attribute - character, numeric, or integer- can be editable	String	Yes	NO Space
Entity Attribute BI Mask Ind	(Yes or No flag)	String	Yes	YES/NO
Entity Attribute BI Mask	Whether or not to Mask the attribute	String	Yes
Entity Attribute BI Data Format	The format of the attribute - character, numeric, or integer	String	Yes
Entity Attribute BI Data Length	The length of the field	String	Yes	Number
Entity Attribute BI Data Precision	if numeric then precision	String	Yes	Number

After you are done filling out the details for a single attribute, click the “SAVE ATTRIBUTE” button to save the file locally and route you back to the Attributes Grid Page.

2. The View Icon - This icon navigates you to another screen which displays the details of the particular attribute.

3. Edit Icon - The Edit icon gives you the ability to edit that particular attribute and save it at every step of the way.

4. Delete Icon - The Delete Icon gives you the ability to delete that particular attribute. A pop up will appear reconfirming your decision to delete the attribute.

After making all changes to the meta model, you can click on the “Final Submit” button in order to update the existing meta model in the HDFS. It is IMPORTANT to remember that you have to click on the “FINAL SUBMIT” button in order for your changes to be permanently saved.