Sprngy - Creating New Applications Demo Part 2

This document details steps to setup new applications using Sprngy Admin UI, defining models and running workloads.

Demo Case: Finance

Example 1: Analyzing Inflation Dataset

Objective

• Determining how one can live an inflation proof lifestyle by analyzing the percentage change in the consumer price index.

• To read data file from google drive and loading the data directly to BDL fact using analytical model.

Overview

Inflation dataset has one entity i.e. inflation and the dataset exhibits the percentage change value (every year) in consumer price index from the year 1947 to 2022.

NOTE: Before we start creating an application make sure you have create a google drive folder on your local through google drive connector, more information on how to create it can be found on this page:

 https://biganalytixs.atlassian.net/l/cp/8P16hNvz , so that when we upload the data file into google drive, it directly reflects onto your google drive folder which is created in your local.

Classifying the Application:

Based on the data technology used and the business use, the application is classified as below:

Step 1: Setting up the application

Now that the business use and classification of the application is established, the application can be created using the UI. In AdminUI, set up the application by going to the Set-up Application tab, select Create New, and filling out the file structure. Since we have just one layer file system we will have just one entity in it.

Step 2: Creating Meta Model

We can now set up the Meta Model in AdminUI:

Add the column names and data types from the dataset into the Create Meta Model page and then click submit. Note that you do not add the as_of_date column, as that will be added automatically.

Step 3: Creating Analytical Model

The data flow from the source i.e. google drive to directly to BDL is achieved using analytical model. Pre-processing of data is also done using analytical model.

Step 4: Running the Analytical Workloads

Running the analytical workload for INFLATION application will load the data directly from your local to the BDL fact db.

Step 6: Importing Database and Dataset into Superset and creating a dashboard for the charts

Once you are in Superset, select the Datasets option from the Data dropdown in the top menu. From there, select the add Dataset option. Set the Database to Apache Hive, select your database from Schema, and select which table you would like to add. Superset will only allow you to add one table at a time, but you can add as many tables as you want one by one.

(see this page for further reference)

A Datetime column is supposed to be added into csv dataset file to get the time series visualization for Superset. Here, for Inflation application we have created "year_new" column which is the copy of "year1" column, but have just added "yyyy-01-01" to get datetime column. And then run the analytical workloads again.
Initially while adding that column it's datatype can be String and then later on in Superset, click on edit symbol beside your dataset name and then under CALCULATED COLUMNS you need to enter the SQL Query "from_unixtime(unix_timestamp(year_new, 'yyyy-MM-dd'))" and select the datatype as DATETIME, click on Save. To plot the time series graph it is necessary to have a Datetime column and this can be done by following the document of /wiki/spaces/BIGANALYTI/pages/1147181.

Example 2: Predicting Wealth Inequalities between Black, White and Hispanic people.

Objective

• Predicting Wealth Inequalities between Black, White and Hispanic people for the upcoming years, using analytical model we are training the machine learning model on the given dataset and dataset has target column as mean_net_worth.

Overview

Wealth Inequalities dataset comprises of every 3 years of the data from the year 1989 to 2019, representing the year, race, mean and median of income, savings, debts, investments and net worth.

Classifying the Application:

Based on the data technology used and the business use, the application is classified as below:

Step 1: Setting up the application

Now that the business use and classification of the application is established, the application can be created using the UI. In AdminUI, set up the application by going to the Set-up Application tab, select Create New, and filling out the file structure. ce weSince we have just one layer file system we will have just one entity in it.

Step 2: Creating Meta Model

We can now set up the Meta Model in AdminUI:

Add the column names and data types from the teams dataset into the Create Meta Model page and then click submit. Note that you do not add the as_of_date column, as that will be added automatically.

Step 3: Creating Ingest Model

Next, create the ingest model in AdminUI. The first part will be defining which processors to use from SDL to FDL. These are the SDL-FDL processors we select for our Teams data. You can refer to this page to understand what each of the processors do.

Step 4: Running Workloads

Once we submit the Ingest Model, we can run the workloads under the Batch Management/Run Workloads page.

First, run the SDL-FDL workload. This will apply the processors we selected in the Ingest Model for the SDL to FDL layer. You can see if the workload ran correctly by going to the FDL/Stage directory in HDFS.

Once you confirm in HDFS that the SDL-FDL workload ran correctly, run the FDL-BDL workload next. This will apply the processors we selected in the Ingest Model for the FDL to BDL layer. You can see if the workload ran correctly by going to the BDL/Fact directory in HDFS.

Step 5: Creating Analytical Model

The data is taken from the BDL fact db and is used for training the ML model onto it and making predictions on the target column using analytical model.

Step 5: Running the Analytical Workloads

Running the analytical workload for INFLATION application will load the data directly from your local to the BDL fact db.

Step 6: Importing Database and Dataset into Superset and creating a dashboard for the charts

Once you are in Superset, select the Datasets option from the Data dropdown in the top menu. From there, select the add Dataset option. Set the Database to Apache Hive, select your database from Schema, and select which table you would like to add. Superset will only allow you to add one table at a time, but you can add as many tables as you want one by one.

(see this page for further reference)

A Datetime column is supposed to be added into csv dataset file to get the time series visualization for Superset. Here, for Wealth Inequalities application we have created "year_new" column which is the copy of "year1" column, but have just added "yyyy-01-01" to get datetime column. And then run the analytical workloads again.
Initially while adding that column it's datatype can be String and then later on in Superset, click on edit symbol beside your dataset name and then under CALCULATED COLUMNS you need to enter the SQL Query "from_unixtime(unix_timestamp(year_new, 'yyyy-MM-dd'))" and select the datatype as DATETIME, click on Save. To plot the time series graph it is necessary to have a Datetime column and this can be done by following the document of FAQs.

Copyright © Sprngy. All rights reserved. Not to be reproduced or distributed without express written consent.