This document details steps to set up new applications using Sprngy Admin UI, defining models, and running workloads.

Demo Case: Twitter

Example 1: Determine user traffic in the USA

Objective - Build a chart that describes Twitter user traffic in the USA

Overview

Accessing twitter data using the Twitter developer account to gather information on tweets sent as per the geo-location. This search can be filtered by a hashtag, date and time, or geographical area. Here, we will be analyzing the number of tweets sent across the USA. We will be using “rtweet“ library to access the data and SprngyBI to create analytics over it.

Classifying the Application

Based on the data technology used and the business use, the application is classified as below:

Quality of Data

Data Lake

Data Lake is a centralized repository to store large amount of raw data

Data Lakehouse / Data Warehouse

Data Lakehouse combines concepts of Data Lake and Datawarehouse providing large storage capacity combined with data management features

Curated

Curating data involves creating or preparing data to make it usable for business analysis.

Correlated

Correlated data means running Algorithms to discover patterns and relationships within the data.

✔

Normalized

Normalizing data involves structuring the data to enable rapid access.

Analyze

Data Analysis involves identifying useful information to support decision-making, often using visualizations

✔

Modelling

This involves building statistical models and testing those.

Step 1: Setting up the Application

Now that the business use and classification of the application are established, the application can be created using the UI. In AdminUI, set up the application by going to the Set-up Application tab, select Create New, and filling out the file structure. Since we have just one layer file system we will have just one entity in it.

Step 2: Create Analytic Model

Since we are accessing data using the Tweeter API we can use Analytic Model directly to load, correlate and analyze the data.

A Sample Analytic Model is as follows:

Module Name

Processor Name

Pipeline Sequence

Variable Name

Assign Type

Parameter

Tweeter

tweeter_data

1

token

CREATE_VAR

rtweet::create_token(app = "BA_Demo123", consumer_key = 'pS10oUlc2s4Y70jX2XFXUGniL', consumer_secret = 'Ru2IVaCNyZsUM7UVHXZ0QFvma7BRrjL3J5xXvmeaqneOleT1q6', access_token = '1546529453131587584-No2NRwnvjWmXpsTDS71yjvTi23r6NQ', access_secret = '4lP01ELVNvqaF5kPMyJM80KwkHoFIxID0Az7jKEVZKuXw')

Tweeter

tweeter_data

2

data

CREATE_VAR

rtweet::search_tweets("lang:en", geocode = lookup_coords("usa"), n = 10000, token = token)

Tweeter

tweeter_data

3

data_lat_lng

CREATE_VAR

rtweet::lat_lng(data)

Tweeter

tweeter_data

4

res

CREATE_VAR

data_lat_lng %>% dataops.overwritedata('/BigAnalytixsPlatform/Tweeter/tweeter_data/BAL','data')

This would fetch the tweeter data, add latitude and longitude to it and save it to the given location.

Step 3: Create Data Table in Hive

In the utilityscripts folder, use the create_hive_ddl_using_spark_df.R script to generate the Hive SQL statement needed to create a data table in Hive. Once you run the script, a .hql file will be created; open this file and copy the statement that was generated. Run it in the hive terminal.

Step 4: Importing Database and Dataset into Superset

After adding the database and dataset (as shown in above screenshots), we can create charts for that dataset. To create charts go to the Chart page from the top menu, then pick the dataset that we just add and select the chart type. Here, we are selecting “deck.gl Scatterplot“ this requires latitude and longitude columns.

Copyright © Sprngy. All rights reserved. Not to be reproduced or distributed without express written consent.