Sprngy - Creating Demo Application using Connectors

This document details steps to set up new applications using Sprngy Admin UI, defining models, and running workloads.

Use Case Category: Social Media

Demo Application 1: Determine twitter user traffic in the USA

Objective - Build a chart that describes Twitter user traffic in the USA

Overview

Accessing twitter data using the Twitter developer account to gather information on tweets sent as per the geo-location. This search can be filtered by a hashtag, date and time, or geographical area. Here, we will be analyzing the number of tweets sent across the USA. We will be using “rtweet“ library to access the data and SprngyBI to create analytics over it.

Classifying the Application

Based on the data technology used and the business use, the application is classified as below:

Quality of Data	Data Lake Data Lake is a centralized repository to store large amount of raw data	Data Lakehouse / Data Warehouse Data Lakehouse combines concepts of Data Lake and Datawarehouse providing large storage capacity combined with data management features
Curated Curating data involves creating or preparing data to make it usable for business analysis.
Correlated Correlated data means running Algorithms to discover patterns and relationships within the data.	✔
Normalized Normalizing data involves structuring the data to enable rapid access.
Analyze Data Analysis involves identifying useful information to support decision-making, often using visualizations	✔
Modelling This involves building statistical models and testing those.

Step 1: Setting up the Application

Now that the business use and classification of the application are established, the application can be created using the UI. In AdminUI, set up the application by going to the Application Configuration page, select Create New, and filling out the file structure. Since we have just one layer data we will have just one entity in it.

Step 2: Create Analytic Model

Since we are accessing data using the Tweeter API we can use Analytic Model directly to load, correlate and analyze the data.

Analytic model can be specified using the AdminUI. A Sample Analytic Model is as follows:

Module Name	Processor Name	Pipeline Sequence	Variable Name	Assign Type	Parameter
Tweeter	tweeter_data	1	token	CREATE_VAR	rtweet::create_token(app = "BA_Demo123", consumer_key = 'pS10oUlc2s4Y70jX2XFXUGniL', consumer_secret = 'Ru2IVaCNyZsUM7UVHXZ0QFvma7BRrjL3J5xXvmeaqneOleT1q6', access_token = '1546529453131587584-No2NRwnvjWmXpsTDS71yjvTi23r6NQ', access_secret = '4lP01ELVNvqaF5kPMyJM80KwkHoFIxID0Az7jKEVZKuXw')
Tweeter	tweeter_data	2	data	CREATE_VAR	rtweet::search_tweets("lang:en", geocode = lookup_coords("usa"), n = 10000, token = token)
Tweeter	tweeter_data	3	data_lat_lng	CREATE_VAR	rtweet::lat_lng(data)
Tweeter	tweeter_data	4	res	CREATE_VAR	data_lat_lng %>% dataops.overwritedata('/BigAnalytixsPlatform/Tweeter/tweeter_data/BAL','data')

This would fetch the tweeter data, add latitude and longitude to it and save it to the given location.

Step 3: Create Data Table in Hive

For now if you run an analytic model please contact to support@sprngy.com to get the generated data access in SpringyBI.

Step 4: Importing Database and Dataset into Superset

After adding the database and dataset (as shown in above screenshots), we can create charts for that dataset. To create charts go to the Chart page from the top menu, then pick the dataset that we just add and select the chart type. Here, we are selecting “deck.gl Scatterplot“ this requires latitude and longitude columns.