This document details steps to set up new applications using MySpringy Sprngy Admin UI, defining models, and running workloads.
Demo Case: Twitter
Example 1: Determine user traffic in the USA
Objective - Build a chart that describes Twitter user traffic in the USA
Overview
Accessing twitter data using the Twitter developer account to gather information on tweets sent as per the geo-location. This search can be filtered by a hashtag, date and time, or geographical area. Here, we will be analyzing the number of tweets sent across the USA. We will be using “rtweet“ library to access the data and springyBi to create analytics over it.
Classifying the Application
Based on the data technology used and the business use, the application is classified as below:
Quality of Data | Data Lake
| Data Lakehouse / Data Warehouse
| ||||
|---|---|---|---|---|---|---|
Curated
| ||||||
Correlated
| ✔ | |||||
Normalized
| ||||||
Analyze
| ✔ | |||||
Modelling
|
Step 1: Setting up the Application
Now that the business use and classification of the application are established, the application can be created using the UI. In AdminUI, set up the application by going to the Set-up Application tab, select Create New, and filling out the file structure. Since we have just one layer file system we will have just one entity in it.
...
Step 2: Create Analytic Model
Since we are accessing data using the Tweeter API we can use Analytic Model directly to load, correlate and analyze the data.
...
This would fetch the tweeter data, add latitude and longitude to it and save it to the given location.
Step 3: Create Data Table in Hive
In the utilityscripts folder, use the create_hive_ddl_using_spark_df.R script to generate the Hive SQL statement needed to create a data table in Hive. Once you run the script, a .hql file will be created; open this file and copy the statement that was generated. Run it in the hive terminal.
...
Step 4: Importing Database and Dataset into Superset
...
After adding the database and dataset (as shown in above screenshots), we can create charts for that dataset. To create charts go to the Chart page from the top menu, then pick the dataset that we just add and select the chart type. Here, we are selecting “deck.gl Scatterplot“ this requires latitude and longitude columns.
...