...
The Meta Model can be built using the Meta Model UI or CSV format. Please consult the product documentation for sequencing.
What type of information can be captured in Meta Model?
...
Data set Description - The module, entity, and description of the data set.
Attribute Description - Name, Description, Format, and Length. This can be captured at both the data model and BI model levels.
Attribute Security - Masking requirement and masking information.Attribute Transforms - Transform rule that needs to be applied at the inbound and outbound. By default, trim is applied to all categorical attributes.
Attribute Filtering - Whether the attribute should be part of the data filtering process and if so what rule.
Attribute IQM - Whether the attribute should be utilized in IQM match. For more information on IQM, please refer to IQM FAQs and BAPCore Sprngy documentation.
Attribute EDA - Whether the attribute should be utilized in EDA analysis.
Attribute Calendar - Whether the attribute should be utilized in calendar join operations.
Attribute Validation - Whether the attribute should be part of the data validation process and if so what rule.
Attribute Quality - Whether the attribute should be part of the data validation process and if so what rule.
...
Meta models are stored within the Meta folder of the core data lake (SPRINGYCORESPRNGYPlatform/Meta/BDL/Fact).
How is the storage and retrieval of Meta Models?
Meta models are stored within the Meta folder of the core data lake using the springycore sprngy API. They can be retrieved using the springycore sprngy API as well.
How is the life cycle of the Meta Model managed?
The most current version of Meta Model is stored in Current folder (SPRINGYCORESPRNGYPlatform/Meta/BDL/Fact/Current). The archived version is stored in Archive folder (SPRINGYCORESPRNGYPlatform/Meta/BDL/Fact/Archive). When a new version is pushed, the current version is moved to the archive folder.
...
Every time an metamodel is used for any data movement, the definition of the meta model is stored in the track folder (SPRINGYCORESPRNGYPlatform/Meta/BDL/Fact/Track) along with the batch_id of the data movement process.
...
The parent meta model coalesces with the entity meta model. The coalesced (entity meta model and parent entity meta model) are stored in the track folder (SPRINGYCORESPRNGYPlatform/Meta/BDL/Fact/Track) along with the batch_id of the data movement process. None of the attributes of the parent entity model are changed when coalesced with the child entity model meta model. this helps in troubleshooting if all parent entity models are appropriately applied to the child entity model for nesting processes.
...
Transform rules are SQL queries you can write in order to filter data by a variable, change certain values in a column, and apply the other functions that are in the Spark SQL library. We can write the transform rules in the meta model by clicking the edit icon next to the column name and going to the Rules page.
...
...
In the Entity Attribute Transform Rule Inbound line, write the SQL query that defines the transformation you are trying to make. In the example above, we are in the NBA Statistics application, and want to rename all occurrences of ‘Philadelphia Sixers’ in the team column to ‘Philadelphia 76ers’. We can write the query “case when team in ('Philadelphia Sixers', 'Philadelphia 76ers') then 'Philadelphia 76ers' else team end as team” as shown above for this transform rule.
...
In the same NBA Statistics application, another change we have to make to the dataset is removing all periods and commas from players' names. To do this, we click on the edit icon next to the player_name column, and under the Transform Rule Inbound, define the query “replace(replace(player_name, '.', ''), ',', '') as player_name”.
Here are a couple things to keep in mind when writing your transform rules:
...
The transform rules that you define will be executed in the SDL-FDL workload when you define the TRANSFORM_RECORDS_PROCESSOR based on the transformation preferences defined in the Ingest Model.