DIQM FAQs

What is DIQM?

DIQM stands for data integration quality module.

What is the purpose of DIQM?

DIQM consists of information per-file basis and contains attributes like - source folder/, source rows, null rows, duplicate rows, imputed rows, distinct rows, error rows, valid rows, archive rows, fact rows, total rows, target information (target folder) and duration.

How is DIQM useful?

DIQM tracks each file for its source records.

Where is DIQM stored?

DIQM is stored in the DIQM folder in the core data lake (SPRNGYPlatform/DIQM/BDL/Fact).

How can DIQM be accessed?

DIQM dashboard is included in SprngyBI.

How do the DIQM records counts match?

For SDL to FDL, the source rows would be equal to the sum of the archive, fact and error provided no nulls or duplicates are dropped.

For FDL to BDL, the source rows would be equal to the sum of archive and fact.

If I put new data in Land but a file containing the data has the same name as one of the entries in DIQM, would SDL to FDL pipe process that file?

It depends on whether that file (with the same file name) was successfully processed in the past. SDL to FDL pipe always checks if the file name in Land has a corresponding entry in DIQM with statuses ‘PROCESSED’ and ‘SUCCESS’.