SDL to FDL Process FAQs

What is SDL?

SDL stands for Staging Data Lake. For more information on SDL, please visit instructional videos in the documentation.

What is FDL?

FDL stands for Fast Data Lake. For more information on FDL, please visit instructional videos in the documentation.

What are the modes supported for data ingestion from SDL to FDL?

Two modes are supported. End of the day mode which overwrites the target folder in FDL i.e. FDL/Stage and intra day mode which appends to FDL/Stage. End of the day mode does not perform delta processing except if there is more than one file to process. Intra day mode performs delta processing.

Intra day mode is required for specific use cases that may require customization which is why it is not available by default. Please write to support@sprngy.com to discuss specific use cases and for a quote.

What happens if I have more than one file in SDL/Land?

All files would be processed together while SDL to FDL pipe runs.

I have placed data in Land, but SDL to FDL pipe fails to extract the data from land and process it per my ingest model?

Ensure that metamodel for the entity and the data in SDL/Land for that entity are compatible. Please consult Metamodel FAQs and springycore API documentation on how to.

I have placed data in Land, but SDL to FDL pipe exits with message that no files to process from land?

SDL to FDL pipe always checks DIQM to see if filename in land has been processed successfully. If it finds an entry in DIQM, it would not process the file in Land. In that case, you should change the file name in land. However while processing this file, if SDL to FDL pipe switches to intraday mode, it will perform delta processing with existing data in FDL/Stage.

Would my file stay in Land after successful processing from SDL to FDL?

No. The default configuration is to move the file to Ingested folder (in SDL).

Do the contents of Land files change while moving from Land to Ingested?

No, the file is moved as is.

What if I would like all of my data back in the format that I had pushed into platform?

You should be able to find all of your files in Land and/or Ingested based on whether they were processed.

Is there an audit mechanism available for data movements in general?

Yes. DIQM and DPQM provide that mechanism. For more information, please consult FAQs for DIQM and DPQM.