Introduction It can be overwhelming when trying to set-up or develop a monitoring system for your data-pipelining / processing jobs. However, it is an important aspect of any recurring or scheduled data-pipeline.
Ideally, your process can be broken down into single steps. Commonly, they can be broken down into Extraction steps (pulling down or reading in data), Transformation steps (pre-processing or manipulations of the data for the desired output), and Load steps (loading the now processed data into a database, or exporting flat files to a shared folder).
