When a job executes, it searches for records that meet filter conditions (usually a Data Lake query). If any are found, it writes to the Amazon Web Services (AWS) S3 bucket for your organization. Files are saved in AWS parquet format.
Files are written and saved according to type (daily, monthly, or full) and date. For example,
s3://pismo-dataplatform-tn -55317847-57cd-45a3-8aed-a8dadd63cc6b/reports/job_name/<type>/<date values>.
Files and paths are generated on the following basis:
Daily - In this case, the type is
dailyand the date partitioning values are
/year=YYYY/month=MM/day=DD/. For example,
Monthly - At your request, or when the need for a periodic reprocessing is identified, a job executes for a closed month of data. In this case, the type is
monthlyand the date partitioning is
year=YYYY/month=MM. For example,
Full - You can generate a complete file, taking into account all past job data without a date filter. In this case, the type is
fulland the partitioning corresponds to the file's generation date. For example.
Target date versus generation date
It's important to note that date partition values correspond to the last available data date, not when the job is executed. So, for example, if a daily job is run on 02/04/2020, the date partition values correspond to 02/03/2020.
Updated 5 months ago