Scheduled reports

Pismo can run batch jobs to generate report files. There are two types of scheduled report files available.

• Base scheduled reports: General information about the performance of your programs
• Regulatory scheduled reports: Information that you might need to provide to regulatory agencies

Report files are written to your organization's dedicated S3 bucket. To start receiving report files, contact your Pismo representative to configure access to your S3 bucket.

📘

Most Pismo customers use Amazon Web Services (AWS) S3 buckets as their cloud storage containers, so this section assumes you're using AWS. However, Pismo does support other cloud services, such as Google Cloud Platform (GCP) and Microsoft Azure. Contact your Pismo representative for information about using a cloud service other than AWS.

The jobs that generate report files all run once a day at 5:00 AM UTC. When a job executes, it searches for records that meet filter conditions. If any are found, it writes the data to a file that it creates in your cloud storage container. Files are saved in Parquet format.

📘

In most cases, a report file is not generated if no records are found that match the filter conditions.

Files are written and saved according to type (daily, monthly, or full) and date. The full type is explained below. The URL takes the form [path to S3 bucket]/reports/[job_name]/[type]/[date values]/[filename].parquet, where [filename] represents a Spark hash.

For example:
s3://pismo-dataplatform-tn -55317847-57cd-45a3-8aed-a8dadd63cc6b/reports/accounting_events/type=daily/year=2020/month=1/day=10/[filename].parquet
In this example, the value for job_name is accounting_events. You can find the value for job_name in the documentation for the specific report file.

Files and paths are generated on the following basis:

• Daily: In this case, the type is daily and the date partitioning values are /year=YYYY/month=MM/day=DD/ . For example, .../reports/accounting_events/daily/2020/1/10/[filename].parquet

• Monthly: At your request, or when there is a need for periodic reprocessing, a job executes for a closed month of data. In this case, the type is monthly and the date partitioning is year=YYYY/month=MM. For example, .../reports/accounting_events/type=monthly/year=2020/month=1/[filename].parquet

• Full: You can generate a complete file, taking into account all past job data without a date filter. In this case, the type is full and the partitioning corresponds to the file's generation date. For example: .../reports/accounting_events/type=full/year=2020/month=1/day=10/[filename].parquet

🚧

The date partition values in the report correspond to the last date when the data was available, not the date when the job was executed. For example, for a daily job that runs on March 11, the date partition values correspond to March 10.