📘

Timeline events versus data events

Timeline events are meant to deliver messages to your end users, such as a credit-limit notification in a mobile banking app. These are not the same as data events.

The Pismo API data platform consumes, treats, and validates events that occur during the course of normal processing (account actions, transactions, payments, auditing, and so on). The data from these events is delivered to you in real-time or via files that you can download.

Delivery works on an at-least-once delivery guarantee. This means that the platform keeps attempting to deliver a message until at least one attempt is successful.

🚧

Idempotent processing is recommended because the at-least-once delivery guarantee can result in duplicate events.

Data events configuration

The major event services that Pismo integrates with are:

• AWS – Simple Notification Service (SNS) and EventBridge
• GCP – Publish/Subscribe
• Azure – Azure Service Bus
• Confluent connector

Pismo APIs generate data event notifications to your cloud storage provider. In addition, for files, Pismo provides delivery via SFTP that does not involve cloud storage. Both delivery methods require some initial configuration and setup. For more information, see:

• Setting up real-time event delivery
• Setting up event file delivery

Understanding Pismo event data

Pismo provides JSON schemas to document and validate events and their possible combinations, payloads, and event types. Event payload data is formatted in JavaScript Object Notation (JSON). For more information, see Event schemas below.

{
    "event_id": "252212bf-cde2-43f9-b28f-33ea2b452b32",
    "domain": "account",
    "event_type": "customer-creation-1",
    "schema_version": 1,
    "org_id": "<Your Org ID>",
    "cid": "7c7737f0-53d4-9b49-9833-e4c47483634e",
    "timestamp": "2019-07-20T19:03:00.000Z",
    "data": {<Varies according to domain, event_type, and schema_version>}
}

• event_id - Uniquely identifies each event.

• domain - Represents a Pismo application domain. Examples include account, authorization, cards, and so on. You can see a complete domain list in the navigation tab when you view Pismo's JSON schemas in HTML.

• event_type - Represents a processing action within a domain. For example, for account, these could include customer-creation-1, limit-change-1, and status-change-1. The number at the end represents the schema version. You can see a complete event type list in the navigation tab when you view Pismo's JSON schemas in HTML.

• schema_version - Represents the event's JSON schema version. The schema version number appears at the end of the event_type field, for example, limit-change-1.

• cid - The correlation identifier field is used to link related API requests and events. For example, if a user makes an authorization request with 10 installments, this generates 1 authorization event and 10 transaction events all containing the same correlation ID.

• timestamp - Indicates when the event occurred in ISO-8601 format.

• data - Contains the event data itself. This content varies according to domain, event_type and schema_version. See Pismo's JSON schemas in HTML for details on event data.

New event notifications

Event notifications and data are sent in real-time and via files. For event file delivery, event data is batched for 5 minutes or until it reaches 10MB in size and then saved as a new AWS S3 bucket object to the /main_stream/ path. When this occurs, a new bucket object event is generated and sent with data containing the object's path, size, and name.

As with all events, the dataplatform/new_bucket_object event generates a new event as well as a real-time notification. The new event is saved as part of the next bucket object.

Sample new file event payload

{
  "org_id": "TN-f770039e-aabd-498c-9134-9fde4cac4d51", 
  "cid": "d426f496-9fae-491e-a8b0-257eb6b8db16", 
  "event_id": "e03e7919-486e-4552-a7da-ae0b5a2b9a1e", 
  "domain": "dataplatform", 
  "event_type": "new_bucket_object", 
  "schema_version": 1, 
  "timestamp": "2020-04-11T19:22:37.946Z", 
  "data": {
    "object_size": 538, 
    "object_name": "main_stream/2020/04/11/19/dataplatform-tn-f770039e-aabd-498c-9134-9fde4cac4d51-1-2020-06-19-19-17-42-45baae47-28d6-4ef9-9f31-9cd1891d469b", 
    "bucket_name": "pismo-dataplatform-tn-f770039e-aabd-498c-9134-9fde4cac4d51"
  }
}

Although this object doesn't have a .json extension, it contains JSON event data and you can download it directly.

AWS CLI command to download bucket object:

aws s3 cp s3://<your bucket name>/main_stream/YYYY/MM/DD/HH/<object name> <download destination path>

JSON schema change notifications

Whenever a JSON schema is created, modified, or deleted, the Pismo platform generates a JSON schema change event. This event contains details about the change, such as the type of change, name of the affected schema, and the previous and new schema version numbers.

Event schemas

Every event generated by an API has an associated contract. Contracts take the form of JSON schemas, which you can use for validation and to understand event payload data. The fields common to all events comprise the base contract. What distinguishes one event from another is its data field (payload).

You can view available event schemas in these ways:

• JSON schemas in HTML
• JSON schemas in S3 bucket

JSON schemas in HTML

The Pismo event schema reference guide provides a user-friendly resource for understanding JSON event schemas. While this HTML is useful to view and understand events, for integration, you should use the S3 bucket JSON schemas.

JSON schemas in S3 bucket

If you have an Amazon Web Services (AWS) cloud account, you can retrieve JSON schemas from Pismo's AWS S3 bucket. You will need to periodically check for updates, as no update notification is sent.

Bucket access

In order to access the JSON Schemas S3 Bucket, you have to complete the AWS Files Integration for your org. There, the bucket access is described and can be tested using the testing script provided in the Testing your AWS Integration section.

You can also obtain S3 bucket JSON schemas using AWS SDKs. To know what SDKs are available, see Tools to Build on AWS.

Bucket path structure

The Pismo documentation S3 bucket has the following path structure:

• json_schemas/ - Parent folder for Pismo JSON schemas.

• json_schemas/<domain>/ - These folders represent Pismo application domains such as account, cards, authorization, and so on, and correspond to an event's domain field.

• json_schemas/<domain>/<event type>.json - Within each domain folder are the schema JSON files, named according to an event's event_type field. For example: json_schemas/account/customer-creation-1.json. Note that the schema version is part of the event_type field.

AWS CLI command to list all schemas:

aws s3 ls --recursive s3://pismo-data-dev-dataplatform-docs/json_schemas

AWS CLI command to list all schemas from a specific domain (account):

aws s3 ls --recursive s3://pismo-data-dev-dataplatform-docs/json_schemas/account

Breaking changes to event schemas

Breaking changes to event schemas are those that require you to update your implementation to avoid disruption. Changes of this type are rare, and Pismo is committed to communicating about such changes frequently and well in advance. Pismo considers changes like the following to be breaking changes:

• Renaming or removing attributes
• Any change to the format of an attribute
• Changing data types
• Changing optional attributes to required
• Tighter constraints (such as adding a limit to a value)

In contrast, non-breaking changes to event schemas typically introduce new features or improve performance, but do not disrupt your current implementation. As with Pismo APIs, these changes shouldn't require you to do anything to keep things working correctly. Non-breaking changes include:

• Adding events
• Adding optional fields to an event
• Changing required field to optional
• Looser constraints (such as expanding the range of allowed values)

Example schema-change event payloads

{
    "event_id": "8004c09f-9480-42a3-9211-6fa62e026806",
    "cid": "1acb8e5f-f242-423e-be32-593650d81d9e",
    "org_id": "TN-1c058bed-991d-42e6-a60c-6b3b9b16dbea",
    "domain": "dataplatform",
    "event_type": "json_schema_change",
    "schema_version": "1",
    "timestamp": "2023-06-22T00:09:06Z",
    "data": {
        "change_type": "INSERT",
        "schema_name": "domain/event_type-1",
        "schema_domain": "domain",
        "schema_event_type": "event_type",
        "schema_version": "1",
        "new_schema": "{\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"title\":\"Example JSON Schema\",\"description\":\"Example new description\",\"required\":[\"some_field\"],\"properties\":{\"some_field\":{\"type\":\"string\",\"title\":\"Some field title\"}}}"
    }
}

{
    "event_id": "e2b3c97f-6ae1-41ac-817a-f6f92166d7dd",
    "cid": "8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c",
    "org_id": "TN-9fc8e510-a27c-4b3b-b9a3-bf65f00256a8",
    "domain": "dataplatform",
    "event_type": "json_schema_change",
    "schema_version": "1",
    "timestamp": "2023-06-22T22:69:36Z",
    "data": {
        "change_type": "MODIFY",
        "schema_name": "domain/event_type-1",
        "schema_domain": "domain",
        "schema_event_type": "event_type",
        "schema_version": "1",
        "old_schema": "{\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"title\":\"Example JSON Schema\",\"description\":\"Example old description\",\"required\":[\"some_field\"],\"properties\":{\"some_field\":{\"type\":\"string\",\"title\":\"Some field title\"}}}",
        "new_schema": "{\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"title\":\"Example JSON Schema\",\"description\":\"Example new description\",\"required\":[\"some_field\"],\"properties\":{\"some_field\":{\"type\":\"string\",\"title\":\"Some field title\"}}}"
    }
}

{
    "event_id": "9fc8e510-a27c-4b3b-b9a3-bf65f00256a8",
    "cid": "1acb8e5f-f242-423e-be32-593650d81d9e",
    "org_id": "TN-9dg84c90-664e-95z7-b94k-9h3l8g76jdud",
    "domain": "dataplatform",
    "event_type": "json_schema_change",
    "schema_version": "1",
    "timestamp": "2023-06-22T11:39:06Z",
    "data": {
        "change_type": "REMOVE",
        "schema_name": "domain/event_type-1",
        "schema_domain": "domain",
        "schema_event_type": "event_type",
        "schema_version": "1",
        "old_schema": "{\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"title\":\"Example JSON Schema\",\"description\":\"Example old description\",\"required\":[\"some_field\"],\"properties\":{\"some_field\":{\"type\":\"string\",\"title\":\"Some field title\"}}}"
    }
}