8. Robotic Data Automation Fabric
Artifact Snapshots Feature
8.1 What are RDAF Snapshots?
RDAF Snapshots are copies of certain RDAF artifacts stored inside the Minio folder snapshot artifacts (i.e., snapshot repository).
Each time a snapshot is created, the user must specify a unique snapshot name.
RDAF will take a copy of each artifact (pipelines, dashboards, etc.), compute the checksum, and store it in Mino. If another copy is already in the repository, it won't overwrite.
A manifest will be created for each snapshot, indicating which (snapshot) has which artifact version (identified by the checksum).
8.2 Snapshot Operations
8.2.1 Create Snapshot
-
It takes a copy of each artifact from the main minio location and computes the checksum. If there is already one artifact with that checksum, it is treated as unchanged; otherwise, it is called new.
-
The Manifest PStream (rda_system_snapshots_details) will have entries for each artifact, corresponding checksum, and location of contents in minio (snapshot repository).
-
The snapshot repository will have a copy even if the artifact is deleted.
-
If there are no changes to artifacts or changes minimal, creating a new snapshot does not consume any additional storage (except a few rows in PStream for manifest)|
8.2.2 Export Snapshot
- Using manifest pstream, it will download all artifacts from the artifact repository and create a
.tarfile. The.tarfile will contain amanifest.jsonwith details of all artifacts and the location of the full artifact in the.tarfile.
8.2.3 Import Snapshot
-
Snapshot
.tarcan be imported into any RDAF. Import snapshot only adds the file contents into the snapshot repository (& manifest stream). It does not update actual artifacts. -
Imported snapshots are different from locally created snapshots. Using the Compare operation, snapshots from two different systems or even the same system from two different timestamps can be compared.
8.2.4 Review Snapshots
-
Any two snapshots can be compared. For each artifact, a report with the following status will be shown.
a) Identical
b) Only in First
c) Only in Second
d) Different
-
There will be a UI for different artifacts to show diffs in HTML format.
8.2.5 Restore Snapshot
-
Each artifact must be manually restored.
-
Accidental restoration to the wrong Snapshot can bring down a production environment, and hence, this automated restore must be provided after careful consideration and implementation of proper workflows.
-
We should generate a Restore Document that shows what actions need to be performed to reach a certain snapshot level.
8.3 What Artifacts are Covered by Snapshots?
-
Dashboards
-
Dashboard Groups
-
Draft Pipelines (latest version)
-
Published Pipelines (latest version)
-
Formatting Templates
-
Persistent Stream (Definition Only)
-
Endpoints (Alerts, incidents, messages)
-
Mappings (Alerts, incidents, messages)
-
Service Blueprints
8.4 What Other Artifacts Should be Covered by Snapshots (planned for 4.0)?
-
Alert Correlation Policies
-
Alert Suppression Policies
-
Teams Configuration
-
FSM Models
-
Limited RDA Objects (some artifacts used in Dashboards)
-
ChatHelper Policies
-
Bulkstats Configurations
8.5 What Artifacts Are Not in the Scope of Snapshots?
-
Datasets
-
PStream Data
8.6 Are Snapshots a replacement for Git Integration?
No. Snapshots are created on demand and stored on the same system. Exported snapshots can be saved outside the system. Git integration will automate and check every change in an external repository with user-level tracking. Snapshots are meant to capture the entire state of a system periodically or before and after upgrades.
8.7 Snapshot Management UI
Go to Home -> Configuration -> RDA Administration -> Snapshots
User can find two subpages under Snapshots
8.7.1 Snapshot list
- Displays a list of all snapshots, including those created locally and those imported. Each snapshot offers the following actions: View Details, Export, Review, and Delete.
8.7.1.1 Create
- Snapshots can be created by selecting the Create option. They are classified as the
local_snapshottype.
8.7.1.2 Import
- Clicking on the import option allows the user to import an exported tar file. Once imported, the snapshots are listed in the Snapshot list with their snapshot_type labeled as 'imported_snapshot'.
Note
Import will support only .tar files.
8.7.1.3 View Details
- Click on View Details, it will show the details of all artifacts that are added for that snapshot
-
The View Details dashboard features a Snapshot Manifest table that displays the details of each artifact within a snapshot. Each row includes a "View" action, which, when clicked, reveals the specific details of the artifact. For instance, clicking "View" on a dashboard will display the dashboard's definition.
-
Additionally, if a new snapshot is created and View Details is clicked, the system will compare it with the previously created snapshot. This comparison will highlight whether an artifact has been Modified, Added, Deleted, or Remains unchanged.
-
For example, If the user adds a new stream(Demo_One) and create a snapshot(Test Stream), the comparison will indicate the status of this new artifact.
8.7.1.4 Export
- The Export option allows user to export existing snapshots. When user exports a snapshot, a tar file is generated and saved in the My Downloads folder. User can then use this exported tar file to import the snapshot into a different setup.
User needs to scroll down and click on Settings below as shown in the screenshot below
After clicking on the Settings the user can see My Downloads and when user clicks on My Downloads it takes the user to the downloaded section
Here in the below screenshot the user can find the Downloaded .tar File
-
The exported file includes a manifest JSON file and a snapshots-artifacts folder.
-
The manifest file provides information on the total number of artifacts and details on which artifacts have changed, remained unchanged, were deleted, or are new.
-
The snapshots-artifacts folder organizes subfolders for each artifact type, as illustrated below.
8.7.1.5 Review
Using the review action, The user can compare two snapshots, resulting in an HTML file that highlights the artifacts that are Unchanged, Modified, Deleted, or Added between the snapshots. This review action generates a tar.gz file, which is saved under My Downloads. When extracted, this tar file contains an index.html file that provides a summary of the comparison.
If we drill down to any particular Unchanged, Deleted, Added, and Changed. If the user clicks on artifact names - will take us to the file
8.7.1.6 Delete
- Deletes the snapshot from the Snapshot list
8.7.2 Snapshot Manifests
- This page provides comprehensive information about all artifacts across every snapshot.
The View option is visible to the user. To access the JSON file, select that option.
8.8 Rdac CLI Usage
Commands to manage RDA Artifact Snapshots
Following are valid sub-commands for snapshot:
list List RDA artifact snapshots
create Create a new snapshot for RDA artifacts
export Export RDA artifact snapshot into a tar.gz file
import Import a snapshot from tar.gz file into Snapshot repository
compare Compare two snapshots
review Compare two snapshots and generate an offline Change Review Report
8.8.1 Rdac Snapshot List
+----------------+-----------------------------------+-------------------+-------+-------------+
| Type | Name | Total Artifacts | New | Unchanged |
|----------------+-----------------------------------+-------------------+-------+-------------|
| local_snapshot | apri_15_2024_test2 | 361 | 0 | 361 |
| local_snapshot | apri_15_2024_test1 | 361 | 1 | 360 |
| local_snapshot | with_snapshot_dashboards_april_16 | 361 | 5 | 356 |
| local_snapshot | snapshot_4 | 357 | 23 | 334 |
| local_snapshot | apr_16_2024_post_change3 | 367 | 1 | 366 |
| local_snapshot | apr_16_2024_post_change2 | 366 | 115 | 251 |
+----------------+-----------------------------------+-------------------+-------+-------------+
usage: snapshot [-h] [--json] [--pattern PATTERN]
optional arguments:
-h, --help show this help message and exit
--json Print detailed information in JSON format instead of
tabular format
--pattern PATTERN Snapshot name pattern (regex)
{
"snapshot_name": "cli_test",
"description": "test",
"snapshot_type": "local_snapshot",
"username": "",
"timestamp": "2024-05-29T10:28:34.639195",
"start_time": "2024-05-29T10:28:34.639195",
"status": "Success",
"Dashboard_total": 167,
"Dashboard_new": 0,
"Dashboard_unchanged": 167,
"DashboardGroup_total": 8,
"DashboardGroup_new": 0,
"DashboardGroup_unchanged": 8,
"PublishedPipeline_total": 37,
"PublishedPipeline_new": 0,
"PublishedPipeline_unchanged": 37,
"DraftPipeline_total": 41,
"DraftPipeline_new": 0,
"DraftPipeline_unchanged": 41,
"Blueprint_total": 11,
"Blueprint_new": 0,
"Blueprint_unchanged": 11,
"FormattingTemplate_total": 2,
"FormattingTemplate_new": 0,
"FormattingTemplate_unchanged": 2,
"PStream_total": 96,
"PStream_new": 0,
"PStream_unchanged": 96,
"Endpoint_total": 6,
"Endpoint_new": 0,
"Endpoint_unchanged": 6,
"Mapping_total": 4,
"Mapping_new": 0,
"Mapping_unchanged": 4,
"end_time": "2024-05-29T10:28:40.774520",
"total_artifacts": 372,
"new_artifacts": 0,
"unchanged_artifacts": 372
}, ...
8.8.2 Rdac Snapshot Create
usage: snapshot [-h] --name SNAPSHOT_NAME --description DESCRIPTION
optional arguments:
-h, --help show this help message and exit
--name SNAPSHOT_NAME Snapshot name. Must be unique.
--description DESCRIPTION
Snapshot Description
Check rdac snapshot list to verify snapshot created
+----------------+------------------+-------------------+-------+-------------+
| Type | Name | Total Artifacts | New | Unchanged |
|----------------+------------------+-------------------+-------+-------------|
| local_snapshot | may_23_2024 | 400 | 42 | 358 |
+----------------+------------------+-------------------+-------+-------------+
8.8.3 Rdac Snapshot Export
usage: snapshot [-h] --name SNAPSHOT_NAME --file FILENAME
optional arguments:
-h, --help show this help message and exit
--name SNAPSHOT_NAME Snapshot to be exported
--file FILENAME Output file (tar.gz)
==== "Example Output"
2024-04-15:19:12:58 [11765] INFO collector Fetching stream rda_system_snapshots_details: offset 0, totalResults=?, limit=0
2024-04-15:19:12:59 [11765] INFO snapshot_manager Exporting 300 files
2024-04-15:19:13:01 [11765] INFO snapshot_manager exported 10 files ...
2024-04-15:19:13:02 [11765] INFO snapshot_manager exported 20 files ...
2024-04-15:19:13:04 [11765] INFO snapshot_manager exported 30 files ...
2024-04-15:19:13:06 [11765] INFO snapshot_manager exported 40 files ...
2024-04-15:19:13:07 [11765] INFO snapshot_manager exported 50 files ...
2024-04-15:19:13:09 [11765] INFO snapshot_manager exported 60 files ...
2024-04-15:19:13:10 [11765] INFO snapshot_manager exported 70 files ...
2024-04-15:19:13:12 [11765] INFO snapshot_manager exported 80 files ...
2024-04-15:19:13:14 [11765] INFO snapshot_manager exported 90 files ...
2024-04-15:19:13:15 [11765] INFO snapshot_manager exported 100 files ...
2024-04-15:19:13:17 [11765] INFO snapshot_manager exported 110 files ...
2024-04-15:19:13:19 [11765] INFO snapshot_manager exported 120 files ...
2024-04-15:19:13:20 [11765] INFO snapshot_manager exported 130 files ...
2024-04-15:19:13:22 [11765] INFO snapshot_manager exported 140 files ...
2024-04-15:19:13:23 [11765] INFO snapshot_manager exported 150 files ...
2024-04-15:19:13:25 [11765] INFO snapshot_manager exported 160 files ...
2024-04-15:19:13:27 [11765] INFO snapshot_manager exported 170 files ...
2024-04-15:19:13:29 [11765] INFO snapshot_manager exported 180 files ...
2024-04-15:19:13:30 [11765] INFO snapshot_manager exported 190 files ...
2024-04-15:19:13:32 [11765] INFO snapshot_manager exported 200 files ...
2024-04-15:19:13:34 [11765] INFO snapshot_manager exported 210 files ...
2024-04-15:19:13:36 [11765] INFO snapshot_manager exported 220 files ...
2024-04-15:19:13:37 [11765] INFO snapshot_manager exported 230 files ...
2024-04-15:19:13:39 [11765] INFO snapshot_manager exported 240 files ...
2024-04-15:19:13:40 [11765] INFO snapshot_manager exported 250 files ...
2024-04-15:19:13:42 [11765] INFO snapshot_manager exported 260 files ...
2024-04-15:19:13:43 [11765] INFO snapshot_manager exported 270 files ...
2024-04-15:19:13:45 [11765] INFO snapshot_manager exported 280 files ...
2024-04-15:19:13:47 [11765] INFO snapshot_manager exported 290 files ...
2024-04-15:19:13:49 [11765] INFO snapshot_manager exported 300 files ...
2024-04-15:19:13:49 [11765] INFO snapshot_manager Creating tar.gz file /tmp/apri_15_2024_test2.tar.gz
2024-04-15:19:13:49 [11765] INFO snapshot_manager Export complete
Example manifest.json in exported Snapshot file
"artifacts": [
{
"artifact_type": "Blueprint",
"artifact_name": "sample blueprint (hourly)",
"checksum": "cb591bd5ff8c4626f23c09b9919a9178",
"size": 502,
"object_name": "snapshot-artifacts/Blueprint/cb591bd5ff8c4626f23c09b9919a9178/data.json",
"artifact_version_visibility": "unchanged",
"snapshot_name": "apri_15_2024_test2",
"snapshot_type": "local_snapshot",
"timestamp": "2024-04-15T16:57:52.417675"
},
{
"artifact_type": "Blueprint",
"artifact_name": "sample blueprint hourly",
"checksum": "c7c3ce304ff8d0744cabb5b08e7e89df",
"size": 502,
"object_name": "snapshot-artifacts/Blueprint/c7c3ce304ff8d0744cabb5b08e7e89df/data.json",
"artifact_version_visibility": "unchanged",
"snapshot_name": "apri_15_2024_test2",
"snapshot_type": "local_snapshot",
"timestamp": "2024-04-15T16:57:52.417675"
},
8.8.4 Rdac Snapshot Import
usage: snapshot [-h] --name SNAPSHOT_NAME --file FILENAME
optional arguments:
-h, --help show this help message and exit
--name SNAPSHOT_NAME New snapshot name. Must be unique
--file FILENAME Input snapshot file (tar.gz)
2024-05-23:03:50:17 [1] INFO snapshot_manager Importing 13 artifacts of type Blueprint
2024-05-23:03:50:17 [1] INFO snapshot_manager Importing 168 artifacts of type Dashboard
2024-05-23:03:50:18 [1] INFO snapshot_manager Importing 2 artifacts of type DashboardGroup
2024-05-23:03:50:18 [1] INFO snapshot_manager Importing 44 artifacts of type DraftPipeline
2024-05-23:03:50:18 [1] INFO snapshot_manager Importing 19 artifacts of type Endpoint
2024-05-23:03:50:18 [1] INFO snapshot_manager Importing 2 artifacts of type FormattingTemplate
2024-05-23:03:50:18 [1] INFO snapshot_manager Importing 5 artifacts of type Mapping
2024-05-23:03:50:18 [1] INFO snapshot_manager Importing 103 artifacts of type PStream
2024-05-23:03:50:19 [1] INFO snapshot_manager Importing 44 artifacts of type PublishedPipeline
2024-05-23:03:50:19 [1] INFO snapshot_manager Published details for snapshot import_may_23_2024, detail row count = 400
Check imported snapshot, using below command
+-------------------+--------------------+-------------------+-------+-------------+
| Type | Name | Total Artifacts | New | Unchanged |
|-------------------+--------------------+-------------------+-------+-------------|
| Imported_snapshot | import_may_23_2024 | 400 | 0 | 400 |
| local_snapshot | may_23_2024 | 400 | 42 | 358 |
| local_snapshot | shdng | 372 | 1 | 371 |
| local_snapshot | snapshot_12 | 372 | 0 | 372 |
| local_snapshot | test_more_spaces | 372 | 8 | 364 |
| local_snapshot | Test Stream | 372 | 8 | 364 |
+-------------------+--------------------+-------------------+-------+-------------+
8.8.5 Rdac Snapshot Compare
usage: snapshot [-h] --first FIRST_SNAPSHOT_NAME --second SECOND_SNAPSHOT_NAME
[--json]
optional arguments:
-h, --help show this help message and exit
--first FIRST_SNAPSHOT_NAME
First snapshot name
--second SECOND_SNAPSHOT_NAME
Second snapshot name
--json Print detailed information in JSON format instead of
tabular format
2024-05-23:04:00:52 [1] INFO collector Fetching stream rda_system_snapshots_details: offset 0, totalResults=?, limit=0
2024-05-23:04:00:53 [1] INFO collector Fetching stream rda_system_snapshots_details: offset 0, totalResults=?, limit=0
[
{
"artifact_type": "Blueprint",
"artifact_name": "Alerts Enricher",
"first_checksum": "e541d734a6fb08a6648ef1d2afe34285",
"first_object": "snapshot-artifacts/Blueprint/e541d734a6fb08a6648ef1d2afe34285/data.json",
"first_size": "537",
"second_checksum": "e541d734a6fb08a6648ef1d2afe34285",
"second_object": "snapshot-artifacts/Blueprint/e541d734a6fb08a6648ef1d2afe34285/data.json",
"second_size": "537",
"change_type": "Identical",
"first_snapshot": "snapshot_cli1",
"second_snapshot": "snapshot_cli2"
},
{
"artifact_type": "Blueprint",
"artifact_name": "ML Metric Selection For IRM",
"first_checksum": "ea50300c747a9e7c509a36ac761b4bb2",
"first_object": "snapshot-artifacts/Blueprint/ea50300c747a9e7c509a36ac761b4bb2/data.json",
"first_size": "1391",
"second_checksum": "f982a9d9301818b6f1255fb98d7c8d34",
"second_object": "snapshot-artifacts/Blueprint/f982a9d9301818b6f1255fb98d7c8d34/data.json",
"second_size": "1390",
"change_type": "Different",
"first_snapshot": "snapshot_cli1",
"second_snapshot": "snapshot_cli2"
},
{
"artifact_type": "Blueprint",
"artifact_name": "OIA Source Stream Merger",
"first_checksum": "a4fcb1d4a0e853de650292d27936455a",
"first_object": "snapshot-artifacts/Blueprint/a4fcb1d4a0e853de650292d27936455a/data.json",
"first_size": "570",
"second_checksum": "a4fcb1d4a0e853de650292d27936455a",
"second_object": "snapshot-artifacts/Blueprint/a4fcb1d4a0e853de650292d27936455a/data.json",
"second_size": "570",
"change_type": "Identical",
"first_snapshot": "snapshot_cli1",
"second_snapshot": "snapshot_cli2"
},
{
"artifact_type": "PublishedPipeline",
"artifact_name": "resolve_ticket_main",
"first_checksum": "",
"first_object": "",
"first_size": "",
"second_checksum": "98e49bd5a92adc26218e7c10d0233488",
"second_object": "snapshot-artifacts/PublishedPipeline/98e49bd5a92adc26218e7c10d0233488/data.json",
"second_size": 55572,
"change_type": "Only in Second",
"first_snapshot": "snapshot_cli1",
"second_snapshot": "snapshot_cli2"
},.....................
8.8.6 Rdac Snapshot Review
usage: snapshot [-h] --first FIRST_SNAPSHOT_NAME --second SECOND_SNAPSHOT_NAME
--output OUTPUT_FILE
optional arguments:
-h, --help show this help message and exit
--first FIRST_SNAPSHOT_NAME
First snapshot name
--second SECOND_SNAPSHOT_NAME
Second snapshot name
--output OUTPUT_FILE Output Change Review Report file (tar.gz format)
2024-05-23:04:05:46 [1] INFO collector Fetching stream rda_system_snapshots_main: offset 0, totalResults=?, limit=1
2024-05-23:04:05:49 [1] INFO collector Fetching stream rda_system_snapshots_main: offset 0, totalResults=?, limit=1
2024-05-23:04:05:50 [1] INFO collector Fetching stream rda_system_snapshots_details: offset 0, totalResults=?, limit=0
2024-05-23:04:05:51 [1] INFO collector Fetching stream rda_system_snapshots_details: offset 0, totalResults=?, limit=0
2024-05-23:04:05:52 [1] INFO diff_widget Processed 10 files ...
2024-05-23:04:05:52 [1] INFO diff_widget Processed 20 files ...
2024-05-23:04:05:52 [1] INFO diff_widget Processed 30 files ...
2024-05-23:04:05:52 [1] INFO diff_widget Processed 40 files ...
2024-05-23:04:05:52 [1] INFO diff_widget Processed 50 files ...
2024-05-23:04:05:52 [1] INFO diff_widget Processed 60 files ...
2024-05-23:04:05:52 [1] INFO diff_widget Processed 70 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 80 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 90 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 100 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 110 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 120 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 130 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 140 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 150 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 160 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 170 files ...
2024-05-23:04:05:53 [1] INFO diff_widget Processed 180 files ...
2024-05-23:04:05:54 [1] INFO diff_widget Processed 190 files ...
2024-05-23:04:05:54 [1] INFO diff_widget Processed 200 files ...
2024-05-23:04:05:54 [1] INFO diff_widget Processed 210 files ...
2024-05-23:04:05:54 [1] INFO diff_widget Processed 220 files ...
2024-05-23:04:05:54 [1] INFO diff_widget Processed 230 files ...
2024-05-23:04:05:54 [1] INFO diff_widget Processed 240 files ...
2024-05-23:04:05:54 [1] INFO diff_widget Processed 250 files ...
2024-05-23:04:05:54 [1] INFO diff_widget Processed 260 files ...
2024-05-23:04:05:54 [1] INFO diff_widget Processed 270 files ...
2024-05-23:04:05:54 [1] INFO diff_widget Processed 280 files ...
2024-05-23:04:05:55 [1] INFO diff_widget Processed 290 files ...
2024-05-23:04:05:55 [1] INFO diff_widget Processed 300 files ...
Saved Change Review Report to file review_ex.tar.gz