Skip to content

Installing OIA (Operations Intelligence & Analytics)

This document provides instructions about fresh Installation of OIA (Operations Intelligence & Analytics), which is also referred as AIOps.

1. Setup & Install

cfxOIA is an application that is installed on top of RDA Fabric platform.

1.1 Tag Verion: 7.0.6

Pre-requisites:

Below are the pre-requisites which need to be in place before installing the OIA (AIOps) application services.

RDAF Deployment CLI Version: 1.1.2

RDAF Infrastructure Services Tag Version: 1.0.2

RDAF Core Platform & Worker Services Tag Version: 3.1.0

RDAF Client (RDAC) Tag Version: 3.1.0

Warning

Please complete all of the above pre-requisites before installing the OIA (AIOps) application services.

Login as rdauser user into on-premise docker registry or RDA Fabric Platform VM on which RDAF deployment CLI was installed (ex: putty)

Before installing the OIA (AIOps) application services, please run the below command to update HAProxy (Loadbalancer) configuration.

rdaf app update-config OIA
rdafk8s app update-config OIA

Run the below rdaf or rdafk8scommand, to make sure all of the RDAF infrastructure services are up and running.

rdaf infra status
rdafk8s infra status

Run the below rdac pods command, to make sure all of the RDAF core platform and worker services are up and running.

rdac pods
+-------+----------------------------------------+----------------+----------+-------------+----------+--------+--------------+---------------+--------------+
| Cat   | Pod-Type                               | Host           | ID       | Site        | Age      |   CPUs |   Memory(GB) | Active Jobs   | Total Jobs   |
|-------+----------------------------------------+----------------+----------+-------------+----------+--------+--------------+---------------+--------------|
| App   | asset-dependency                       | rda-asset-depe | 090669bf |             | 20:18:21 |      8 |        47.03 |               |              |
| App   | authenticator                          | rda-identity-5 | 57905b20 |             | 20:19:11 |      8 |        47.03 |               |              |
| App   | cfxdimensions-app-access-manager       | rda-access-man | 6338ad29 |             | 20:18:44 |      8 |        47.03 |               |              |
| App   | cfxdimensions-app-notification-service | rda-notificati | bb9e3e7b |             | 20:09:52 |      8 |        31.33 |               |              |
| App   | cfxdimensions-app-resource-manager     | rda-resource-m | e5a28e16 |             | 20:18:34 |      8 |        47.03 |               |              |
| App   | user-preferences                       | rda-user-prefe | fd09d3ba |             | 20:18:08 |      8 |        47.03 |               |              |
| Infra | api-server                             | rda-api-server | b1b910d9 |             | 20:19:22 |      8 |        47.03 |               |              |
| Infra | collector                              | rda-collector- | 99553e51 |             | 20:18:17 |      8 |        47.03 |               |              |
| Infra | registry                               | rda-registry-7 | a46cd712 |             | 20:19:15 |      8 |        47.03 |               |              |
| Infra | scheduler                              | rda-scheduler- | d5537051 | *leader*    | 20:18:26 |      8 |        47.03 |               |              |
| Infra | worker                                 | rda-worker-54d | 1f769792 | rda-site-01 | 20:06:48 |      4 |        15.6  | 0             | 0            |
+-------+----------------------------------------+----------------+----------+-------------+----------+--------+--------------+---------------+--------------+

Run the below rdac healthcheck command to check the health status of all of the RDAF core platform and worker services.

All of the dependency checks should show as ok under Status column.

rdac healthcheck
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------+
| Cat       | Pod-Type                               | Host         | ID       | Site        | Health Parameter                                    | Status   | Message                                               |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------|
| rda_infra | api-server                             | rda-api-serv | b1b910d9 |             | service-status                                      | ok       |                                                       |
| rda_infra | api-server                             | rda-api-serv | b1b910d9 |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | asset-dependency                       | rda-asset-de | 090669bf |             | service-status                                      | ok       |                                                       |
| rda_app   | asset-dependency                       | rda-asset-de | 090669bf |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | authenticator                          | rda-identity | 57905b20 |             | service-status                                      | ok       |                                                       |
| rda_app   | authenticator                          | rda-identity | 57905b20 |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | authenticator                          | rda-identity | 57905b20 |             | DB-connectivity                                     | ok       |                                                       |
| rda_app   | cfxdimensions-app-access-manager       | rda-access-m | 6338ad29 |             | service-status                                      | ok       |                                                       |
| rda_app   | cfxdimensions-app-access-manager       | rda-access-m | 6338ad29 |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | cfxdimensions-app-access-manager       | rda-access-m | 6338ad29 |             | service-dependency:registry                         | ok       | 1 pod(s) found for registry                           |
| rda_app   | cfxdimensions-app-access-manager       | rda-access-m | 6338ad29 |             | service-initialization-status                       | ok       |                                                       |
| rda_app   | cfxdimensions-app-access-manager       | rda-access-m | 6338ad29 |             | DB-connectivity                                     | ok       |                                                       |
| rda_app   | cfxdimensions-app-notification-service | rda-notifica | bb9e3e7b |             | service-status                                      | ok       |                                                       |
| rda_app   | cfxdimensions-app-notification-service | rda-notifica | bb9e3e7b |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | cfxdimensions-app-notification-service | rda-notifica | bb9e3e7b |             | service-initialization-status                       | ok       |                                                       |
| rda_app   | cfxdimensions-app-notification-service | rda-notifica | bb9e3e7b |             | DB-connectivity                                     | ok       |                                                       |
| rda_app   | cfxdimensions-app-resource-manager     | rda-resource | e5a28e16 |             | service-status                                      | ok       |                                                       |
| rda_app   | cfxdimensions-app-resource-manager     | rda-resource | e5a28e16 |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | cfxdimensions-app-resource-manager     | rda-resource | e5a28e16 |             | service-dependency:registry                         | ok       | 1 pod(s) found for registry                           |
| rda_app   | cfxdimensions-app-resource-manager     | rda-resource | e5a28e16 |             | service-dependency:cfxdimensions-app-access-manager | ok       | 1 pod(s) found for cfxdimensions-app-access-manager   |
| rda_app   | cfxdimensions-app-resource-manager     | rda-resource | e5a28e16 |             | service-initialization-status                       | ok       |                                                       |
| rda_app   | cfxdimensions-app-resource-manager     | rda-resource | e5a28e16 |             | DB-connectivity                                     | ok       |                                                       |
| rda_infra | collector                              | rda-collecto | 99553e51 |             | service-status                                      | ok       |                                                       |
| rda_infra | collector                              | rda-collecto | 99553e51 |             | minio-connectivity                                  | ok       |                                                       |
| rda_infra | collector                              | rda-collecto | 99553e51 |             | opensearch-connectivity:default                     | ok       |                                                       |
| rda_infra | registry                               | rda-registry | a46cd712 |             | service-status                                      | ok       |                                                       |
| rda_infra | registry                               | rda-registry | a46cd712 |             | minio-connectivity                                  | ok       |                                                       |
| rda_infra | scheduler                              | rda-schedule | d5537051 |             | service-status                                      | ok       |                                                       |
| rda_infra | scheduler                              | rda-schedule | d5537051 |             | minio-connectivity                                  | ok       |                                                       |
| rda_infra | scheduler                              | rda-schedule | d5537051 |             | DB-connectivity                                     | ok       |                                                       |
| rda_app   | user-preferences                       | rda-user-pre | fd09d3ba |             | service-status                                      | ok       |                                                       |
| rda_app   | user-preferences                       | rda-user-pre | fd09d3ba |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | user-preferences                       | rda-user-pre | fd09d3ba |             | service-dependency:registry                         | ok       | 1 pod(s) found for registry                           |
| rda_app   | user-preferences                       | rda-user-pre | fd09d3ba |             | service-initialization-status                       | ok       |                                                       |
| rda_app   | user-preferences                       | rda-user-pre | fd09d3ba |             | DB-connectivity                                     | ok       |                                                       |
| rda_infra | worker                                 | rda-worker-5 | 1f769792 | rda-site-01 | service-status                                      | ok       |                                                       |
| rda_infra | worker                                 | rda-worker-5 | 1f769792 | rda-site-01 | minio-connectivity                                  | ok       |                                                       |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------+

Installing OIA (AIOps) Application Services:

Run the below command to deploy RDAF OIA (AIOps) application services. (Note: Below shown tag name is a sample one for a reference only, for actual tag, please contact CloudFabrix support team at support@cloudfabrix.com)

rdaf app install OIA --tag 7.0.6
rdafk8s app install OIA --tag 7.0.6

After installing the OIA (AIOps) application services, run the below command to see the running status of the deployed application services.

rdaf app status
+---------------------------------+----------------+-----------------+--------------+-------+
| Name                            | Host           | Status          | Container Id | Tag   |
+---------------------------------+----------------+-----------------+--------------+-------+
| rda-alert-ingester              | 192.168.125.46 | Up 20 Hours ago | 610bb0e286d6 | 7.0.6 |
| rda-alert-processor             | 192.168.125.46 | Up 20 Hours ago | 79ee6788f73e | 7.0.6 |
| rda-app-controller              | 192.168.125.46 | Up 20 Hours ago | 6c672102d5ff | 7.0.6 |
| rda-collaboration               | 192.168.125.46 | Up 20 Hours ago | 34f25c05afce | 7.0.6 |
| rda-configuration-service       | 192.168.125.46 | Up 20 Hours ago | 112ccaf4b0e6 | 7.0.6 |
| rda-dataset-caas-all-alerts     | 192.168.125.46 | Up 20 Hours ago | 2b48d4dfbfd0 | 7.0.6 |
| rda-dataset-caas-current-alerts | 192.168.125.46 | Up 20 Hours ago | 03cdc77ddf1f | 7.0.6 |
| rda-event-consumer              | 192.168.125.46 | Up 20 Hours ago | 21113ba951a1 | 7.0.6 |
| rda-file-browser                | 192.168.125.46 | Up 20 Hours ago | 425dac228fc9 | 7.0.6 |
| rda-ingestion-tracker           | 192.168.125.46 | Up 20 Hours ago | 8a984a536a97 | 7.0.6 |
| rda-irm-service                 | 192.168.125.46 | Up 20 Hours ago | 258aadc0c1af | 7.0.6 |
| rda-ml-config                   | 192.168.125.46 | Up 20 Hours ago | bf23d58903f7 | 7.0.6 |
| rda-notification-service        | 192.168.125.46 | Up 20 Hours ago | a15c5232b25d | 7.0.6 |
| rda-reports-registry            | 192.168.125.46 | Up 20 Hours ago | 3890b5dfb8ae | 7.0.6 |
| rda-smtp-server                 | 192.168.125.46 | Up 20 Hours ago | 6aadab781947 | 7.0.6 |
| rda-webhook-server              | 192.168.125.46 | Up 20 Hours ago | 6bf555aed18b | 7.0.6 |
+---------------------------------+--------------+-----------------+--------------+-------+
rdafk8s app status
+---------------------------------+----------------+-----------------+--------------+-------+
| Name                            | Host           | Status          | Container Id | Tag   |
+---------------------------------+----------------+-----------------+--------------+-------+
| rda-alert-ingester              | 192.168.125.46 | Up 20 Hours ago | 610bb0e286d6 | 7.0.6 |
| rda-alert-processor             | 192.168.125.46 | Up 20 Hours ago | 79ee6788f73e | 7.0.6 |
| rda-app-controller              | 192.168.125.46 | Up 20 Hours ago | 6c672102d5ff | 7.0.6 |
| rda-collaboration               | 192.168.125.46 | Up 20 Hours ago | 34f25c05afce | 7.0.6 |
| rda-configuration-service       | 192.168.125.46 | Up 20 Hours ago | 112ccaf4b0e6 | 7.0.6 |
| rda-dataset-caas-all-alerts     | 192.168.125.46 | Up 20 Hours ago | 2b48d4dfbfd0 | 7.0.6 |
| rda-dataset-caas-current-alerts | 192.168.125.46 | Up 20 Hours ago | 03cdc77ddf1f | 7.0.6 |
| rda-event-consumer              | 192.168.125.46 | Up 20 Hours ago | 21113ba951a1 | 7.0.6 |
| rda-file-browser                | 192.168.125.46 | Up 20 Hours ago | 425dac228fc9 | 7.0.6 |
| rda-ingestion-tracker           | 192.168.125.46 | Up 20 Hours ago | 8a984a536a97 | 7.0.6 |
| rda-irm-service                 | 192.168.125.46 | Up 20 Hours ago | 258aadc0c1af | 7.0.6 |
| rda-ml-config                   | 192.168.125.46 | Up 20 Hours ago | bf23d58903f7 | 7.0.6 |
| rda-notification-service        | 192.168.125.46 | Up 20 Hours ago | a15c5232b25d | 7.0.6 |
| rda-reports-registry            | 192.168.125.46 | Up 20 Hours ago | 3890b5dfb8ae | 7.0.6 |
| rda-smtp-server                 | 192.168.125.46 | Up 20 Hours ago | 6aadab781947 | 7.0.6 |
| rda-webhook-server              | 192.168.125.46 | Up 20 Hours ago | 6bf555aed18b | 7.0.6 |
+---------------------------------+--------------+-----------------+--------------+-------+

Configuring OIA (AIOps) Application:

Login into RDAF portal as admin@cfx.com user.

Create a new Service Blueprint for OIA (AIOps) application and Machine Learning (ML) application.

For OIA (AIOps) Application: Go to Main Menu --> Configuration --> Artifacts --> Service Blueprints --> View details --> Click on Add and copy & paste the below configuration and Click on Save

name: cfxOIA
id: 81a1a2202
version: 2023_02_12_01
category: ITOM
comment: Operations Intelligence & Analytics (AIOps)
enabled: true
type: Service
provider: CloudFabrix Software, Inc.
attrs: {}
apps:
    -   label: cfxOIA
        appType: dimensions
        appName: incident-room-manager
        icon_url: /assets/img/applications/OIA.png
        permission: app:irm:read
service_pipelines: []

For Machine Learning (ML) Application: Go to Main Menu --> Configuration --> Artifacts --> Service Blueprints --> View details --> Click on Add and copy & paste the below configuration and Click on Save

name: cfxML
id: 81a1a030
version: 2023_02_12_01
category: ITOM
comment: Machine Learning (ML) Experiments
enabled: true
type: Service
provider: CloudFabrix Software, Inc.
attrs: {}
apps:
    -   label: cfxML
        appType: dimensions
        appName: ml-config
        icon_url: /assets/img/applications/ML.png
        permission: app:irm:read
service_pipelines: []

CFXOIA_App_ML_App_Add

2. Upgrade

This section provides instructions on how to upgrade existing deployment of RDAF platform and it's application OIA (Operations Intelligence & Analytics), which is also referred as AIOps.

2.1 Upgrade from 7.0.x to 7.0.6

Upgrade Prerequisites

Below are the pre-requisites which need to be in place before upgrafing the OIA (AIOps) application services.

RDAF Deployment CLI Version Upgrade: From 1.0.6 or higher to 1.1.2

RDAF Infrastructure Services Tag Version: From 1.0.1 or higher to 1.0.2 (Note: Not applicable if the services are already running at 1.0.2 version)

RDAF Core Platform & Worker Services Tag Version: From 3.0.9 to 3.1.0

RDAF Client (RDAC) Tag Version: From 3.0.9 to 3.1.0

Warning

Please complete all of the above pre-requisites before installing the OIA (AIOps) application services.

On-premise docker-registry

Login into RDAF on-premise docker-registry VM or RDAF platform VM as rdauser using SSH client on which rdaf CLI was installed and run the below command to verify status of the docker-registry service.

rdaf status
+-----------------+---------------+------------+--------------+-------+
| Name            | Host          | Status     | Container Id | Tag   |
+-----------------+---------------+------------+--------------+-------+
| docker-registry | 111.92.12.140 | Up 4 weeks | 71b8036fc64f | 1.0.1 |
+-----------------+---------------+------------+--------------+-------+

RDAF Infrastructure, Platform and Application services:

Login into RDAF on-premise docker-registry VM or RDAF platform VM as rdauser using SSH client on which rdaf CLI was installed and run the below command to verify status of the RDAF platform's infrastructure, core platform, application and worker services.

rdafk8s infra status
+----------------+--------------+-----------------+--------------+------------------------------+
| Name           | Host         | Status          | Container Id | Tag                          |
+----------------+--------------+-----------------+--------------+------------------------------+
| haproxy        | 111.92.12.41 | Up 6 days       | 245a37201207 | 1.0.2                        |
| keepalived     | 111.92.12.41 | Not Provisioned | N/A          | N/A                          |
| nats           | 111.92.12.41 | Up 6 days       | 15469a93d96f | 1.0.2                        |
| minio          | 111.92.12.41 | Up 6 days       | 3fd3f97bf25b | RELEASE.2022-11-07T23-47-39Z |
| mariadb        | 111.92.12.41 | Up 6 days       | 0fa1a0027993 | 1.0.2                        |
| opensearch     | 111.92.12.41 | Up 6 days       | dae308716400 | 1.0.2                        |
| zookeeper      | 111.92.12.41 | Up 6 days       | 4d8f61b4ab17 | 1.0.2                        |
| kafka          | 111.92.12.41 | Up 6 days       | 0dee08cd9c59 | 1.0.2                        |
| redis          | 111.92.12.41 | Up 6 days       | d1eccf90846e | 1.0.2                        |
| redis-sentinel | 111.92.12.41 | Up 6 days       | 683beb7b913e | 1.0.2                        |
+----------------+--------------+-----------------+--------------+------------------------------+
rdafk8s platform status
+--------------------------+--------------+-----------+--------------+-------+
| Name                     | Host         | Status    | Container Id | Tag   |
+--------------------------+--------------+-----------+--------------+-------+
| cfx-rda-access-manager   | 111.92.12.41 | Up 6 days | e487cdf24b46 | 3.0.9 |
| cfx-rda-resource-manager | 111.92.12.41 | Up 6 days | a7a21a31a26e | 3.0.9 |
| cfx-rda-user-preferences | 111.92.12.41 | Up 6 days | 9306d8da4b5a | 3.0.9 |
| portal-backend           | 111.92.12.41 | Up 6 days | 55df761dad1d | 3.0.9 |
| portal-frontend          | 111.92.12.41 | Up 6 days | 2183f00efa64 | 3.0.9 |
| rda_api_server           | 111.92.12.41 | Up 6 days | 3ba6256d1694 | 3.0.9 |
| rda_asset_dependency     | 111.92.12.41 | Up 6 days | d1a8b76bb114 | 3.0.9 |
| rda_collector            | 111.92.12.41 | Up 6 days | 441427d2bb1e | 3.0.9 |
| rda_identity             | 111.92.12.41 | Up 6 days | 2c1215d9155a | 3.0.9 |
| rda_registry             | 111.92.12.41 | Up 6 days | 7358e6ee6298 | 3.0.9 |
| rda_scheduler            | 111.92.12.41 | Up 6 days | ee72c66f8c80 | 3.0.9 |
+--------------------------+--------------+-----------+--------------+-------+
rdafk8s worker status
+------------+--------------+-----------+--------------+-------+
| Name       | Host         | Status    | Container Id | Tag   |
+------------+--------------+-----------+--------------+-------+
| rda_worker | 111.92.12.43 | Up 6 days | 88f4916ce18e | 3.0.9 |
| rda_worker | 111.92.12.43 | Up 6 days | 88f491612345 | 3.0.9 |
+------------+--------------+-----------+--------------+-------+
rdafk8s app status
+------------------------------+--------------+-----------+--------------+-------+
| Name                         | Host         | Status    | Container Id | Tag   |
+------------------------------+--------------+-----------+--------------+-------+
| all-alerts-cfx-rda-dataset-  | 111.92.12.42 | Up 6 days | 58a75c01c51f | 7.0.5 |
| caas                         |              |           |              |       |
| cfx-rda-alert-ingester       | 111.92.12.42 | Up 6 days | bc9a78953b73 | 7.0.5 |
| cfx-rda-alert-processor      | 111.92.12.42 | Up 6 days | 28401e5c2570 | 7.0.5 |
| cfx-rda-app-builder          | 111.92.12.42 | Up 6 days | be8f100056fd | 7.0.5 |
| cfx-rda-app-controller       | 111.92.12.42 | Up 6 days | a7a4ef35097d | 7.0.5 |
| cfx-rda-collaboration        | 111.92.12.42 | Up 6 days | d9d980b28a2b | 7.0.5 |
| cfx-rda-configuration-       | 111.92.12.42 | Up 6 days | db1a45835e1a | 7.0.5 |
| service                      |              |           |              |       |
| cfx-rda-event-consumer       | 111.92.12.42 | Up 6 days | baf09bad3ce1 | 7.0.5 |
| cfx-rda-file-browser         | 111.92.12.42 | Up 6 days | 32ccdfca8d8f | 7.0.5 |
| cfx-rda-ingestion-tracker    | 111.92.12.42 | Up 6 days | 1030345f2179 | 7.0.5 |
| cfx-rda-irm-service          | 111.92.12.42 | Up 6 days | 89d931f7d7b8 | 7.0.5 |
| cfx-rda-ml-config            | 111.92.12.42 | Up 6 days | 57fc39489a08 | 7.0.5 |
| cfx-rda-notification-service | 111.92.12.42 | Up 6 days | 408dbebb33c5 | 7.0.5 |
| cfx-rda-reports-registry     | 111.92.12.42 | Up 6 days | 3296cba8b3e4 | 7.0.5 |
| cfx-rda-smtp-server          | 111.92.12.42 | Up 6 days | 0f9884b6e7c8 | 7.0.5 |
| cfx-rda-webhook-server       | 111.92.12.42 | Up 6 days | a4403dee414e | 7.0.5 |
| current-alerts-cfx-rda-      | 111.92.12.42 | Up 6 days | d6cc63214103 | 7.0.5 |
| dataset-caas                 |              |           |              |       |
+------------------------------+--------------+-----------+--------------+-------+

Important

Please take RDAF platform's full data backup before performing an upgrade. For more information on RDAF platform's backup and restore commands using rdaf CLI, please refer at RDAF Platform Backup

Download RDAF Platform & OIA Images

  • Login into On-premise docker registry VM as rdauser using SSH client and run the below command to download RDAF platform's and OIA (AIOps) application service's updated images.
rdaf registry fetch --tag 1.0.2,3.1.0,7.0.6
  • Please wait until all of the RDAF platform's and OIA (AIOps) application service's images are downloaded. Run the below command to verify if the images are downloaded appropriately.
rdaf registry list-tags

Upgrade RDAF deployment CLI on RDAF Platform VM

Please follow and repeat the steps to download and upgrade the rdaf deployment CLI on RDAF platform VM using the steps outlined under RDAF CLI Upgrade on On-premise docker registry VM section.

Upgrade RDAF Platform & OIA Services

RDAF Platform Services Upgrade:

Run the below command to upgrade RDAF platform's services to version

rdafk8s platform upgrade --tag 3.1.0

Once above command is completed, run the below command to verify all of RDAF platform's services are upgraded to the specified version and all of their corresponding containers are in running state.

kubectl get pods -n rda-fabric -l app_category=rdaf-platform

RDAF Client CLI Upgrade:

Run the below command to upgrade the RDAF client CLI rdac to latest version.

rdafk8s rdac_cli upgrade --tag 3.1.0

After rdac CLI is upgraded, run the below commands to see all of the running RDAF platform's services pods.

rdac pods 
+-------+----------------------------------------+--------------+----------+-------------+-----------------+--------+--------------+---------------+--------------+
| Cat   | Pod-Type                               | Host         | ID       | Site        | Age             |   CPUs |   Memory(GB) | Active Jobs   | Total Jobs   |
|-------+----------------------------------------+--------------+----------+-------------+-----------------+--------+--------------+---------------+--------------|
| App   | cfxdimensions-app-access-manager       | d412efb99f2e | ccb83d20 |             | 4:13:45         |      8 |        31.21 |               |              |
| App   | cfxdimensions-app-notification-service | 34c2ea6675d5 | 93ac81de |             | 1 day, 18:33:27 |      8 |        31.21 |               |              |
| App   | cfxdimensions-app-resource-manager     | ec87d2ee6387 | 33ee28ca |             | 4:13:31         |      8 |        31.21 |               |              |
| App   | user-preferences                       | 520bca813ddf | f4ca7d44 |             | 4:13:14         |      8 |        31.21 |               |              |
| Infra | api-server                             | 0656b4230f44 | 6d4d40ab |             | 0:33:06         |      8 |        31.21 |               |              |
| Infra | collector                              | 6336341682ad | 042af0af |             | 4:11:19         |      8 |        31.21 |               |              |
| Infra | registry                               | cae649622fba | 4e4c4a4d |             | 4:11:03         |      8 |        31.21 |               |              |
| Infra | scheduler                              | 3ab379305be1 | b2bb9915 | *leader*    | 4:10:59         |      8 |        31.21 |               |              |
+-------+----------------------------------------+--------------+----------+-------------+-----------------+--------+--------------+---------------+--------------+

Run the below command to verify functional health of each platform's service and verify all of their status is in OK state.

rdac healthcheck
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------+
| Cat       | Pod-Type                               | Host         | ID       | Site        | Health Parameter                                    | Status   | Message                                               |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------|
| rda_infra | api-server                             | 0656b4230f44 | 6d4d40ab |             | service-status                                      | ok       |                                                       |
| rda_infra | api-server                             | 0656b4230f44 | 6d4d40ab |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | asset-dependency                       | e006dfd39d9b | 9f02a8f1 |             | service-status                                      | ok       |                                                       |
| rda_app   | asset-dependency                       | e006dfd39d9b | 9f02a8f1 |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | authenticator                          | 1782a79e36c5 | adda9bc0 |             | service-status                                      | ok       |                                                       |
| rda_app   | authenticator                          | 1782a79e36c5 | adda9bc0 |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | authenticator                          | 1782a79e36c5 | adda9bc0 |             | DB-connectivity                                     | ok       |                                                       |
| rda_app   | cfxdimensions-app-access-manager       | d412efb99f2e | ccb83d20 |             | service-status                                      | ok       |                                                       |
| rda_app   | cfxdimensions-app-access-manager       | d412efb99f2e | ccb83d20 |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | cfxdimensions-app-access-manager       | d412efb99f2e | ccb83d20 |             | service-dependency:registry                         | ok       | 1 pod(s) found for registry                           |
| rda_app   | cfxdimensions-app-access-manager       | d412efb99f2e | ccb83d20 |             | service-initialization-status                       | ok       |                                                       |
| rda_app   | cfxdimensions-app-access-manager       | d412efb99f2e | ccb83d20 |             | DB-connectivity                                     | ok       |                                                       |                                                  |
| rda_app   | cfxdimensions-app-notification-service | 34c2ea6675d5 | 93ac81de |             | service-status                                      | ok       |                                                       |
| rda_app   | cfxdimensions-app-notification-service | 34c2ea6675d5 | 93ac81de |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | cfxdimensions-app-notification-service | 34c2ea6675d5 | 93ac81de |             | service-initialization-status                       | ok       |                                                       |
| rda_app   | cfxdimensions-app-notification-service | 34c2ea6675d5 | 93ac81de |             | DB-connectivity                                     | ok       |                                                       |
| rda_app   | cfxdimensions-app-resource-manager     | ec87d2ee6387 | 33ee28ca |             | service-status                                      | ok       |                                                       |
| rda_app   | cfxdimensions-app-resource-manager     | ec87d2ee6387 | 33ee28ca |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | cfxdimensions-app-resource-manager     | ec87d2ee6387 | 33ee28ca |             | service-dependency:registry                         | ok       | 1 pod(s) found for registry                           |
| rda_app   | cfxdimensions-app-resource-manager     | ec87d2ee6387 | 33ee28ca |             | service-dependency:cfxdimensions-app-access-manager | ok       | 1 pod(s) found for cfxdimensions-app-access-manager   |
| rda_app   | cfxdimensions-app-resource-manager     | ec87d2ee6387 | 33ee28ca |             | service-initialization-status                       | ok       |                                                       |
| rda_app   | cfxdimensions-app-resource-manager     | ec87d2ee6387 | 33ee28ca |             | DB-connectivity                                     | ok       |                                                       |
| rda_infra | collector                              | 6336341682ad | 042af0af |             | service-status                                      | ok       |                                                       |
| rda_infra | collector                              | 6336341682ad | 042af0af |             | minio-connectivity                                  | ok       |                                                       |
| rda_infra | collector                              | 6336341682ad | 042af0af |             | opensearch-connectivity:default                     | ok       |                                                       |
| rda_infra | scheduler                              | 3ab379305be1 | b2bb9915 |             | service-status                                      | ok       |                                                       |
| rda_infra | scheduler                              | 3ab379305be1 | b2bb9915 |             | minio-connectivity                                  | ok       |                                                       |
| rda_infra | scheduler                              | 3ab379305be1 | b2bb9915 |             | DB-connectivity                                     | ok       |                                                       |
| rda_app   | user-preferences                       | 520bca813ddf | f4ca7d44 |             | service-status                                      | ok       |                                                       |
| rda_app   | user-preferences                       | 520bca813ddf | f4ca7d44 |             | minio-connectivity                                  | ok       |                                                       |
| rda_app   | user-preferences                       | 520bca813ddf | f4ca7d44 |             | service-dependency:registry                         | ok       | 1 pod(s) found for registry                           |
| rda_app   | user-preferences                       | 520bca813ddf | f4ca7d44 |             | service-initialization-status                       | ok       |                                                       |
| rda_app   | user-preferences                       | 520bca813ddf | f4ca7d44 |             | DB-connectivity                                     | ok       |                                                       |                                      |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------+

RDAF Worker Service Upgrade:

Run the below command to upgrade RDAF worker services to latest version.

rdafk8s worker upgrade --tag 3.1.0

After upgrading the RDAF worker service using the above command, run the below command to verify it's running status and the version.

kubectl get pods -n rda-fabric -l app_category=rdaf-worker
+------------+--------------+-------------+--------------+-------+
| Name       | Host         | Status      | Container Id | Tag   |
+------------+--------------+-------------+--------------+-------+
| rda_worker | 111.92.12.60 | Up 1 minute | 4ce2a8f13d16 | 3.1.0 |
+------------+--------------+-------------+--------------+-------+

Run the below command to verify the functional health of the each RDAF worker service and verify that all of their status is in OK state.

rdac healthcheck
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------+
| Cat       | Pod-Type                               | Host         | ID       | Site        | Health Parameter                                    | Status   | Message                                               |
|-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------|
| rda_infra | api-server                             | 0656b4230f44 | 6d4d40ab |             | service-status                                      | ok       |                                                       |
...
...
| rda_infra | worker                                 | 4ce2a8f13d16 | d627124d | rda-site-01 | service-status                                      | ok       |                                                       |
| rda_infra | worker                                 | 4ce2a8f13d16 | d627124d | rda-site-01 | minio-connectivity                                  | ok       |                                                       |
+-----------+----------------------------------------+--------------+----------+-------------+-----------------------------------------------------+----------+-------------------------------------------------------+

Create Kafka Topics for OIA Application Services:

Download the below script and execute it on where rdafk8s setup was run during the initial RDAF platform setup. Please make sure the file /opt/rdaf/rdaf.cfg exist which is required for the below script to execute successfully.

wget https://macaw-amer.s3.amazonaws.com/releases/rdaf-platform/1.1.2/add_kafka_topics.py
python add_kafka_topics.py upgrade

RDAF OIA Application Services Upgrade:

Run the below command to upgrade the RDAF OIA (AIOps) application services to latest version.

rdafk8s app upgrade OIA --tag 7.0.6

Once above command is completed, run the below command to verify all of the RDAF OIA application services are upgraded to the specified version and all of their corresponding containers are in running state.

kubectl get pods -n rda-fabric -l app_category=rdaf-application

Wait for 3 to 5 minutes and run the below command to verify the functional health of each RDAF OIA application service and verify all of their status is in OK state.

rdac healthcheck