Guide to using RDAF deployment CLI for Kubernetes Environments.
1. RDAF Deployment CLI for Kubernetes
RDA Fabric deployment CLI is a comprehensive command line management tool that is used to setup, install/deploy and manage CloudFabrix on-premise Docker registry, RDA Fabric platform, infrastructure and application services.
RDA Fabric platform, infrastructure and application services are supported to be deployed on a Kubernetes Cluster or as a Standalone Container Services using docker-compose utility.
Please refer for RDAF Platform deployment using Standalone Container Services
RDAF CLI uses docker-compose as underlying container management utility for deploying and managing RDA Fabric environment when it need to be deployed on non-kubernetes cluster environment.
For deploying RDAF platform on Kubernetes cluster environment, RDAF CLI provides rdafk8s command utility using which installtion and configuration management operations can be performed.
RDAF CLI can be installed on on-premise docker registry VM if it is provisioned or on one of the RDA Fabric platform VMs or both to install, configure and manage on-premise docker registry service and RDA Fabric platform services.
Info
RDAF deployment CLI rdafk8s uses helm and kubectl to automate the RDA Fabric platform deployment and lifecycle operations on Kubernetes cluster environment. For more information about these native Kubernetes management command utility tools, please refer About Helm and About Kubectl
Pre-requisites:
RDAF deployment CLI for Kubernetes Cluster environment uses kubectl for some of the life-cycle management operations. Please make sure kubectl command line utility is installed on which RDAF deployment CLI is going to be installed.
For Ubuntu based OS, please follow the below steps to install kubectl.
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
Refresh the available packages for Ubuntu OS using the below command.
Install kubectl CLI
After installing kubectl command line utility, please follow the below steps to copy the Kubernetes Cluster configuration using which kubectl can communicate with the Kubernetes Cluster and perform lifecycle management operations.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Note
/etc/kubernetes/admin.conf file need to be copied from Kubernetes Cluster master node to RDAF deployment CLI host to the home directory of the user, i.e. rdauser
To validate if kubectl command is worknig from RDAF deployment host, please run the below command and make sure it is returning the results.
NAME STATUS ROLES AGE VERSION
k8rdapfm01 Ready <none> 116d v1.23.4
k8rdasvc01 Ready <none> 116d v1.23.4
k8rdaworker01 Ready <none> 116d v1.23.4
k8smaster01 Ready control-plane,master 116d v1.23.4
1.1 CLI Installation:
Please download the RDAF deployment bundles from the below provided links.
Tip
In a restricted environment where there is no direct internet access, please download RDAF Deployment CLI offline bundle.
RDAF Deployment CLI offline bundle for RHEL: offline-rhel-1.2.1.tar.gz
RDAF Deployment CLI offline bundle for Ubuntu: offline-ubuntu-1.2.1.tar.gz
RDAF Deployment CLI bundle: rdafcli-1.2.1.tar.gz
Note
For latest RDAF Deployment CLI versioned package, please contact support@cloudfabrix.com
Login as rdauser user into on-premise docker registry or RDA Fabric Platform VM using any SSH client tool (ex: putty)
Run the following command to install the RDA Fabric deployment CLI tool.
Note
Once the above commands run successfully, logout and logback in to a new session.
Run the below command to verify installed RDAF deployment CLI version
Run the below command to view the RDAF deployment CLI help
Documented commands (type help <topic>):
========================================
app help platform rdac_cli reset setregistry status worker
backup infra prune_images registry restore setup validate
Documented commands (type help <topic>):
========================================
app infra rdac_cli reset setup worker
help platform registry setregistry status
1.2 On-premise Docker Registry setup:
CloudFabrix support hosting an on-premise docker registry which will download and synchronize RDA Fabric's platform, infrastructure and application services from CloudFabrix's public docker registry that is securely hosted on AWS and from other public docker registries as well. For more information on on-premise docker registry, please refer Docker registry access for RDAF platform services
1.2.1 rdaf registry setup
Run rdaf registry --help to see available CLI options to deploy and manage on-premise docker registry.
Run rdaf registry setup --help to see available CLI options.
Run the below command to setup and configure on-premise docker registry service. In the below command example, 10.99.120.140 is the machine on which on-premise registry service is going to installed.
cfxregistry.cloudfabrix.io is the CloudFabrix's public docker registry hosted on AWS from which RDA Fabric docker images are going to be downloaded.
1.2.2 rdaf registry install
Run the below command to install the on-premise docker registry service.
Info
- For latest tag version, please contact support@cloudfabrix.com
- On-premise docker registry service runs on port TCP/5000. This port may need to be enabled on firewall device if on-premise docker registry service and RDA Fabric service VMs are deployed in different network environments.
Run the below command to upgrade the on-premise docker registry service to latest version.
To check the status of the on-premise docker registry service, run the below command.
1.2.3 rdaf registry fetch
Once on-premise docker registry service is installed, run the below command to download one or more tags to pre-stage the docker images for RDA Fabric services deployment for fresh install or upgrade.
Minio object storage service image need to be downloaded explicitly using the below command.
Info
Note: It may take few minutes to few hours depends on the outbound internet access bandwidth and the number of docker images to be downloaded. The default location path for the downloaded docker images is /opt/rdaf/data/docker/registry. This path can be overridden/changed during rdaf registry setup command using --install-root option if needed.
1.2.4 rdaf registry list-tags
Run the below command to list the downloaded images and their corresponding tags / versions.
1.2.5 rdaf registry delete-images
Run the below command to delete one or more tags and corresponding docker images from on-premise docker registry.
Important
When on-premise docker repository service is used, please make sure to add the insecure-registries parameter to /etc/docker/daemon.json file and restart the docker daemon as shown below on all of RDA Fabric VMs before the deployment.
...
...
Kernel Version: 5.4.0-110-generic
Operating System: Ubuntu 20.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.741GiB
Name: rdaf-onprem-docker-repo
ID: OLZF:ZKWN:TIQJ:ZMNV:2STT:JHR3:3RAT:TAL5:TF47:OGVQ:LHY7:RMHH
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
10.99.120.140:5000
127.0.0.0/8
Live Restore Enabled: true
Tip
The location of the on-premise docker registry's CA certificate file ca.crt is located under /opt/rdaf-registry/cert/ca. This file ca.crt need to be copied to the machine on which RDAF CLI is used to setup, configure and install RDA Fabric platform and all of the required services using on-premise docker registry. This step is not applicable when cloud hosted docker registry cfxregistry.cloudfabrix.io is used.
1.3 RDAF Platform setup
1.3.1 rdafk8s setregistry
When on-premise docker registry is deployed, change the default docker registry configuration to on-premise docker registry host to pull and install the RDA Fabric services.
Please refer rdafk8s setregistry --help for detailed command options.
Configure the Docker registry for the platform
optional arguments:
-h, --help show this help message and exit
--debug Enable debug logs for the CLI operations (Optional)
--host DOCKER_REGISTRY_HOST
Hostname/IP of the Docker registry
--port DOCKER_REGISTRY_PORT
Port of the Docker registry
--user DOCKER_REGISTRY_USER
Username of the Docker registry (Optional)
--password DOCKER_REGISTRY_PASSWORD
Password of the Docker registry (Optional)
--cert-path CERT_PATH
path of the Docker registry ca cert
- Copy the
ca.crtfile from on-premise registry.
scp rdauser@<on-premise-registry-ip>:/opt/rdaf-registry/cert/ca/ca.crt /opt/rdaf-registry/registry-ca-cert.crt
- Run the below command to set the docker-registry to on-premise one.
rdafk8s setregistry --host <on-premise-docker-registry-ip-or-dns> --port 5000 --cert-path /opt/rdaf-registry/registry-ca-cert.crt
Tip
Please verify if on-premise registry is accessible on port 5000 using either of the below commands.
- telnet
<on-premise-docker-registry-ip-or-dns>5000 - curl -vv telnet://
<on-premise-docker-registry-ip-or-dns>:5000
1.3.2 rdafk8s setup
Run the below rdafk8s setup command to create the RDAF platform's deployment configuration. It is a pre-requisite before RDAF infrastructure, platform and application services can be installed on Kubernetes Cluser.
It will prompt for all the necessary configuration details.
- Accept the EULA
- Enter the rdauser SSH password for all of the Kubernetes worker nodes on which RDAF services are going to be installed.
What is the SSH password for the SSH user used to communicate between hosts
SSH password:
Re-enter SSH password:
Tip
Please make sure rdauser's SSH password on all of the Kubernetes cluster worker nodes is same during the rdafk8s setup command.
- Enter additional IP address(es) or DNS names that can be as SANs (Subject alt names) while generating self-signed certificates. This is an optional configuration, but it is important to include any public facing IP addresse(s) that is/are different from worker node's ip addresses which are specified as part of the
rdafk8s setupcommand.
Tip
SANs (Subject alt names) also known as multi-domain certificates which allows to create a single unified SSL certificate which includes more than one Common Name (CN). Common Name can be an IP Address or DNS Name or a wildcard DNS Name (ex: *.acme.com)
Provide any Subject alt name(s) to be used while generating SAN certs
Subject alt name(s) for certs[]: 100.30.10.10
- Enter Kubernetes worker node IPs on which RDAF Platform services need to be installed. For HA configuration, please enter comma separated values. Minimum of 2 worker nodes are required for the HA configuration. If it is a non-HA deployment, only one Kubernetes worker node's ip address or DNS name is required.
What are the host(s) on which you want the RDAF platform services to be installed?
Platform service host(s)[rda-platform-vm01]: 192.168.125.141,192.168.125.142
- Answer if the RDAF application services are going to be deployed in HA mode or standalone.
- Enter Kubernetes worker node IPs on which RDAF Application services (OIA/AIA) need to be installed. For HA configuration, please enter comma separated values. Minimum of 2 hosts or more are required for the HA configuration. If it is a non-HA deployment, only one Kubernetes worker node's ip address or DNS name is required.
What are the host(s) on which you want the application services to be installed?
Application service host(s)[rda-platform-vm01]: 192.168.125.143,192.168.125.144
- Enter the name of the Organization. In the below example,
ACME_IT_Servicesis used as the Organization name. It is for a reference only.
What is the organization you want to use for the admin user created?
Admin organization[CloudFabrix]: ACME_IT_Services
What is the ca cert to use to communicate to on-prem docker registry
Docker Registry CA cert path[]:
- Enter Kubernetes worker node IPs on which RDAF Worker services need to be installed. For HA configuration, please enter comma separated values. Minimum of 2 hosts or more are required for the HA configuration. If it is a non-HA deployment, only one Kubernetes worker node's ip address or DNS name is required.
What are the host(s) on which you want the Worker to be installed?
Worker host(s)[rda-platform-vm01]: 192.168.125.145
- Enter Kubernetes worker node IPs on which RDAF NATs infrastructure service need to be installed. For HA configuration, please enter comma separated values. Minimum of 2 Kubernetes worker nodes are required for the
NATsHA configuration. If it is a non-HA deployment, only one Kubernetes worker node's IP or DNS Name is required.
What is the "host/path-on-host" on which you want the Nats to be deployed?
Nats host/path[192.168.125.141]: 192.168.125.145,192.168.125.146
- Enter Kubernetes worker node IPs on which RDAF Minio infrastructure service need to be installed. For HA configuration, please enter comma separated values. Minimum of 4 Kubernetes worker nodes are required for the
MinioHA configuration. If it is a non-HA deployment, only one Kubernetes worker node's IP or DNS Name is required.
What is the "host/path-on-host" where you want Minio to be provisioned?
Minio server host/path[192.168.125.141]: 192.168.125.145,192.168.125.146,192.168.125.147,192.168.125.148
- Change the default
Miniouser credentials if needed or press Enter to accept the defaults.
What is the user name you want to give for Minio root user that will be created and used by the RDAF platform?
Minio user[rdafadmin]:
What is the password you want to use for the newly created Minio root user?
Minio password[Q8aJ63PT]:
- Enter Kubernetes worker node IPs on which RDAF MariaDB infrastructure service need to be installed. For HA configuration, please enter comma separated values. Minimum of 3 Kubernetes worker nodes are required for the
MariDBdatabase HA configuration. If it is a non-HA deployment, only one Kubernetes worker node's IP or DNS Name is required.
What is the "host/path-on-host" on which you want the MariaDB server to be provisioned?
MariaDB server host/path[192.168.125.141]: 192.168.125.145,192.168.125.146,192.168.125.147
- Change the default
MariaDBuser credentials if needed or press Enter to accept the defaults.
What is the user name you want to give for MariaDB admin user that will be created and used by the RDAF platform?
MariaDB user[rdafadmin]:
What is the password you want to use for the newly created MariaDB root user?
MariaDB password[jffqjAaZ]:
- Enter Kubernetes worker node IPs on which RDAF Opensearch infrastructure service need to be installed. For HA configuration, please enter comma separated values. Minimum of 3 Kubernetes worker nodes are required for the
OpensearchHA configuration. If it is a non-HA deployment, only one Kubernetes worker node's IP or DNS Name is required.
What is the "host/path-on-host" on which you want the opensearch server to be provisioned?
opensearch server host/path[192.168.125.141]: 192.168.125.145,192.168.125.146,192.168.125.147
- Change the default
Opensearchuser credentials if needed or press Enter to accept the defaults.
What is the user name you want to give for Opensearch admin user that will be created and used by the RDAF platform?
Opensearch user[rdafadmin]:
What is the password you want to use for the newly created Opensearch admin user?
Opensearch password[sLmr4ICX]:
- Enter Kubernetes worker node IPs on which RDAF Zookeeper infrastructure service need to be installed. For HA configuration, please enter comma separated values. Minimum of 3 Kubernetes worker nodes are required for the
ZookeeperHA configuration. If it is a non-HA deployment, only one Kubernetes worker node's IP or DNS Name is required.
What is the "host/path-on-host" on which you want the Zookeeper server to be provisioned?
Zookeeper server host/path[192.168.125.141]: 192.168.125.145,192.168.125.146,192.168.125.147
- Enter Kubernetes worker node IPs on which RDAF Kafka infrastructure service need to be installed. For HA configuration, please enter comma separated values. Minimum of 3 Kubernetes worker nodes are required for the
KafkaHA configuration. If it is a non-HA deployment, only one Kubernetes worker node's IP or DNS Name is required.
What is the "host/path-on-host" on which you want the Kafka server to be provisioned?
Kafka server host/path[192.168.125.141]: 192.168.125.145,192.168.125.146,192.168.125.147
- Enter Kubernetes worker node IPs on which RDAF Redis infrastructure service need to be installed. For HA configuration, please enter comma separated values. Minimum of 3 Kubernetes worker nodes are required for the
RedisHA configuration. If it is a non-HA deployment, only one Kubernetes worker node's IP or DNS Name is required.
What is the "host/path-on-host" on which you want the Redis server to be provisioned?
Redis server host/path[192.168.125.141]: 192.168.125.145,192.168.125.146,192.168.125.147
- Enter RDAF infrastructure service
HAProxy(load-balancer) host(s) ip address or DNS name. For HA configuration, please enter comma separated values. Minimum of 2 hosts are required for theHAProxyHA configuration. If it is a non-HA deployment, only one RDAFHAProxyservice host's ip address or DNS name is required.
What is the host on which you want HAProxy to be provisioned?
HAProxy host[192.168.125.141]: 192.168.125.145,192.168.125.146
- Select the network interface name which is used for UI portal access. Ex:
eth0orens160etc.
What is the network interface on which you want the rdaf to be accessible externally?
Advertised external interface[eth0]: ens160
- Enter the
HAProxyservice's virtual IP address when it is configured in HA configuration. Virtual IP address should be an unused IP address. This step is not applicable whenHAProxyservice is deployed in non-HA configuration.
What is the host on which you want the platform to be externally accessible?
Advertised external host[192.168.125.143]: 192.168.125.149
After entering the required inputs as above, rdaf setup generates self-signed SSL certificates, creates the required directory structure, configures SSH key based authentication on all of the RDAF hosts and generates rdaf.cfg configuration file under /opt/rdaf directory.
It creates the below director structure on all of the RDAF hosts.
- /opt/rdaf/cert: It contains the generated self-signed SSL certificates for all of the RDAF hosts.
- /opt/rdaf/config: It contains the required configuration file for each deployed RDAF service where applicable.
- /opt/rdaf/data: It contains the persistent data for some of the RDAF services.
- /opt/rdaf/deployment-scripts: It contains the docker-compose
.ymlfile of the services that are configured to be provisioned on RDAF host. - /opt/rdaf/logs: It contains the RDAF services log files.
1.3.3 rdafk8s infra
rdafk8s infra command is used to deploy and manage RDAF infrastructure services on Kubernetes Cluster. Run the below command to view available CLI options.
usage: infra [-h] [--debug] {status,install,upgrade} ...
Manage infra services
positional arguments:
{status,install,upgrade}
status Status of the RDAF Infra
install Install the RDAF Infra containers
upgrade Upgrade the RDAF Infra containers
optional arguments:
-h, --help show this help message and exit
--debug Enable debug logs for the CLI operations
1.3.3.1 Install infra services
rdafk8s infra install command is used to deploy / install RDAF infrastructure services on Kubernetes Cluster. Run the below command to view the available CLI options.
usage: infra install [-h] --tag TAG [--service SERVICES]
optional arguments:
-h, --help show this help message and exit
--tag TAG Tag to use for the docker images of the infra components
--service SERVICES Restrict the scope of the command to a specific service
Run the below command to deploy all RDAF infrastructure services. (Note: Below shown tag name is a sample one for a reference only, for actual tag, please contact CloudFabrix support team at support@cloudfabrix.com.)
Run the below command to install a specific RDAF infrastructure service. Below are the supported infrastructure services. (Note: Below shown tag name is a sample one for a reference only, for actual tag, please contact CloudFabrix support team at support@cloudfabrix.com)
- haproxy
- nats
- mariadb
- opensearch
- kafka
- zookeeper
- redis
- graphdb
1.3.3.2 Status check
Run the below command to see the status of all of the deployed RDAF infrastructure services.
+-------------------+----------------+-------------------+--------------+-----------------------------+
| Name | Host | Status | Container Id |Tag |
+-------------------+----------------+-------------------+--------------+-----------------------------+
| haproxy | 192.168.125.45 | Up 2 minutes | 9ccc3d80a6f9 | 1.0.3.1 |
| keepalived | 192.168.125.45 | Not Provisioned | N/A | NA |
| rda-nats | 192.168.125.45 | Up 2 Minutes ago | 933d71b962ff | 1.0.3 |
| rda-nats | 192.168.125.45 | Up 2 Minutes ago | 882444e853b9 | 1.0.3 |
| rda-nats | 192.168.125.45 | Up 2 Minutes ago | a4176f63b854 | 1.0.3 |
| rda-minio | 192.168.125.45 | Up 2 Minutes ago | 3627f42cb432 | RELEASE.2023-09-30T07-02-29Z |
| rda-mariadb | 192.168.125.45 | Up 1 Minutes ago | 44451eb942d2 | 1.0.3 |
| rda-opensearch | 192.168.125.45 | Up 1 Minutes ago | c12e204d6059 | 1.0.3 |
| rda-kafka | 192.168.125.45 | Up 43 Seconds ago | 273fd9905f29 | 1.0.3 |
| rda-redis-master | 192.168.125.45 | Up 1 Minutes ago | 12a82ce2769b | 1.0.3 |
| rda-redis-replica | 192.168.125.45 | Up 59 Seconds ago | b2854fd8d563 | 1.0.3 |
+-------------------+--------------+-------------------+--------------+------------------------------+
Below are the Kubernetes Cluster kubectl get pods commands to check the status of RDA Fabric infrastructure services.
NAME READY STATUS RESTARTS AGE
opensearch-cluster-master-0 1/1 Running 0 7m12s
rda-kafka-0 1/1 Running 2 (5m26s ago) 7m10s
rda-kafka-zookeeper-0 1/1 Running 0 7m10s
rda-mariadb-mariadb-galera-0 1/1 Running 0 8m8s
rda-minio-65f755bb5f-9tjdm 1/1 Running 0 8m25s
rda-nats-0 3/3 Running 0 8m57s
rda-nats-box-7b7b46969b-79lnf 1/1 Running 0 8m57s
rda-redis-master-0 1/1 Running 0 7m9s
rda-redis-replicas-0 1/1 Running 0 7m9s
Below kubectl get pods command provides additional details of deployed RDAF Infrastructure services (PODs) along with their worker node(s) on which they were deployed.
In order to get detailed status of the each RDAF Infrastructure service POD, run the below kubectl describe pod command.
Name: rda-nats-0
Namespace: rda-fabric
Priority: 0
Node: k8rdapfm01/10.95.125.45
Start Time: Sun, 12 Feb 2023 00:36:39 +0000
Labels: app=rda-fabric-services
app_category=rdaf-infra
app_component=rda-nats
controller-revision-hash=rda-nats-64747cd755
...
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned rda-fabric/rda-nats-0 to k8rdapfm01
Normal Pulling 10m kubelet Pulling image "10.95.125.140:5000/rda-platform-nats:1.0.2"
Normal Pulled 10m kubelet Successfully pulled image "10.95.125.140:5000/rda-platform-nats:1.0.2" in 3.102792187s
Normal Created 10m kubelet Created container nats
1.3.3.3 Restart infra services
Run the below command to restart one of the RDAF infrastructure services.
Run the below commands to restart more than one of RDAF infrastructure services.
kubectl delete pod <rdaf-infrastructure-pod-name1> <rdaf-infrastructure-pod-name2> ... -n rda-fabric
Above kubectl delete pod command will stop and delete existing RDAF Infrastructure service POD and will redeploy the service.
Danger
Restarting RDAF infrastructure service or services is a disruptive operation which will impact all of the RDAF dependant services and causes a downtime. When RDAF platform is deployed in Production environment, please perform these operations only during a scheduled downtime.
1.3.3.4 Upgrade infra services
Run the below command to upgrade all RDAF infrastructure services to a newer version.
Run the below command to upgrade a specific RDAF infrastructure service to a newer version.
Tip
Above shown tag version is a sample one and for a reference only, for actual newer versioned tag, please contact CloudFabrix support team at support@cloudfabrix.com
Below are the supported RDAF Infrastructure services.
- haproxy
- nats
- mariadb
- opensearch
- kafka
- zookeeper
- redis
- graphdb
Danger
Please take full configuration and data backup of RDAF platform before any upgrade process. Upgrading RDAF infrastructure service or services is a disruptive operation which will impact all of the RDAF dependant services and causes a downtime. When RDAF platform is deployed in Production environment, please perform upgrade operation only during a scheduled downtime.
1.3.4 rdak8s platform
rdafk8s platform command is used to deploy and manage RDAF core platform services on Kubernetes Cluster. Run the below command to view available CLI options.
usage: platform [-h] [--debug] {} ...
Manage the RDAF Platform
positional arguments:
{} commands
status Status of the RDAF Platform
install Install the RDAF platform containers
upgrade Upgrade the RDAF platform containers
generate-certs Generate certificates for hosts belonging to this
installation
reset-admin-user
reset the password of user
optional arguments:
-h, --help show this help message and exit
--debug Enable debug logs for the CLI operations
1.3.4.1 Install platform services
rdafk8s platform install command is used to deploy / install RDAF core platform services. Run the below command to view the available CLI options.
usage: platform install [-h] --tag TAG [--service SERVICES]
optional arguments:
-h, --help show this help message and exit
--tag TAG Tag to use for the docker images of the platform
components
--service SERVICES Restrict the scope of the command to specific service
Run the below command to deploy all RDAF core platform services. (Note: Below shown tag name is a sample one for a reference only, for actual tag, please contact CloudFabrix support team at support@cloudfabrix.com)
As part of the installation of RDAF core platform services, it creates a default tenant admin user called admin@cfx.com
The default password for admin@cfx.com is admin1234
On first login onto RDAF UI portal, it prompts for resetting the above default password to user's choice.
In order to access RDAF UI portal, open a web browser and type the HAProxy server's IP address if it is a non-HA deployment or HAProxy server's virtual IP address if it is an HA deployment as shown below.
1.3.4.2 Status check
Run the below command to see the status of all of the deployed RDAF core platform services.
+--------------------------+---------------+-------------------+--------------+---------------------+
| Name | Host | Status | Container Id | Tag |
+--------------------------+---------------+-------------------+--------------+---------------------+
| cfx-rda-access-manager | 192.168.125.141 | Up 2 minutes | 7da846eb3d79 | 3.4.1 |
| cfx-rda-resource-manager | 192.168.125.141 | Up 2 minutes | 46f7cccc2ad4 | 3.4.1 |
| cfx-rda-user-preferences | 192.168.125.141 | Up 2 minutes | 76e3ca13d497 | 3.4.1 |
| portal-backend | 192.168.125.141 | Up About a minute | 9f0a7f6eb201 | 3.4.1 |
| portal-frontend | 192.168.125.141 | Up About a minute | f379b1cd5a6b | 3.4.1 |
| rda_api_server | 192.168.125.141 | Up About a minute | c6f0bb5d8a93 | 3.4.1 |
| rda_asm | 192.168.125.141 | Up About a minute | ff845c5e9298 | 3.4.1 |
| rda_asset_dependency | 192.168.125.141 | Up About a minute | e7de8b4ad2f3 | 3.4.1 |
| rda_collector | 192.168.125.141 | Up About a minute | 06aa08a80041 | 3.4.1 |
| rda_identity | 192.168.125.141 | Up About a minute | 9c396d11d111 | 3.4.1 |
| rda_registry | 192.168.125.141 | Up About a minute | 14be8c7db66c | 3.4.1 |
| rda_sched_admin | 192.168.125.141 | Up About a minute | 7e254ab6f70f | 3.4.1 |
| rda_scheduler | 192.168.125.141 | Up About a minute | a48e28b84512 | 3.4.1 |
+--------------------------+---------------+-------------------+--------------+---------------------+
Below are the Kubernetes Cluster kubectl commands to check the status of RDA Fabric core platform services.
NAME READY STATUS RESTARTS AGE
rda-access-manager-668d68bb67-tks95 1/1 Running 0 11d
rda-api-server-b6c888bdd-nv4jm 1/1 Running 0 11d
rda-asset-dependency-7969f7b657-cmx24 1/1 Running 0 11d
rda-collector-6bd6c79475-52hvg 1/1 Running 0 11d
rda-identity-679864f487-74xtd 1/1 Running 0 11d
rda-portal-59dbd8cc6d-b2sfn 2/2 Running 0 11d
rda-registry-7767f58949-jw6s8 1/1 Running 0 11d
rda-resource-manager-84c9995887-bmzw4 1/1 Running 0 11d
rda-scheduler-5b87b9798f-fxztt 1/1 Running 0 11d
rda-user-preferences-7469dfb75d-b4l5r 1/1 Running 0 11d
Below kubectl get pods command provides additional details of deployed RDAF core platform services (PODs) along with their worker node(s) on which they were deployed.
In order to get detailed status of the each RDAF core platform service POD, run the below kubectl describe pod command.
Name: rda-collector-6bd6c79475-52hvg
Namespace: rda-fabric
Priority: 0
Node: hari-k8-cluster-infra10819/10.95.108.19
Start Time: Tue, 31 Jan 2023 05:00:57 +0000
Labels: app=rda-fabric-services
app_category=rdaf-platform
app_component=rda-collector
pod-template-hash=6bd6c79475
...
...
QoS Class: Burstable
Node-Selectors: rdaf_platform_services=allow
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
pod-type=rda-tenant:NoSchedule
Events: <none>
1.3.4.3 Upgrade platform services
Run the below command to upgrade all RDAF core platform services to a newer version.
Below are the RDAF core platform services
- cfx-rda-access-manager
- cfx-rda-resource-manager
- cfx-rda-user-preferences
- portal-backend
- portal-frontend
- rda-api-server
- rda-asm
- rda-asset-dependency
- rda-collector
- rda-identity
- rda-registry
- rda-scheduler
Run the below command to upgrade a specific RDAF core platform service to a newer version.
Tip
Above shown tag version is a sample one and for a reference only, for actual newer versioned tag, please contact CloudFabrix support team at support@cloudfabrix.com
Danger
Please take full configuration and data backup of RDAF platform before any upgrade process. Upgrading RDAF core platform service or services is a disruptive operation which will impact all of the RDAF dependant services and causes a downtime. When RDAF platform is deployed in Production environment, please perform upgrade operation only during a scheduled downtime.
1.3.4.4 Restarting platform services
Run the below command to restart one of the RDAF core platform services.
Run the below commands to restart more than one of RDAF core platform services.
Above kubectl delete pod command will stop and delete existing RDAF core platform service POD and will redeploy the service.
Danger
Restarting RDAF core platform service or services is a disruptive operation which will impact all of the RDAF dependant services and causes a downtime. When RDAF platform is deployed in Production environment, please perform these operations only during a scheduled downtime.
1.3.4.5 Start/Stop platform services
Run the below command to get all of the deployed RDAF core platform services.
Run the below command to stop one of the RDAF core platform services.
Above kubectl scale --replicas=0 command will stop and terminate selected RDAF core platform service POD.
Danger
Stopping RDAF core platform service or services is a disruptive operation which will impact all of the RDAF dependant services and causes a downtime. When RDAF platform is deployed in Production environment, please perform these operations only during a scheduled downtime.
Run the below command to start one of the RDAF core platform services.
Above kubectl scale --replicas=1 command will deploy and start the selected RDAF core platform service POD.
Note
When using above command, please select replicas value as 1 for non-HA deployments and 2 for HA deployments.
1.3.4.6 Reset password
Run the below command to reset the default user's admin@cfx.com password to factory default. i.e. admin1234 and will force the user to reset the default password to tenant admin user's choice.
Warning
Use above command option only in a scenario where tenant admin users are not able to access RDAF UI portal because of external IAM (identity and access management) tool such as Active Directory / LDAP / SSO is down or not accessible and default tenant admin user's password is forgotten or lost.
1.3.4.7 Generate SSL Certificates
Self-signed SSL certificates are used for RDAF infrastructure, core platform services and for RDAF CLI as well. This manual step is not usually needed as it will be run automatically during rdafk8s setup execution.
However, this command is useful to re-generate self-signed SSL certificates and overwrite existing ones if there is a need.
After re-generating the SSL certificates, please restart RDAF infrastructure, core platform, application, worker and agent services.
Danger
Re-generating self-signed SSL certificates is a disruptive operation which will impact all of the RDAF dependant services and causes a downtime. When RDAF platform is deployed in Production environment, please perform these operations only during a scheduled downtime.
1.3.5 rdafk8s app
rdafk8s app command is used to deploy and manage RDAF application services. Run the below command to view available CLI options.
The supported application services are below.
- OIA: Operations Intelligence and Analytics (Also known as AIOps)
- AIA: Asset Intelligence and Analytics
usage: ('app',) [-h] [--debug] {} ...
Manage the RDAF Apps
positional arguments:
{} commands
status Status of the App
install Install the App service containers
upgrade Upgrade the App service containers
update-config
Updated configurations of one or more components
optional arguments:
-h, --help show this help message and exit
--debug Enable debug logs for the CLI operations
1.3.5.1 Install OIA/AIA services
rdafk8s app install command is used to deploy / install RDAF OIA/AIA application services. Run the below command to view the available CLI options.
usage: ('app',) install [-h] --tag TAG [--service SERVICES] {AIA,OIA}
positional arguments:
{AIA,OIA} Select the APP to act on
optional arguments:
-h, --help show this help message and exit
--tag TAG Tag to use for the docker images of the app components
--service SERVICES Restrict the scope of the command to specific service
Run the below command to deploy RDAF OIA / AIA application services. (Note: Below shown tag name is a sample one for a reference only, for actual tag, please contact CloudFabrix support team at support@cloudfabrix.com)
1.3.5.2 Restarting app services
Run the below command to restart one of the RDAF application (OIA/AIA) services.
Run the below commands to restart more than one of RDAF application (OIA/AIA) services.
Above kubectl delete pod command will stop and delete existing RDAF application service POD and will redeploy the service.
Below are the RDAF OIA application services
- all-alerts-cfx-rda-dataset-caas
- cfx-rda-alert-ingester
- cfx-rda-alert-processor
- cfx-rda-app-builder
- cfx-rda-app-controller
- cfx-rda-collaboration
- cfx-rda-configuration-service
- cfx-rda-event-consumer
- cfx-rda-file-browser
- cfx-rda-ingestion-tracker
- cfx-rda-irm-service
- cfx-rda-ml-config
- cfx-rda-notification-service
- cfx-rda-reports-registry
- cfx-rda-smtp-server
- cfx-rda-webhook-server
- current-alerts-cfx-rda-dataset-caas
Danger
Stopping and Starting RDAF application OIA / AIA service or services is a disruptive operation which will impact the availability of these application services. When RDAF platform is deployed in Production environment, please perform these operations only during a scheduled downtime.
1.3.5.3 Start/Stop application services
Run the below command to get all of the deployed RDAF OIA/AIA application services.
Run the below command to stop one of the RDAF core platform services.
Above kubectl scale --replicas=0 command will stop and terminate selected RDAF application service POD.
Danger
Stopping RDAF application service or services is a disruptive operation which will impact all of the RDAF dependant services and causes a downtime. When RDAF platform is deployed in Production environment, please perform these operations only during a scheduled downtime.
Run the below command to start one of the RDAF OIA/AIA application services.
Above kubectl scale --replicas=1 command will deploy and start the selected RDAF application service POD.
Note
When using above command, please select replicas value as 1 for non-HA deployments and 2 for HA deployments.
1.3.5.4 Status check
Run the below command to see the status of all of the deployed RDAF application services.
+------------------------------+-----------------+---------------+--------------+---------------------+
| Name | Host | Status | Container Id | Tag |
+------------------------------+-----------------+---------------+--------------+---------------------+
| all-alerts-cfx-rda-dataset- | 192.168.125.146 | Up 17 seconds | 83ac2a138b83 | 7.4.1 |
| caas | | | | |
| cfx-rda-alert-ingester | 192.168.125.146 | Up 16 seconds | 9fc56e92002d | 7.4.1 |
| cfx-rda-alert-processor | 192.168.125.146 | Up 15 seconds | 9880d500527b | 7.4.1 |
| cfx-rda-app-builder | 192.168.125.146 | Up 15 seconds | baa67fffafa7 | 7.4.1 |
| cfx-rda-app-controller | 192.168.125.146 | Up 14 seconds | 2d0728cbdba7 | 7.4.1 |
| cfx-rda-collaboration | 192.168.125.146 | Up 13 seconds | 4ba81174c926 | 7.4.1 |
| cfx-rda-configuration- | 192.168.125.146 | Up 12 seconds | 33f1e3379a6c | 7.4.1 |
| service | | | | |
| cfx-rda-event-consumer | 192.168.125.146 | Up 12 seconds | fa3e19f6f6b7 | 7.4.1 |
| cfx-rda-file-browser | 192.168.125.146 | Up 11 seconds | 58cad20da8a5 | 7.4.1 |
| cfx-rda-ingestion-tracker | 192.168.125.146 | Up 10 seconds | 9bab4b1b0be4 | 7.4.1 |
| cfx-rda-irm-service | 192.168.125.146 | Up 10 seconds | c62128cc5e2d | 7.4.1 |
| cfx-rda-ml-config | 192.168.125.146 | Up 9 seconds | 3abf1f54ddd7 | 7.4.1 |
| cfx-rda-notification-service | 192.168.125.146 | Up 8 seconds | db28a2065906 | 7.4.1 |
| cfx-rda-reports-registry | 192.168.125.146 | Up 7 seconds | 2c0586f91742 | 7.4.1 |
| cfx-rda-smtp-server | 192.168.125.146 | Up 6 seconds | 5847535b72bc | 7.4.1 |
| cfx-rda-webhook-server | 192.168.125.146 | Up 5 seconds | 69acfb72fdc3 | 7.4.1 |
| current-alerts-cfx-rda- | 192.168.125.146 | Up 4 seconds | de9e818435e3 | 7.4.1 |
| dataset-caas | | | | |
+------------------------------+-----------------+---------------+--------------+---------------------+
Below are the Kubernetes Cluster kubectl get pods commands to check the status of RDA Fabric application (OIA/AIA) services.
NAME READY STATUS RESTARTS AGE
rda-alert-ingester-697b867778-9b9bw 1/1 Running 0 11d
rda-alert-processor-f4c4f844c-h59qj 1/1 Running 0 11d
rda-app-controller-5f866cdc94-9d6gx 1/1 Running 0 11d
rda-collaboration-547598cfcf-k6vmt 1/1 Running 0 11d
rda-configuration-service-764d48db8c-94vz5 1/1 Running 0 11d
rda-dataset-caas-all-alerts-7fdf7df6d9-8nkh8 1/1 Running 0 11d
rda-dataset-caas-current-alerts-766bd95dd5-h5tlb 1/1 Running 0 11d
rda-event-consumer-64f7d96b46-h9jzw 1/1 Running 0 11d
rda-file-browser-7f4cd9764c-p99fg 1/1 Running 0 11d
rda-ingestion-tracker-688459ddc4-c9zs7 1/1 Running 0 11d
rda-irm-service-5b754f9687-bv5kw 1/1 Running 0 11d
rda-ml-config-585cd7fd6d-42pzn 1/1 Running 0 11d
rda-notification-service-7798bd7b9d-sdlcn 1/1 Running 0 11d
rda-reports-registry-64474997fd-7wp9c 1/1 Running 0 11d
rda-smtp-server-84995684dc-7c8qd 1/1 Running 0 11d
rda-webhook-server-59c5654b95-mdhhf 1/1 Running 0 11d
Below kubectl get pods command provides additional details of deployed RDAF application (OIA/AIA) services (PODs) along with their worker node(s) on which they were deployed.
In order to get detailed status of the each RDAF application (OIA/AIA) service POD, run the below kubectl describe pod command.
Name: rda-webhook-server-59c5654b95-mdhhf
Namespace: rda-fabric
Priority: 0
Node: hari-k8-cluster-infra10822/10.95.108.22
Start Time: Tue, 31 Jan 2023 05:41:40 +0000
Labels: app=rda-fabric-services
app_category=rdaf-application
app_component=rda-webhook-server
pod-template-hash=59c5654b95
...
...
QoS Class: Burstable
Node-Selectors: rdaf_application_services=allow
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
pod-type=rda-tenant:NoSchedule
Events: <none>
1.3.5.5 Upgrade app OIA/AIA services
Run the below command to upgrade all RDAF OIA / AIA application services to a newer version.
Below are the RDAF OIA application services
- all-alerts-cfx-rda-dataset-caas
- cfx-rda-alert-ingester
- cfx-rda-alert-processor
- cfx-rda-app-builder
- cfx-rda-app-controller
- cfx-rda-collaboration
- cfx-rda-configuration-service
- cfx-rda-event-consumer
- cfx-rda-file-browser
- cfx-rda-ingestion-tracker
- cfx-rda-irm-service
- cfx-rda-ml-config
- cfx-rda-notification-service
- cfx-rda-reports-registry
- cfx-rda-smtp-server
- cfx-rda-webhook-server
- current-alerts-cfx-rda-dataset-caas
Run the below command to upgrade a specific RDAF OIA application service to a newer version.
Tip
Above shown tag version is a sample one and for a reference only, for actual newer versioned tag, please contact CloudFabrix support team at support@cloudfabrix.com
Danger
Please take full configuration and data backup of RDAF platform before any upgrade process. Upgrading RDAF OIA / AIA application service or services is a disruptive operation which will impact the availability of these services. When RDAF platform is deployed in Production environment, please perform upgrade operation only during a scheduled downtime.
1.3.5.6 Update HAProxy configuration
Run the below command to update the necessary HAProxy load-balancer configuration for RDAF OIA / AIA application services.
After deploying the RDAF OIA application services, it is mandatory to run the rdaf app update-config which will apply and restart the HAProxy load-balancer service automatically.
1.3.6 rdafk8s worker
rdafk8s worker command is used to deploy and manage RDAF worker services. Run the below command to view available CLI options.
usage: worker [-h] [--debug] {} ...
Manage the RDAF Worker
positional arguments:
{} commands
status Status of the RDAF Worker
install Install the RDAF Worker containers
upgrade Upgrade the RDAF Worker containers
optional arguments:
-h, --help show this help message and exit
--debug Enable debug logs for the CLI operations
1.3.6.1 Install worker service(s)
rdafk8s worker install command is used to deploy / install RDAF worker services. Run the below command to view the available CLI options.
usage: worker install [-h] --tag TAG
optional arguments:
-h, --help show this help message and exit
--tag TAG Tag to use for the docker images of the worker components
Run the below command to deploy all RDAF worker services. (Note: Below shown tag name is a sample one for a reference only, for actual tag, please contact CloudFabrix support team at support@cloudfabrix.com)
1.3.6.2 Status check
Run the below command to see the status of all of the deployed RDAF worker services.
+------------+-----------------+---------------+--------------+---------------------+
| Name | Host | Status | Container Id | Tag |
+------------+-----------------+---------------+--------------+---------------------+
| rda_worker | 192.168.125.149 | Up 18 seconds | 29cdeefd9d95 | 3.4.1 |
+------------+-----------------+---------------+--------------+---------------------+
Below are the Kubernetes Cluster kubectl get pods commands to check the status of RDA Fabric worker services.
NAME READY STATUS RESTARTS AGE
rda-worker-749b977b95-cf757 1/1 Running 0 11d
rda-worker-749b977b95-xkb5w 1/1 Running 0 11d
Below kubectl get pods command provides additional details of deployed RDAF worker services (PODs) along with their worker node(s) on which they were deployed.
In order to get detailed status of the each RDAF worker service POD, run the below kubectl describe pod command.
Name: rda-worker-749b977b95-cf757
Namespace: rda-fabric
Priority: 0
Node: hari-k8-cluster-infra10820/10.95.108.20
Start Time: Tue, 31 Jan 2023 05:18:11 +0000
Labels: app=rda-fabric-services
app_category=rdaf-worker
app_component=rda-worker
pod-template-hash=749b977b95
...
...
QoS Class: Burstable
Node-Selectors: rdaf_worker_services=allow
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
pod-type=rda-tenant:NoSchedule
Events: <none>
1.3.6.3 Upgrade worker services
Run the below command to upgrade all RDAF worker service(s) to a newer version.
Tip
Above shown tag version is a sample one and for reference only, for actual newer versioned tag, please contact CloudFabrix support team at support@cloudfabrix.com
Danger
Upgrading RDAF worker service or services is a disruptive operation which will impact all of the worker jobs. When RDAF platform is deployed in Production environment, please perform upgrade operation only during a scheduled downtime.
1.3.6.4 Restarting worker services
Run the below command to restart one of the RDAF worker services.
Run the below commands to restart more than one of RDAF worker services.
Above kubectl delete pod command will stop and delete existing RDAF worker service POD and will redeploy the service.
Below is the RDAF worker service
- rda-worker
Danger
Stopping and Starting RDAF worker service(s) is a disruptive operation which will impact all of the worker jobs. When RDAF platform is deployed in Production environment, please perform these operations only during a scheduled downtime.
1.3.6.5 Start/Stop worker services
Run the below command to get all of the deployed RDAF worker services.
Run the below command to stop one of the RDAF worker services.
Above kubectl scale --replicas=0 command will stop and terminate RDAF worker service POD.
Danger
Stopping RDAF worker service(s) is a disruptive operation which will impact all of the RDAF dependant services and causes a downtime. When RDAF platform is deployed in Production environment, please perform these operations only during a scheduled downtime.
Run the below command to start one of the RDAF worker service(s).
Above kubectl scale --replicas=1 command will deploy and start the selected RDAF worker service POD.
Note
When using above command, please select replicas value as 1 or more for non-HA deployments and 2 or more for HA deployments.
1.3.9 rdafk8s rdac_cli
rdafk8s rdac_cli command allows you to install and upgrade RDA client CLI utility which interacts with RDA Fabric services and operations.
usage: rdac-cli [-h] [--debug] {} ...
Install RDAC CLI
positional arguments:
{} commands
install Install the RDAC CLI
upgrade Upgrade the RDAC CLI
optional arguments:
-h, --help show this help message and exit
--debug Enable debug logs for the CLI operations
- To install RDA client CLI, run the below command
- To upgrade RDA client CLI version, run the below command
- Run the below command to see RDA client CLI help and available subcommand options.
Run with one of the following commands
agent-bots List all bots registered by agents for the current tenant
agents List all agents for the current tenant
alert-rules Alert Rule management commands
bot-catalog-generation-from-file Generate bot catalog for given sources
bot-package Bot Package management commands
bots-by-source List bots available for given sources
check-credentials Perform credential check for one or more sources on a worker pod
checksum Compute checksums for pipeline contents locally for a given JSON file
compare Commands to compare different RDA systems using different RDA Config files
content-to-object Convert data from a column into objects
copy-to-objstore Deploy files specified in a ZIP file to the Object Store
dashboard User defined dashboard management commands
dashgroup User defined dashboard-group management commands
dataset Dataset management commands
demo Demo related commands
deployment Service Blueprints (Deployments) management commands
event-gw-status List status of all ingestion endpoints at all the event gateways
evict Evict a job from a worker pod
file-ops Perform various operations on local files
file-to-object Convert files from a column into objects
fmt-template Formatting Templates management commands
healthcheck Perform healthcheck on each of the Pods
invoke-agent-bot Invoke a bot published by an agent
jobs List all jobs for the current tenant
logarchive Logarchive management commands
object RDA Object management commands
output Get the output of a Job using jobid.
pipeline Pipeline management commands
playground Start Webserver to access RDA Playground
pods List all pods for the current tenant
project Project management commands. Projects can be used to link different tenants / projects from this RDA Fabric or a remote RDA Fabric.
pstream Persistent Stream management commands
purge-outputs Purge outputs of completed jobs
read-stream Read messages from an RDA stream
reco-engine Recommendation Engine management commands
restore Commands to restore backed-up artifacts to an RDA Platform
run Run a pipeline on a worker pod
run-get-output Run a pipeline on a worker, and Optionally, wait for the completion, get the final output
schedule Pipeline execution schedule management commands
schema Dataset Model Schema management commands
secret Credentials (Secrets) management commands
set-pod-log-level Update the logging level for a given RDA Pod.
shell Start RDA Client interactive shell
site-profile Site Profile management commands
site-summary Show summary by Site and Overall
stack Application Dependency Mapping (Stack) management commands
staging-area Staging Area based data ingestion management commands
subscription Show current CloudFabrix RDA subscription details
synthetics Data synthesizing management commands
verify-pipeline Verify the pipeline on a worker pod
viz Visualize data from a file within the console (terminal)
watch Commands to watch various streams such sas trace, logs and change notifications by microservices
web-server Start Webserver to access RDA Client data using REST APIs
worker-obj-info List all worker pods with their current Object Store configuration
write-stream Write data to the specified stream
positional arguments:
command RDA subcommand to run
optional arguments:
-h, --help show this help message and exit
Tip
Please refer RDA Client CLI Usage for detailed information.
1.3.10 rdafk8s log_monitoring
rdafk8s log_monitoring command is used to deploy and manage log monitoring services, through which the RDAF infrastructure, platform, application, and worker service logs are streamed in real-time.
As part of the log monitoring services, it installs the following services:
- Fluentbit: It is a log shipping agent that streams the logs in real-time and ingests them into the
Logstashservice. - Logstash: It is a log processing agent that normalizes and extracts key attributes from log messages, such as timestamp, severity, process name, process function, container name, etc., before ingesting them into an index store service for analytics and visualization.
Run the below command to view available CLI options.
usage: log_monitoring [-h] [--debug] {upgrade,install,status,up,down} ...
Manage the RDAF log monitoring
positional arguments:
{upgrade,install,status,up,down}
commands
upgrade Upgrade log monitoring components
install Install log monitoring components
status Status of the RDAF log monitoring
up Create the RDAF log monitoring Containers
down Delete the RDAF log monitoring Containers
optional arguments:
-h, --help show this help message and exit
--debug Enable debug logs for the CLI operations
1.3.10.1 Install Log Monitoring
rdafk8s log_monitoring install command is used to deploy / install RDAF log monitoring services. Run the below command to view the available CLI options.
usage: log_monitoring install [-h] --log-monitoring-host LOG_MONITORING_HOST
--tag TAG
log_monitoring install: error: the following arguments are required: --log-monitoring-host, --tag
To deploy all RDAF log monitoring services, execute the following command. Please note that it is mandatory to specify the host for the Logstash service deployment using the --log-monitoring-host option.
Note
Below shown Logstash host ip address is for a reference only. For the latest log monitoring services tag, please contact CloudFabrix support team at support@cloudfabrix.com.
{"status":"CREATED","message":"'rdaf-log-monitoring' created."}
{"status":"CREATED","message":"'role-log-monitoring' created."}
{"status":"OK","message":"'rdaf-log-monitoring' updated."}
{"status":"CREATED","message":"'role-log-monitoring' created."}
2023-10-31 09:04:33,752 [rdaf.component.log_monitoring] INFO - Creating rdaf_services_logs pstream...
{
"retention_days": 15,
"timestamp": "@timestamp",
"search_case_insensitive": true,
"_settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"refresh_interval": "60s"
}
}
Persistent stream saved.
2023-10-31 09:04:41,064 [rdaf.component.log_monitoring] INFO - Successfully installed and configured rdaf log monitoring...
1.3.10.2 Status check
Run the below command to see the status of all of the deployed RDAF log monitoring services.
+---------------------+----------------------+---------------------------+-------------------------+---------+
| Name | Host | Status | Container Id | Tag |
+---------------------+----------------------+---------------------------+-------------------------+---------+
| rda-fluentbit | 10.95.125.52 | Up 18 Hours ago | 92e83de200a2 | 1.0.2 |
| rda-logstash | 10.95.125.52 | Up 18 Hours ago | 00c1929fdea4 | 1.0.2 |
+---------------------+----------------------+---------------------------+-------------------------+---------+
Below are the Kubernetes Cluster commands to check the status of RDA Fabric log monitoring services.
---------------------------------------------------------------------------------------------------------------------------
rda-logstash-796f9d9f75-vchpq 1/1 Running 0 14m
rda-fluentbit-9clrn 1/1 Running 0 15m
---------------------------------------------------------------------------------------------------------------------------
Run the following commands to obtain additional details of the deployed RDAF log monitoring services (PODs), including information about the node(s) on which they were deployed.
To get the detailed status of each RDAF log monitoring service POD, run the below command.
---------------------------------------------------------------------------------------------------------------------------
Name: rda-logstash-796f9d9f75-vchpq
Namespace: rda-fabric
Priority: 0
Node: k8workernode/10.95.125.52
Start Time: Tue, 31 Oct 2023 09:04:40 +0000
Labels: app=rda-fabric-services
app_category=rdaf-infra
app_component=rda-logstash
pod-template-hash=796f9d9f75
Annotations: cni.projectcalico.org/containerID: 0e4208d848db8c99fb977e5bf3f32876b31a090b29e632385dddb2c7cbed599c
cni.projectcalico.org/podIP: 192.168.75.76/32
cni.projectcalico.org/podIPs: 192.168.75.76/32
Status: Running
IP: 192.168.75.76
IPs:
IP: 192.168.75.76
...
...
QoS Class: Burstable
Node-Selectors: rdaf_monitoring_services=allow
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
pod-type=rda-tenant:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16m default-scheduler Successfully assigned rda-fabric/rda-logstash-796f9d9f75-vchpq to k8workernode
Normal Pulling 16m kubelet Pulling image "cfxregistry.cloudfabrix.io:443/rda-platform-logstash:1.0.2"
Normal Pulled 15m kubelet Successfully pulled image "cfxregistry.cloudfabrix.io:443/rda-platform-logstash:1.0.2" in 1m6.675838828s
Normal Created 15m kubelet Created container rda-logstash
Normal Started 15m kubelet Started container rda-logstash
------------------------------------------------------------------------------------------------------------------------------------------
1.3.10.3 Upgrade Log Monitoring
Run the below command to upgrade all RDAF log monitoring to a newer version.
1.3.10.4 Restart Log Monitoring services
Run the below command to restart one of the RDAF log monitoring services.
Run the below command to restart more than one of RDAF log monitoring services.
Restarting the log monitoring service using rdafk8s CLI commands.
a) To Stop
Run the below command to Stop all RDAF log monitoring services.
---------------------------------------------------------------------------------------------------------------------------
2023-11-01 05:28:21,425 [rdaf.component.log_monitoring] INFO - Deleting service: rda-logstash
deployment.apps/rda-logstash scaled
2023-11-01 05:28:21,527 [rdaf.component.log_monitoring] INFO - Deleting service: rda-logstash
release "rda-fluentbit" uninstalled
---------------------------------------------------------------------------------------------------------------------------
Run the below command to Start all RDAF log monitoring services.
1.3.10.5 Add Log Monitoring dashboard
Login to RDAF UI portal as MSP admin user.
Go to Main Menu --> Configuration --> RDA Administration --> Dashboards --> User Dashboards --> Click on Add and create a new dashboard by copying the below Dashboard configuration for RDAF log monitoring services.
{
"name": "rdaf-platform-log-analytics",
"label": "RDAF Platform Logs",
"description": "RDAF Platform service's log analysis dashboard",
"version": "23.01.14.1",
"enabled": true,
"dashboard_style": "tabbed",
"status_poller": {
"stream": "rdaf_services_logs",
"frequency": 15,
"columns": [
"@timestamp"
],
"sorting": [
{
"@timestamp": "desc"
}
],
"query": "`@timestamp` is after '${timestamp}'",
"defaults": {
"@timestamp": "$UTCNOW"
},
"action": "refresh"
},
"dashboard_filters": {
"time_filter": true,
"columns_filter": [
{
"id": "@timestamp",
"label": "Timestamp",
"type": "DATETIME"
},
{
"id": "service_name",
"label": "Service Name",
"type": "TEXT"
},
{
"id": "service_category",
"label": "Service Category",
"type": "TEXT"
},
{
"id": "log_severity",
"label": "Log Severity",
"type": "TEXT"
},
{
"id": "log",
"label": "Log Message",
"type": "TEXT"
},
{
"id": "log.text",
"label": "Log Message Text",
"type": "SIMPLE_TEXT"
},
{
"id": "process_name",
"label": "Process Name",
"type": "TEXT"
},
{
"id": "process_function",
"label": "Process Function",
"type": "TEXT"
},
{
"id": "thread_id",
"label": "Thread ID",
"type": "TEXT"
},
{
"id": "k8s_pod_name",
"label": "POD Name",
"type": "TEXT"
},
{
"id": "k8s_container_name",
"label": "Container Name",
"type": "TEXT"
}
],
"group_filters": [
{
"stream": "rdaf_services_logs",
"title": "Log Severity",
"group_by": [
"log_severity"
],
"ts_column": "@timestamp",
"agg": "value_count",
"column": "_id",
"type": "int"
},
{
"stream": "rdaf_services_logs",
"title": "Service Name",
"group_by": [
"service_name"
],
"ts_column": "@timestamp",
"limit": 50,
"agg": "value_count",
"column": "_id",
"type": "int"
},
{
"stream": "rdaf_services_logs",
"title": "Service Category",
"group_by": [
"service_category"
],
"ts_column": "@timestamp",
"agg": "value_count",
"column": "_id",
"type": "int"
},
{
"stream": "rdaf_services_logs",
"title": "POD Name",
"group_by": [
"k8s_pod_name"
],
"ts_column": "@timestamp",
"agg": "value_count",
"limit": 200,
"column": "_id",
"type": "int"
}
]
},
"dashboard_sections": [
{
"title": "Overall Summary",
"show_filter": true,
"widgets": [
{
"title": "Log Severity Trend",
"widget_type": "timeseries",
"stream": "rdaf_services_logs",
"ts_column": "@timestamp",
"max_width": 12,
"height": 3,
"min_width": 12,
"chartProperties": {
"yAxisLabel": "Count",
"xAxisLabel": null,
"legendLocation": "bottom"
},
"interval": "15Min",
"group_by": [
"log_severity"
],
"series_spec": [
{
"column": "log_severity",
"agg": "value_count",
"type": "int"
}
],
"widget_id": "06413884"
},
{
"widget_type": "pie_chart",
"title": "Logs by Severity",
"stream": "rdaf_services_logs",
"ts_column": "@timestamp",
"column": "_id",
"agg": "value_count",
"group_by": [
"log_severity"
],
"type": "str",
"style": {
"color-map": {
"ERROR": [
"#ef5350",
"#ffffff"
],
"WARNING": [
"#FFA726",
"#ffffff"
],
"INFO": [
"#388e3c",
"#ffffff"
],
"DEBUG": [
"#000000",
"#ffffff"
],
"UNKNOWN": [
"#bcaaa4",
"#ffffff"
]
}
},
"min_width": 4,
"height": 4,
"max_width": 4,
"widget_id": "b2ffa8e9"
},
{
"widget_type": "pie_chart",
"title": "Logs by RDA Host IP",
"stream": "rdaf_services_logs",
"ts_column": "@timestamp",
"column": "_id",
"agg": "value_count",
"group_by": [
"host"
],
"type": "str",
"min_width": 4,
"height": 4,
"max_width": 4,
"widget_id": "a4f2d8bd"
},
{
"widget_type": "pie_chart",
"title": "Logs by Service Category",
"stream": "rdaf_services_logs",
"ts_column": "@timestamp",
"column": "_id",
"agg": "value_count",
"group_by": [
"service_category"
],
"type": "str",
"min_width": 4,
"height": 4,
"max_width": 4,
"widget_id": "89ac5ce9"
},
{
"widget_type": "pie_chart",
"title": "Logs by Service Name",
"stream": "rdaf_services_logs",
"ts_column": "@timestamp",
"column": "_id",
"agg": "value_count",
"group_by": "service_name",
"type": "int",
"min_width": 4,
"height": 4,
"max_width": 4,
"widget_id": "4b267fce"
}
]
},
{
"title": "App Services",
"show_filter": true,
"widgets": [
{
"widget_type": "tabular",
"title": "Log Messages",
"stream": "rdaf_services_logs",
"extra_filter": "service_category in ['rda_app_svcs', 'rda_pfm_svcs']",
"ts_column": "@timestamp",
"sorting": [
{
"@timestamp": "desc"
}
],
"columns": {
"@timestamp": {
"title": "Timestamp",
"type": "DATETIME"
},
"state_color2": {
"type": "COLOR-MAP",
"source-column": "log_severity",
"color-map": {
"INFO": "#388e3c",
"ERROR": "#ef5350",
"WARNING": "#ffa726",
"DEBUG": "#000000"
}
},
"log_severity": {
"title": "Severity",
"htmlTemplateForRow": "<span class='badge' style='background-color: {{ row.state_color2 }}' > {{ row.log_severity }} </span>"
},
"service_name": "Service Name",
"process_name": "Process Name",
"process_function": "Process Function",
"log": "Message"
},
"widget_id": "6895c8f0"
}
]
},
{
"title": "Infra Services",
"show_filter": true,
"widgets": [
{
"widget_type": "tabular",
"title": "Log Messages",
"stream": "rdaf_services_logs",
"extra_filter": "service_category in ['rda_infra_svcs']",
"ts_column": "@timestamp",
"sorting": [
{
"@timestamp": "desc"
}
],
"columns": {
"@timestamp": {
"title": "Timestamp",
"type": "DATETIME"
},
"log_severity": {
"title": "Severity",
"htmlTemplateForRow": "<span class='badge' style='background-color: {{ row.state_color2 }}' > {{ row.log_severity }} </span>"
},
"state_color2": {
"type": "COLOR-MAP",
"source-column": "log_severity",
"color-map": {
"INFO": "#388e3c",
"ERROR": "#ef5350",
"WARNING": "#ffa726",
"DEBUG": "#000000",
"UNKNOWN": "#bcaaa4"
}
},
"service_name": "Service Name",
"process_name": "Process Name",
"log": "Message",
"minio_object": "Minio Object"
},
"widget_id": "98f10587"
}
]
}
]
}
1.3.11 rdafk8s reset
rdafk8s reset command allows the user to reset the RDAF platform configuration by performing the below operations.
- Stop RDAF application, worker, platform & infrastructure services
- Delete RDAF application, worker, platform & infrastructure services and its data
- Delete all Docker images and volumes RDAF application, worker, platform & infrastructure services
- Delete RDAF platform configuration
Danger
rdafk8s reset command is a disruptive operation as it clears entire RDAF platform footprint. It's primary purpose is to use only in Demo or POC environments ("NOT" in Production) where it requires to re-install entire RDAF platform from scratch.