Guide to Install and Configure RDA Fabric Edge Services

1. Install RDA Fabric Edge Services

RDA Worker:

RDA worker nodes are stateless data processing entities that can be installed closed to the source of data generation (ex: on-prem/enterprise/edge sites etc.). Worker nodes execute bots and pipelines and communicate with the RDAF platform that is responsible for scheduling and orchestrating jobs (pipelines) across various worker nodes.

Using worker nodes you can ingest and process/transform data locally without having to send all the data to centralized locations like an analytics platform or data warehouse. Two or more worker nodes in one environment can work as a group for load balancing and scale. RDAF platform can orchestrate data sharing or routing among worker nodes in distributed environments (ex: Worker nodes in edge location exchange data with workers in DC or workers in cloud).

Workers are essentially containerized service nodes and can be installed using Docker-compose or in a Kubernetes environment. Workers are typically installed on VMs that are located on-premises / cloud / edge environments.

RDA Event Gateway:

RDA Event Gateway is a type of RDA Agent that can send streaming data to the RDA Fabric platform. If user wants to send logs/events in real-time to the RDAF platform, users can install Event Gateway in their local environment and configure event sources to send data to Event Gateway.

Similar to RDA worker nodes, event gateways are also containerized services and can be installed using Docker-compose or in a Kubernetes environment. RDA event gateways are typically installed on VMs that are located on-premises / cloud / edge environments.

Log Sources: For instance, to send syslogs from your Linux servers to the RDA platform, you can install Event Gateway and configure rsyslog on your Linux servers to send data to Event Gateway, which in turn can send data to the RDAF platform.
Existing Log Shippers: Users can also use existing log shippers like Splunk Universal Forwarder, Elasticsearch beats, Fluentd, rsyslog, syslog-ng, etc. to route / send data to Event Gateway.
Endpoints: Event Gateway supports endpoints and each endpoint is configured to send data to a stream on RDAF platform. For example, you can configure an endpoint with a port and protocol/type (ex: TCP/syslog) and all syslog sources can send data to that endpoint.

RDA Edge Collector :

RDA Edge Collector is a type of RDA agent that can discover and collect IT asset data in an agentless manner and send the collected data to the RDA Fabric platform. Edge Collector agent is primarily used to discover the IT assets (i.e. Servers, Switches, Routers, Firewall, Load Balancers, Storage Arrays etc.) that provide inventory data over SNMP and SSH protocols.

Similar to RDA worker & event gateway, edge collectors are also containerized services and can be installed using Docker-compose or in a Kubernetes environment. RDA edge collectors are typically installed on VMs that are located on-premises / cloud / edge environments.

1.1 RDA Worker Installation

Prerequisites:

Linux OS
Memory - 8GB
Disk - 50GB
Python 3.7.4 or above
Docker container runtime environment (18.09.2 or above)
Docker-compose utility (1.27.x or above)

Installation Steps:

Step-1:

The RDA worker node registers and communicates with the RDAF platform using a configuration file that contains your tenant ID, data fabric access tokens, and object storage credentials.

Download RDA Fabric Configuration from the portal by going to Configuration --> Fabric Configuration --> RDA network configuration and copy it to the local filesystem where the worker node is going to be installed.

Save the file as rda_network_config.json

RDAFNetworkConfig

Create the below directory structure

sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/worker/config
sudo mkdir -p /opt/rdaf/worker/logs
sudo chown -R `id -u`:`id -g` /opt/rdaf

Copy the downloaded RDA Fabric configuration file as shown below.

cp rda_network_config.json /opt/rdaf/network_config/rda_network_config.json

Create common.yml file for RDA Worker to configure logger settings as shown below.

cd /opt/rdaf/worker/config

cat > common.yml << 'EOF'
version: 1
disable_existing_loggers: false
formatters:
  standard:
    format: "%(asctime)s %(levelname)s %(module)s - PID=%(process)s %(message)s"
handlers:
  console:
    class: logging.StreamHandler
    level: INFO
    formatter: standard
    stream: ext://sys.stdout
  file_handler:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: standard
    filename: /logs/${pod_type}-${pod_id}.log
    maxBytes: 10485760 # 10MB
    backupCount: 5
    encoding: utf8
root:
  level: INFO
  handlers: [console, file_handler]
  propogate: yes

EOF

Step-2: Docker Login

Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.

docker login -u='readonly' -p='readonly' cfxregistry.cloudfabrix.io

Step-3: Create Docker Compose File

Create docker compose configuration file for RDA Worker as shown below.

Tip

Note-1: Optionally change the worker group name (also known as Site) in the docker-compose file by updating the WORKER_GROUP value. In this example, the worker group name is specified as rda_worker_group01

Note-2: Adjust mem_limit and memswap_limit as per the workload requirements. In the below configuration, these parameters are set to 16GB

cd /opt/rdaf/worker

cat > rda-worker-docker-compose.yml <<EOF
version: '3.1'
services:
  rda_worker:
    image: cfxregistry.cloudfabrix.io/ubuntu-rda-worker-all:daily
    restart: always
    network_mode: host
    mem_limit: 16G
    memswap_limit: 16G
    shm_size: 1gb
    volumes:
    - /opt/rdaf/network_config:/network_config
    - /opt/rdaf/worker/config:/loggingConfigs
    - /opt/rdaf/worker/logs:/logs
    logging:
      driver: "json-file"
      options:
        max-size: "25m"
        max-file: "5"
    environment:
      RESOURCE_NAME:
      RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
      LOGGER_CONFIG_FILE: /loggingConfigs/common.yml
      WORKER_GROUP: rda_worker_group01
      LABELS: name=rda_worker_01
      RDA_SELF_HEALTH_RESTART_AFTER_FAILURES: 3
      CAPACITY_FILTER: mem_percent < 95

EOF

Tip

Note-1: RDA worker(s) communicates with RDA Fabric that is running in cloud or on-premise datacenter over ports 4222/TCP & 9443/TCP. Please make sure RDA worker(s) has outbound network access over these network ports. In addition, make sure RDA Fabric is configured to allow inbound network traffic for the same ports to accept the traffic from RDA worker(s).

Note-2: Please verify rda_network_config.json is configured with publicly accessible IP/FQDN of RDA Fabric for NATs and Minio endpoints.

Step-4: Bring Up RDA Worker

cd /opt/rdaf/worker

docker-compose -f rda-worker-docker-compose.yml pull 
docker-compose -f rda-worker-docker-compose.yml up --d

Step-5: Check Worker Status

Check worker node status using docker ps command and ensure that worker is up and running, without any restarts. If you see that the worker is restarting, make sure you copied the RDA network config file to the correct location.

docker ps | grep worker

Step-6: Verify RDA Worker in RDA Fabric portal

A newly installed worker will authenticate with the RDA Fabric platform and it will show up in RDA Fabric portal under Configuration --> Fabric Components --> Workers --> View Details.

Step-7: Verify Worker using RDA Client (rdac) utility

If you have installed RDA Client (rdac) command line utility, you can also verify newly created worker using rdac pods command.

1.2 RDA Event Gateway Installation

Prerequisites:

Linux OS
Memory - 8GB
Disk - 50GB
Python 3.7.4 or above
Docker container runtime environment (18.09.2 or above)
Docker-compose utility (1.27.x or above)

1.2.1 Installation Steps

Step-1:

The RDA event gateway registers and communicates with the RDA Fabirc platform using a configuration file that contains your tenant ID, data fabric access tokens, and object storage credentials.

Download RDA Fabric Configuration from the portal by going to Configuration --> Fabric Configuration --> RDA network configuration and copy it to the local filesystem where the event gateway is going to be installed.

Save the file as rda_network_config.json

RDAFNetworkConfig

Create the below directory structure

sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/event_gateway/config/main
sudo mkdir -p /opt/rdaf/event_gateway/config/snmptrap
sudo mkdir -p /opt/rdaf/event_gateway/certs
sudo mkdir -p /opt/rdaf/event_gateway/logs
sudo mkdir -p /opt/rdaf/event_gateway/log_archive
sudo chown -R `id -u`:`id -g` /opt/rdaf

Copy the downloaded RDA Fabric configuration file as shown below.

cp rda_network_config.json /opt/rdaf/network_config/rda_network_config.json

Step-2: Docker Login

Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.

docker login -u='readonly' -p='readonly' cfxregistry.cloudfabrix.io

Step-3: Create Docker Compose File

Create docker compose configuration file for RDA event gateway as shown below.

Info

Note: Optionally change the agent group name in the docker-compose file by updating the AGENT_GROUP value. In this example, the agent group name is specified as event_gateway_site01

cd /opt/rdaf/event_gateway

cat > event-gateway-docker-compose.yml <<EOF
version: '3.1'
services:
  rda_event_gateway:
    image: cfxregistry.cloudfabrix.io/ubuntu-rda-event-gateway:daily
    restart: always
    network_mode: host
    mem_limit: 6G
    memswap_limit: 6G
    volumes:
    - /opt/rdaf/network_config:/network_config
    - /opt/rdaf/event_gateway/config:/event_gw_config
    - /opt/rdaf/event_gateway/certs:/certs
    - /opt/rdaf/event_gateway/logs:/logs
    - /opt/rdaf/event_gateway/log_archive:/tmp/log_archive
    logging:
      driver: "json-file"
      options:
        max-size: "25m"
        max-file: "5"
    environment:
      RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
      EVENT_GW_MAIN_CONFIG: /event_gw_config/main/main.yml
      EVENT_GW_SNMP_TRAP_CONFIG: /event_gw_config/snmptrap/trap_template.json
      AGENT_GROUP: event_gateway_site01
      EVENT_GATEWAY_CONFIG_DIR: /event_gw_config
      LOGGER_CONFIG_FILE: /event_gw_config/main/logging.yml
      RDA_SELF_HEALTH_RESTART_AFTER_FAILURES: 3
    entrypoint: ["/docker-entry-point.sh"]
EOF

Step-4: Create SNMP Trap Configuration File

cd /opt/rdaf/event_gateway/config/snmptrap

cat > trap_template.json <<EOF
{
    "1.3.6.1.6.3.1.1.5.4": {
        "action": "forward",
        "template": {
            "message": "Link status changed to up for interface '{{vbValue[3]}}' for device {{ipAddress}}",
            "device": "{{ipAddress}}",
            "trapOid": "{{trapOid}}",
            "snmpVersion": "{{snmpVersion}}",
            "timestamp": "{{timestamp}}",
            "timestampEpoch": "{{timestampEpoch}}"
        }
    },
    "1.3.6.1.6.3.1.1.5.3": {
        "action": "forward",
        "template": {
            "message": "Link status changed to down for interface '{{vbValue[3]}}' for device {{ipAddress}}",
            "device": "{{ipAddress}}",
            "trapOid": "{{trapOid}}",
            "snmpVersion": "{{snmpVersion}}",
            "timestamp": "{{timestamp}}",
            "timestampEpoch": "{{timestampEpoch}}"
        }
    }
}
EOF

Step-5: Bring Up Event Gateway

cd /opt/rdaf/event_gateway

docker-compose -f event-gateway-docker-compose.yml pull 
docker-compose -f event-gateway-docker-compose.yml up --d

Step-6: Check event gateway status

Check event gateway service status using docker ps command and ensure that event gateway is up and running, without any restarts. If you see that the event gateway is restarting, make sure you copied the RDA network configuration file to the correct location.

docker ps | grep gateway

Step-7: Verify RDA Event Gateway in RDA Fabric portal

A newly installed event gateway will authenticate with the RDA Fabric platform and it will show up in RDA Fabric portal under Configuration --> Fabric Components --> Agents --> View Details.

Step-8: Verify RDA Event Gateway using RDA Client (rdac) utility

If you have installed RDA Client (rdac) command line utility, you can also verify newly created event gateway using rdac agents command.

1.2.2 SSL Configuration for endpoints

Run the below command on event gateway to generate self-signed certificate files. Fill in the answers for the below prompts.

openssl req -x509 -newkey rsa:4096 -nodes -out cert.pem -keyout key.pem -days 365

Country Name (2 letter code)
State or Province Name (full name)
Locality Name (eg, city)
Organization Name (eg, company)
Organizational Unit Name (eg, section)
Common Name (eg, your name or your server's hostname)
Email Address

Above command generates two files under the current working directory, cert.pem and key.pem

Copy above files to /opt/rdaf/event_gateway/certs directory

cp cert.pem /opt/rdaf/event_gateway/certs
cp key.pem /opt/rdaf/event_gateway/certs

1.2.3 Endpoints Configuration

RDA event gateway support below end point types.

Syslog over TCP: Recieve syslog events over TCP protocol
Syslog over UDP: Recieve syslog events over UDP protocol
HTTP: Receive log events over HTTP protocol
TCP: Receive log events over TCP protocol
SNMP Traps: Receive SNMP traps over UDP protocol
Filebeat: Receive log events over HTTP protocol from log shipping agents such as filebeat & winlogbeat

Event gateway with the default configuration for each of the above end points as shown below. The endpoint configuration file is located @ /opt/rdaf/event_gateway/config/endpoint.yml

endpoints:

# Endpoint - Syslog Log events over TCP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: syslog_tcp_events
  enabled: false
  type: syslog_tcp
  port: 5140
  ssl: false
  ssl_cert_dir: /certs
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: syslog_events_archive # Log archive name
  stream: "NULL"
  direct_to_stream: syslog_tcp_event_stream

# Endpoint - Syslog Log events over UDP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: syslog_udp_events
  enabled: false
  type: syslog_udp
  port: 5141
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: syslog_events_archive # Log archive name
  stream: "NULL"
  direct_to_stream: syslog_udp_event_stream

# Endpoint - Events over HTTP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: http_events
  enabled: false
  type: http
  ssl: false
  ssl_cert_dir: /certs
  content_type: auto
  port: 5142
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: http_events_archive # Log archive name
  stream: "NULL"
  direct_to_stream: http_event_stream

# Endpoint - Events in JSON format over TCP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: tcp_json_events
  enabled: false
  type: tcp_json
  ssl: false
  ssl_cert_dir: /certs
  port: 5143
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: tcp_json_events_archive # Log archive name
  stream: "NULL"
  direct_to_stream: tcp_json_event_stream

# Endpoint - Events from Filebeat agent
# type: filebeat - It is applicable for both Filebeat and Winlogbeat log shipping agents
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: filebeat_events # URL is implicit, http://ip:port/filebeat_events
  type: filebeat
  enabled: false
  ssl: false
  ssl_cert_dir: /certs
  xpack_features: min
  port: 5144
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: filebeat_log_events_archive # Log archive name
  stream: "NULL"
  direct_to_stream: filebeat_event_stream

# Endpoint - Windows log events from Winlogbeat agent
# type: filebeat - It is applicable for both Filebeat and Winlogbeat log shipping agents
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: winlogbeat_events # URL is implicit, http://ip:port/winlogbeat_events
  type: filebeat
  enabled: false
  ssl: false
  ssl_cert_dir: /certs
  xpack_features: min
  port: 5145
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: winlogbeat_log_events_archive # Log archive name
  stream: "NULL"
  direct_to_stream: winlogbeat_event_stream

# Endpoint - SNMP Traps over UDP protocol
# attrs: <Custom attributes to be added for each trap event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: snmp_trap_events
  enabled: false
  type: snmp_trap
  port: 5146
  attrs:
    site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
    archive_name: snmp_trap_events_archive # Log archive name
  stream: "NULL"
  direct_to_stream: snmp_trap_event_stream

Info

For filebeat type endpoint, the supported version of the filebeat and winlogbeat log shipping agent is 7.8.1

1.3 RDA Edge Collector Installation

Prerequisites:

Linux OS
Memory - 8GB
Disk - 50GB
Python 3.7.4 or above
Docker container runtime environment (18.09.2 or above)
Docker-compose utility (1.27.x or above)

Installation Steps:

Step-1:

The RDA Edge Collector agent registers and communicates with the RDAF platform using a configuration file that contains your tenant ID, data fabric access tokens, and object storage credentials.

Download RDA Fabric Configuration from the portal by going to Configuration --> Fabric Configuration --> RDA network configuration and copy it to the local filesystem where the edge collector agent is going to be installed.

Save the file as rda_network_config.json

RDAFNetworkConfig

Create the below directory structure

sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/edgecollector/cred
sudo chown -R `id -u`:`id -g` /opt/rdaf

Copy the downloaded RDA Fabric configuration file as shown below.

cp rda_network_config.json /opt/rdaf/network_config/rda_network_config.json

Step-2: Docker Login

Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.

docker login -u='readonly' -p='readonly' cfxregistry.cloudfabrix.io

Step-3: Create Docker Compose File

Create docker compose configuration file for RDA Worker as shown below.

Tip

Note-1: Optionally change the edge collector's group name (also known as Site) in the docker-compose file by updating the agent-group-name value. In this example, the edge collector's group name is specified as edgecollector_group_01

Note-2: Adjust mem_limit and memswap_limit as per the workload requirements. In the below configuration, these parameters are set to 6GB

cd /opt/rdaf/edgecollector

cat > rda-edgecollector-docker-compose.yml <<EOF
version: '3.1'
services:
 rda_edgecollector_agent:
  image: 'cfxregistry.cloudfabrix.io/cfxcollector:daily'
  container_name: rda_edgecollector_agent
  restart: always
  network_mode: host
  mem_limit: 6G
  memswap_limit: 6G
  volumes:
   - /opt/rdaf/network_config:/network_config
   - /opt/rdaf/edgecollector:/cfxedgecollector
   - /opt/rdaf/edgecollector/cred:/cred
  environment:
   RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
   PYTHONPATH: /opt/cfx-rda-edgeagent
  logging:
   driver: "json-file"
   options:
    max-size: "25m"
    max-file: "5"
  ulimits:
   nproc:
    soft: 64000
    hard: 128000
   nofile:
    soft: 64000
    hard: 128000
  entrypoint:
   - /bin/bash
   - '-c'
   - >-
    cd /opt/cfx-rda-edgeagent/src/; python -c 'import edgecollector_rda_agent ;
    edgecollector_rda_agent.run()' --creddir /cred/ --agent-group-name ec-group-01

EOF

Step-4: Bring Up RDA Edge Collector

cd /opt/rdaf/edgecollector

docker-compose -f rda-edgecollector-docker-compose.yml pull 
docker-compose -f rda-edgecollector-docker-compose.yml up --d

Step-5: Check Edge Collector Status

Check Edge Collector agent status using docker ps command and ensure that it is up and running, without any restarts. If you see that if it is restarting, make sure you copied the RDA network config file to the correct location.

docker ps | grep edgecollector

Step-6: Verify RDA Edge Collector in RDA Fabric portal

A newly installed Edge Collector will authenticate with the RDA Fabric platform and it will show up in RDA Fabric portal under Configuration --> Fabric Components --> Agents --> View Details.

Step-7: Verify Edge Collector using RDA Client (rdac) utility

If you have installed RDA Client (rdac) command line utility, you can also verify newly created Edge Collector using rdac agents command.