Guide to Install and Configure RDA Fabric Edge Services
1. Install RDA Fabric Edge Services
RDA Worker:
RDA worker nodes are stateless data processing entities that can be installed closed to the source of data generation (ex: on-prem/enterprise/edge sites etc.). Worker nodes execute bots and pipelines and communicate with the RDAF platform that is responsible for scheduling and orchestrating jobs (pipelines) across various worker nodes.
Using worker nodes you can ingest and process/transform data locally without having to send all the data to centralized locations like an analytics platform or data warehouse. Two or more worker nodes in one environment can work as a group for load balancing and scale. RDAF platform can orchestrate data sharing or routing among worker nodes in distributed environments (ex: Worker nodes in edge location exchange data with workers in DC or workers in cloud).
Workers are essentially containerized service nodes and can be installed using Docker-compose or in a Kubernetes environment. Workers are typically installed on VMs that are located on-premises / cloud / edge environments.
RDA Event Gateway:
RDA Event Gateway is a type of RDA Agent that can send streaming data to the RDA Fabric platform. If user wants to send logs/events in real-time to the RDAF platform, users can install Event Gateway in their local environment and configure event sources to send data to Event Gateway.
Similar to RDA worker nodes, event gateways are also containerized services and can be installed using Docker-compose or in a Kubernetes environment. RDA event gateways are typically installed on VMs that are located on-premises / cloud / edge environments.
-
Log Sources: For instance, to send syslogs from your Linux servers to the RDA platform, you can install Event Gateway and configure rsyslog on your Linux servers to send data to Event Gateway, which in turn can send data to the RDAF platform.
-
Existing Log Shippers: Users can also use existing log shippers like Splunk Universal Forwarder, Elasticsearch beats, Fluentd, rsyslog, syslog-ng, etc. to route / send data to Event Gateway.
-
Endpoints: Event Gateway supports endpoints and each endpoint is configured to send data to a stream on RDAF platform. For example, you can configure an endpoint with a port and protocol/type (ex: TCP/syslog) and all syslog sources can send data to that endpoint.
RDA Edge Collector :
RDA Edge Collector is a type of RDA agent that can discover and collect IT asset data in an agentless manner and send the collected data to the RDA Fabric platform. Edge Collector agent is primarily used to discover the IT assets (i.e. Servers, Switches, Routers, Firewall, Load Balancers, Storage Arrays etc.) that provide inventory data over SNMP and SSH protocols.
Similar to RDA worker & event gateway, edge collectors are also containerized services and can be installed using Docker-compose or in a Kubernetes environment. RDA edge collectors are typically installed on VMs that are located on-premises / cloud / edge environments.
1.1 RDA Worker Installation
Prerequisites:
- Linux OS
- Memory - 8GB
- Disk - 50GB
- Python 3.7.4 or above
- Docker container runtime environment (18.09.2 or above)
- Docker-compose utility (1.27.x or above)
Installation Steps:
Step-1:
The RDA worker node registers and communicates with the RDAF platform using a configuration file that contains your tenant ID, data fabric access tokens, and object storage credentials.
Download RDA Fabric Configuration from the portal by going to Configuration --> Fabric Configuration --> RDA network configuration
and copy it to the local filesystem where the worker node is going to be installed.
- Save the file as
rda_network_config.json
- Create the below directory structure
sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/worker/config
sudo mkdir -p /opt/rdaf/worker/logs
sudo chown -R `id -u`:`id -g` /opt/rdaf
- Copy the downloaded RDA Fabric configuration file as shown below.
- Create common.yml file for RDA Worker to configure logger settings as shown below.
cd /opt/rdaf/worker/config
cat > common.yml << 'EOF'
version: 1
disable_existing_loggers: false
formatters:
standard:
format: "%(asctime)s %(levelname)s %(module)s - PID=%(process)s %(message)s"
handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: standard
stream: ext://sys.stdout
file_handler:
class: logging.handlers.RotatingFileHandler
level: INFO
formatter: standard
filename: /logs/${pod_type}-${pod_id}.log
maxBytes: 10485760 # 10MB
backupCount: 5
encoding: utf8
root:
level: INFO
handlers: [console, file_handler]
propogate: yes
EOF
Step-2: Docker Login
Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.
Step-3: Create Docker Compose File
Create docker compose configuration file for RDA Worker as shown below.
Tip
Note-1: Optionally change the worker group name (also known as Site) in the docker-compose file by updating the WORKER_GROUP value. In this example, the worker group name is specified as rda_worker_group01
Note-2: Adjust mem_limit and memswap_limit as per the workload requirements. In the below configuration, these parameters are set to 16GB
cd /opt/rdaf/worker
cat > rda-worker-docker-compose.yml <<EOF
version: '3.1'
services:
rda_worker:
image: cfxregistry.cloudfabrix.io/ubuntu-rda-worker-all:daily
restart: always
network_mode: host
mem_limit: 16G
memswap_limit: 16G
shm_size: 1gb
volumes:
- /opt/rdaf/network_config:/network_config
- /opt/rdaf/worker/config:/loggingConfigs
- /opt/rdaf/worker/logs:/logs
logging:
driver: "json-file"
options:
max-size: "25m"
max-file: "5"
environment:
RESOURCE_NAME:
RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
LOGGER_CONFIG_FILE: /loggingConfigs/common.yml
WORKER_GROUP: rda_worker_group01
LABELS: name=rda_worker_01
RDA_SELF_HEALTH_RESTART_AFTER_FAILURES: 3
CAPACITY_FILTER: mem_percent < 95
EOF
Tip
Note-1: RDA worker(s) communicates with RDA Fabric that is running in cloud or on-premise datacenter over ports 4222/TCP & 9443/TCP. Please make sure RDA worker(s) has outbound network access over these network ports. In addition, make sure RDA Fabric is configured to allow inbound network traffic for the same ports to accept the traffic from RDA worker(s).
Note-2: Please verify rda_network_config.json is configured with publicly accessible IP/FQDN of RDA Fabric for NATs and Minio endpoints.
Step-4: Bring Up RDA Worker
cd /opt/rdaf/worker
docker-compose -f rda-worker-docker-compose.yml pull
docker-compose -f rda-worker-docker-compose.yml up -d
cd /opt/rdaf/worker
docker compose -f rda-edgecollector-docker-compose.yml pull
docker compose -f rda-edgecollector-docker-compose.yml up -d
Check worker node status using docker ps
command and ensure that worker is up and running, without any restarts. If you see that the worker is restarting, make sure you copied the RDA network config file to the correct location.
Step-6: Verify RDA Worker in RDA Fabric portal
A newly installed worker will authenticate with the RDA Fabric platform and it will show up in RDA Fabric portal under Configuration --> Fabric Components --> Workers --> View Details
.
Step-7: Verify Worker using RDA Client (rdac) utility
If you have installed RDA Client (rdac) command line utility, you can also verify newly created worker using rdac pods
command.
1.2 RDA Event Gateway Installation
Prerequisites:
- Linux OS
- Memory - 8GB
- Disk - 50GB
- Python 3.7.4 or above
- Docker container runtime environment (18.09.2 or above)
- Docker-compose utility (1.27.x or above)
1.2.1 Installation Steps
Step-1:
The RDA event gateway registers and communicates with the RDA Fabirc platform using a configuration file that contains your tenant ID, data fabric access tokens, and object storage credentials.
Download RDA Fabric Configuration from the portal by going to Configuration --> Fabric Configuration --> RDA network configuration
and copy it to the local filesystem where the event gateway is going to be installed.
- Save the file as
rda_network_config.json
- Create the below directory structure
sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/event_gateway/config/main
sudo mkdir -p /opt/rdaf/event_gateway/config/snmptrap
sudo mkdir -p /opt/rdaf/event_gateway/certs
sudo mkdir -p /opt/rdaf/event_gateway/logs
sudo mkdir -p /opt/rdaf/event_gateway/log_archive
sudo chown -R `id -u`:`id -g` /opt/rdaf
- Copy the downloaded RDA Fabric configuration file as shown below.
Step-2: Docker Login
Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.
Step-3: Create Docker Compose File
Create docker compose configuration file for RDA event gateway as shown below.
Info
Note: Optionally change the agent group name in the docker-compose file by updating the AGENT_GROUP value. In this example, the agent group name is specified as event_gateway_site01
cd /opt/rdaf/event_gateway
cat > event-gateway-docker-compose.yml <<EOF
version: '3.1'
services:
rda_event_gateway:
image: cfxregistry.cloudfabrix.io/ubuntu-rda-event-gateway:daily
restart: always
network_mode: host
mem_limit: 6G
memswap_limit: 6G
volumes:
- /opt/rdaf/network_config:/network_config
- /opt/rdaf/event_gateway/config:/event_gw_config
- /opt/rdaf/event_gateway/certs:/certs
- /opt/rdaf/event_gateway/logs:/logs
- /opt/rdaf/event_gateway/log_archive:/tmp/log_archive
logging:
driver: "json-file"
options:
max-size: "25m"
max-file: "5"
environment:
RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
EVENT_GW_MAIN_CONFIG: /event_gw_config/main/main.yml
EVENT_GW_SNMP_TRAP_CONFIG: /event_gw_config/snmptrap/trap_template.json
EVENT_GW_SNMP_TRAP_ALERT_CONFIG: /event_gw_config/snmptrap/trap_to_alert_go.yaml
AGENT_GROUP: event_gateway_site01
EVENT_GATEWAY_CONFIG_DIR: /event_gw_config
LOGGER_CONFIG_FILE: /event_gw_config/main/logging.yml
RDA_SELF_HEALTH_RESTART_AFTER_FAILURES: 3
entrypoint: ["/docker-entry-point.sh"]
EOF
Step-4: Create SNMP Trap Configuration File
Info
Note: Download * trap_to_alert_go.yaml file and copy to /opt/rdaf/event_gateway/config/snmptrap directory
cd /opt/rdaf/event_gateway/config/snmptrap/
wget https://macaw-amer.s3.amazonaws.com/releases/rda-edge-services/event_gateway/trap_to_alert_go.yaml
cd /opt/rdaf/event_gateway/config/snmptrap
cat > trap_template.json <<EOF
{
"1.3.6.1.6.3.1.1.5.4": {
"action": "forward",
"template": {
"message": "Link status changed to up for interface '{{vbValue[3]}}' for device {{ipAddress}}",
"device": "{{ipAddress}}",
"trapOid": "{{trapOid}}",
"snmpVersion": "{{snmpVersion}}",
"timestamp": "{{timestamp}}",
"timestampEpoch": "{{timestampEpoch}}"
}
},
"1.3.6.1.6.3.1.1.5.3": {
"action": "forward",
"template": {
"message": "Link status changed to down for interface '{{vbValue[3]}}' for device {{ipAddress}}",
"device": "{{ipAddress}}",
"trapOid": "{{trapOid}}",
"snmpVersion": "{{snmpVersion}}",
"timestamp": "{{timestamp}}",
"timestampEpoch": "{{timestampEpoch}}"
}
}
}
EOF
Step-5: Bring Up Event Gateway
cd /opt/rdaf/event_gateway
docker-compose -f event-gateway-docker-compose.yml pull
docker-compose -f event-gateway-docker-compose.yml up -d
Step-6: Check event gateway status
Check event gateway service status using docker ps
command and ensure that event gateway is up and running, without any restarts. If you see that the event gateway is restarting, make sure you copied the RDA network configuration file to the correct location.
Step-7: Verify RDA Event Gateway in RDA Fabric portal
A newly installed event gateway will authenticate with the RDA Fabric platform and it will show up in RDA Fabric portal under Configuration --> Fabric Components --> Agents --> View Details
.
Step-8: Verify RDA Event Gateway using RDA Client (rdac) utility
If you have installed RDA Client (rdac) command line utility, you can also verify newly created event gateway using rdac agents
command.
1.2.2 SSL Configuration for endpoints
Run the below command on event gateway to generate self-signed certificate files. Fill in the answers for the below prompts.
- Country Name (2 letter code)
- State or Province Name (full name)
- Locality Name (eg, city)
- Organization Name (eg, company)
- Organizational Unit Name (eg, section)
- Common Name (eg, your name or your server's hostname)
- Email Address
Above command generates two files under the current working directory, cert.pem
and key.pem
Copy above files to /opt/rdaf/event_gateway/certs directory
1.2.3 Endpoints Configuration
RDA event gateway support below end point types.
- Syslog over TCP: Recieve syslog events over TCP protocol
- Syslog over UDP: Recieve syslog events over UDP protocol
- HTTP: Receive log events over HTTP protocol
- TCP: Receive log events over TCP protocol
- SNMP Traps: Receive SNMP traps over UDP protocol
- Filebeat: Receive log events over HTTP protocol from log shipping agents such as
filebeat
&winlogbeat
Event gateway with the default configuration for each of the above end points as shown below. The endpoint configuration file is located @ /opt/rdaf/event_gateway/config/endpoint.yml
endpoints:
# Endpoint - Syslog Log events over TCP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: syslog_tcp_events
enabled: false
type: syslog_tcp
port: 5140
ssl: false
ssl_cert_dir: /certs
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: syslog_events_archive # Log archive name
stream: "NULL"
direct_to_stream: syslog_tcp_event_stream
# Endpoint - Syslog Log events over UDP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: syslog_udp_events
enabled: false
type: syslog_udp
port: 5141
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: syslog_events_archive # Log archive name
stream: "NULL"
direct_to_stream: syslog_udp_event_stream
# Endpoint - Events over HTTP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: http_events
enabled: false
type: http
ssl: false
ssl_cert_dir: /certs
content_type: auto
port: 5142
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: http_events_archive # Log archive name
stream: "NULL"
direct_to_stream: http_event_stream
# Endpoint - Events in JSON format over TCP protocol
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: tcp_json_events
enabled: false
type: tcp_json
ssl: false
ssl_cert_dir: /certs
port: 5143
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: tcp_json_events_archive # Log archive name
stream: "NULL"
direct_to_stream: tcp_json_event_stream
# Endpoint - Events from Filebeat agent
# type: filebeat - It is applicable for both Filebeat and Winlogbeat log shipping agents
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: filebeat_events # URL is implicit, http://ip:port/filebeat_events
type: filebeat
enabled: false
ssl: false
ssl_cert_dir: /certs
xpack_features: min
port: 5144
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: filebeat_log_events_archive # Log archive name
stream: "NULL"
direct_to_stream: filebeat_event_stream
# Endpoint - Windows log events from Winlogbeat agent
# type: filebeat - It is applicable for both Filebeat and Winlogbeat log shipping agents
# attrs: <Custom attributes to be added for each log event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: winlogbeat_events # URL is implicit, http://ip:port/winlogbeat_events
type: filebeat
enabled: false
ssl: false
ssl_cert_dir: /certs
xpack_features: min
port: 5145
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: winlogbeat_log_events_archive # Log archive name
stream: "NULL"
direct_to_stream: winlogbeat_event_stream
# Endpoint - SNMP Traps over UDP protocol
# attrs: <Custom attributes to be added for each trap event, provide one or more attributes in key: value format>
# stream: <Write the log events to a Stream within RDA Fabric>
- name: snmp_trap_events
enabled: false
type: snmp_trap
port: 5146
attrs:
site_code: event_gateway_site01 # Site Name / Code where Event gateway is deployed
archive_name: snmp_trap_events_archive # Log archive name
stream: "NULL"
direct_to_stream: snmp_trap_event_stream
Info
For filebeat type endpoint, the supported version of the filebeat and winlogbeat log shipping agent is 7.8.1
1.3 RDA Edge Collector Installation
Note
Edge Collector or Edge Collector Agent wherever mentioned below in the Document refers to same component.
Prerequisites:
- Linux OS
- Memory - 8GB
- Disk - 50GB
- Python 3.7.4 or above
-
Docker Version (18.09.2 or above)
-
Docker Compose (1.27.x and above)
Installation Steps:
Step-1:
The RDA Edge Collector agent registers and communicates with the RDAF platform using a configuration file that contains your tenant ID, data fabric access tokens, and object storage credentials.
Download RDA Fabric Configuration from the portal by going to Configuration --> RDA Administration --> Network
and copy it to the local filesystem where the edge collector agent is going to be installed.
- Save the file as
rda_network_config.json
- Create the below directory structure
sudo mkdir -p /opt/rdaf/network_config
sudo mkdir -p /opt/rdaf/edgecollector/cred
sudo chown -R `id -u`:`id -g` /opt/rdaf
- Copy the downloaded RDA Fabric configuration file as shown below.
Step-2: Docker Login
Run the below command to create and save the docker login session into CloudFabrix's secure docker repository.
Step-3: Create Docker Compose File
Create docker compose configuration file for RDA Worker as shown below.
Tip
Note-1: Optionally change the edge collector's group name (also known as Site) in the docker-compose file by updating the agent-group-name value. In this example, the edge collector's group name is specified as edgecollector_group_01
Note-2: Adjust mem_limit and memswap_limit as per the workload requirements. In the below configuration, these parameters are set to 6GB
cd /opt/rdaf/edgecollector
cat > rda-edgecollector-docker-compose.yml <<EOF
version: '3.1'
services:
rda_edgecollector_agent:
image: 'cfxregistry.cloudfabrix.io/cfxcollector:daily'
container_name: rda_edgecollector_agent
restart: always
network_mode: host
mem_limit: 6G
memswap_limit: 6G
volumes:
- /opt/rdaf/network_config:/network_config
- /opt/rdaf/edgecollector:/cfxedgecollector
- /opt/rdaf/edgecollector/cred:/cred
environment:
RDA_NETWORK_CONFIG: /network_config/rda_network_config.json
PYTHONPATH: /opt/cfx-rda-edgeagent
logging:
driver: "json-file"
options:
max-size: "25m"
max-file: "5"
ulimits:
nproc:
soft: 64000
hard: 128000
nofile:
soft: 64000
hard: 128000
entrypoint:
- /bin/bash
- '-c'
- >-
cd /opt/cfx-rda-edgeagent/src/; python -c 'import edgecollector_rda_agent ;
edgecollector_rda_agent.run()' --creddir /cred/ --agent-group-name ec-group-01
EOF
cd /opt/rdaf/edgecollector
docker-compose -f rda-edgecollector-docker-compose.yml pull
docker-compose -f rda-edgecollector-docker-compose.yml up -d
Step-5: Check Edge Collector Status
Check Edge Collector agent status using docker ps
command and ensure that it is up and running, without any restarts. If you see that if it is restarting, make sure you copied the RDA network config file to the correct location.
Step-6: Verify RDA Edge Collector in RDA Fabric portal
A newly installed Edge Collector will authenticate with the RDA Fabric platform and it will show up in RDA Fabric portal under Fabric Health --> Agents
.
Step-7: Verify Edge Collector using RDA Client (rdac) utility
If you have installed RDA Client (rdac) command line utility, you can also verify newly created Edge Collector using rdac agents
command.