Bots From Extension: rn
RDA Networking & Streaming
This extension provides 10 bots.
Bot @rn:read-stream
Bot Position In Pipeline: Source
Read one more rows from a specified stream in a chunked manner. No acknowledgment is expected.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | RDA Network stream name | |
| batch_count | Text | 100 | Wait for batch_count messages or batch_wait_time seconds for each batch |
| batch_wait_time | Text | 1.0 | Wait for batch_count messages or batch_wait_time seconds for each batch |
| group* | Text | Consumer group name. If specified, must be a valid group name without any spaces. |
Example Pipelines Using this Bot
- dli-process-synthetic-syslogs
- ebonding-stream-to-elasticsearch-kibana-v2
- ebonding-stream-to-email
- ebonding-stream-to-pagerduty
- ebonding-stream-to-slack
- ebonding-stream-to-twilio-sms-v2
- li-filebeat-events-to-prod-env
- li-http-events-to-prod-env
- li-replay-logs-to-dev-env
- li-stream-tcp-syslogs
- li-tcp-syslog-events-to-dev-env
- li-tcp-syslog-events-to-prod-env
- li-udp-syslog-events-to-prod-env
- li-windows-events-to-prod-env
Bot @rn:read-stream-with-ack
Bot Position In Pipeline: Source
Read one message at a time from the specified stream and send ack to publisher
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | RDA Network stream name | |
| group* | Text | Consumer group name. If specified, must be a valid group name without any spaces. |
Bot @rn:service-invoke
Bot Position In Pipeline: Sink
Pipeline as a Microservice (PaaM) request invoker. Make a service request with input dataframe and output the data returnedd by the service. This bot can be used to make a service request call to any microservice that is implemented by @rn:service-receive-request
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Initiate a call to this service and wait for the response | |
| timeout | Text | 10.0 | Timeout in seconds. Default is 10s. |
| action | Text | fail | Acion to perform if the requst fails. Must be one of continue, fail, skip-block, skip-pipeline. If 'continue', bot send original input as output. |
Bot @rn:service-receive-request
Bot Position In Pipeline: Source
Pipeline as a Microservice (PaaM) request receeiver. This bot can be used convert any pipeline into a microservice which will receive to requests and send response back to the client.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Name for the service which this pipeline implements using request-response paradigm. | |
| desription | Text | Optional description of the service. | |
| min_rows | Text | 0 | Minimum number of rows in input. Default is 0. If specified it must >= 0. |
| max_rows | Text | 0 | Maximum number of rows in input. If specified it must >= 0. Default is no limit. |
Bot @rn:service-respond-request
Bot Position In Pipeline: Sink
Pipeline as a Microservice (PaaM) request responder. This bot used conjuction with @rn:service-receive-request
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| flags | Text | Optional flags. Currently no specific flags are expected. |
Bot @rn:write-macaw-notifcation-asset-db
Bot Position In Pipeline: Source Sink
Send Macaw Notification to stream ASSET_DB_UPDATED to notify that a dataset has been updated
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| dataset* | Text | Input Dataset Name to be published to Asset DB | |
| tag* | Text | Dataset Tag Name for Asset DB | |
| model* | Text | Dataset Model Name for Asset DB | |
| submitted_by | Text | rda_pipeline | Optional. Name of the data publisher |
| tag_prefix | Text | Optional. Tag prefix to be used in Asset DB | |
| customerName | Text | Optional. Customer Name used in Asset DB | |
| bucketName | Text | Optional.if not provided the bucketName will be picked from dataset-location bot | |
| objectName | Text | Optional.if not provided the objectName will be picked from dataset-location bot |
Bot @rn:write-stats-to-stream
Bot Position In Pipeline: Sink
Write input dataframe stats (row count, byte count) to specified stream.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | RDA network stream name | |
| groupby | Text | Optional comma separated list of columns. If not specified, publishes total row count and byte_count. If Specified, computes group level row count and byte count. |
This bot also accepts wildcard parameters. Any additional name = 'value' parameters are passed to the bot.
Example Pipelines Using this Bot
- dli-generate-synthetic-syslogs
- dli-process-synthetic-syslogs
- ebonding-servicenow-to-stream-v2
- ebonding-stream-to-elasticsearch-kibana-v2
- ebonding-stream-to-email
- ebonding-stream-to-pagerduty
- ebonding-stream-to-slack
- ebonding-stream-to-twilio-sms-v2
- li-filebeat-events-to-prod-env
- li-http-events-to-prod-env
- li-replay-logs-to-dev-env
- li-stream-tcp-syslogs
- li-tcp-syslog-events-to-dev-env
- li-tcp-syslog-events-to-prod-env
- li-udp-syslog-events-to-prod-env
- li-windows-events-to-prod-env
Bot @rn:write-stream
Bot Position In Pipeline: Sink
Write input dataframe as messages to a specified stream. Each row is encoded as a JSON object. No acknowledgment is expected.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | RDA network stream name | |
| compression | Text | auto | Compress data while publishing. Possible values are 'auto', 'yes' or 'no'. 'auto' would enable compression if size meets a specific criteria |
Example Pipelines Using this Bot
- dli-generate-synthetic-syslogs
- dli-process-synthetic-syslogs
- ebonding-servicenow-to-stream-v2
- li-stream-tcp-syslogs
Bot @rn:write-stream-by-row
Bot Position In Pipeline: Sink
Write each row of input dataframe to a stream as specified by stream_column. No acknowledgment is expected.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| stream_column* | Text | Name of the column in input dataframe that contains destination stream name | |
| status_column | Text | Name of the column in output dataframe to capture status |
Bot @rn:write-stream-with-ack
Bot Position In Pipeline: Sink
Write input dataframe as messages to a specified stream. Each row is encoded as a JSON object. Ensures that subscriber has acknowledged the message.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | RDA network stream name | |
| timeout | Text | 10.0 | Timeout in seconds for ACK from the subscriber |
| retries | Text | 3 | Number of retries |