Bots From Extension: datanetwork
Data Network - Read, Write and poll data from internal Kafka topics
This extension provides 5 bots.
Bot *dn:list-groups
Bot Position In Pipeline: Source
List consumer groups in the data network.
This bot expects a Full CFXQL.
Bot applies the Query on the data that is already loaded from previous bot or from a source.
Bot *dn:list-topics
Bot Position In Pipeline: Source
List topics in Kafka.
This bot expects a Full CFXQL.
Bot applies the Query on the data that is already loaded from previous bot or from a source.
Bot @dn:list-topics-partitions
Bot Position In Pipeline: Source
List topics with the partitions in a consumer group.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name | Text | Kafka topic name | |
| group* | Text | Consumer group name | |
| offset_reset | Text | latest | Stream offset reset position. Valid values are 'earliest' or 'latest' |
| topic_configurations | Text | no | Collect topic configuration details such as retention.ms, max.message.bytes and flush.ms |
Bot @dn:read-stream
Bot Position In Pipeline: Source
Read data from specified stream in the data network. This bot implements an infinite loop for receiving the data.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Stream name | |
| group* | Text | Data receiver (consumer) group name | |
| offset_reset | Text | latest | Stream offset reset position. Valid values are 'earliest' or 'latest' |
| batch_size | Text | 100 | Maximum rows to read in each batch |
| batch_wait | Text | 5 | Maximum number of seconds to wait once one or more events are received. |
| lazy_commit | Text | no | Commit the messages only after the pipeline has completed processing them. |
| max_poll_interval_ms | Text | 300000 | Milliseconds to wait for the next pipeline steps to complete. If this is exceeded, the bot will fail. Default is 5 mins |
| retention_ms | Text | maximum time in milliseconds to retain a log before discarding old log segments to free up space. Default 7 days |
Bot @dn:write-stream
Bot Position In Pipeline: Sink
Write data to specified stream in the data network
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Kafka topic name | |
| partition | Text | The topic partition ID to write to. If not provided, uses the configured built-in partitioner | |
| partition_key | Text | The key to be applied to the messages, all messages with the same key will be sent to the same partition. Only used when partiition parameter is not provided |
|
| retention_ms | Text | maximum time in milliseconds to retain a log before discarding old log segments to free up space. Default 7 days |