Bots From Extension: kafka-v2
Kafka - Read, Write and poll data from external Kafka topics
This extension provides 6 bots.
Bot @kafka-v2:alter-topic
Bot Position In Pipeline: Source
Create or Update existing topic.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Kafka topic name | |
| num_partitions | Text | Number of partitions for the topic. Only used when creating new topic. These are advanced options and should be changed only if the configs set at broker level are not working as expected |
|
| replication_factor | Text | Replication factor for the topics. Only used when creating new topic. These are advanced options and should be changed only if the configs set at broker level are not working as expected |
|
| retention_ms | Text | maximum time in milliseconds to retain a log before discarding old log segments to free up space. Default 7 days |
This bot also accepts wildcard parameters. Any additional name = 'value' parameters are passed to the bot.
Bot *kafka-v2:list-groups
Bot Position In Pipeline: Source
List consumer groups.
This bot expects a Full CFXQL.
Bot applies the Query on the data that is already loaded from previous bot or from a source.
Bot *kafka-v2:list-topics
Bot Position In Pipeline: Source
List topics in Kafka.
This bot expects a Full CFXQL.
Bot applies the Query on the data that is already loaded from previous bot or from a source.
Bot @kafka-v2:list-topics-partitions
Bot Position In Pipeline: Source
List topics with the partitions in a consumer group.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name | Text | Kafka topic name | |
| group | Text | Consumer group name | |
| offset_reset | Text | earliest | Stream offset reset position. Valid values are 'earliest' or 'latest' |
| topic_configurations | Text | no | Collect topic configuration details such as retention.ms, max.message.bytes and flush.ms |
Bot @kafka-v2:read-stream
Bot Position In Pipeline: Source
Read data from specified kafka topic. This bot implements an infinite loop for receiving the data.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Stream name | |
| group* | Text | Data receiver (consumer) group name | |
| offset_reset | Text | earliest | Stream offset reset position. Valid values are 'earliest' or 'latest' |
| batch_size | Text | 100 | Maximum rows to read in each batch |
| batch_wait | Text | 5 | Maximum number of seconds to wait once one or more events are received. |
| lazy_commit | Text | no | Commit the messages only after the pipeline has completed processing them. |
| max_poll_interval_ms | Text | 300000 | Milliseconds to wait for the next pipeline steps to complete. If this is exceeded, the bot will fail. Default is 5 mins |
| retention_ms | Text | maximum time in milliseconds to retain a log before discarding old log segments to free up space. Default 7 days |
|
| commit_retries | Text | 5 | If lazy_commit is set to 'yes', number of times commit will be tried before failing. |
| commit_retry_interval | Text | 5 | If lazy_commit is set to 'yes', number of seconds to wait before retries. |
| num_partitions | Text | Number of partitions for the topic. Only used when creating new topic. These are advanced options and should be changed only if the configs set at broker level are not working as expected |
|
| replication_factor | Text | Replication factor for the topics. Only used when creating new topic. These are advanced options and should be changed only if the configs set at broker level are not working as expected |
This bot also accepts wildcard parameters. Any additional name = 'value' parameters are passed to the bot.
Bot @kafka-v2:write-stream
Bot Position In Pipeline: Sink
Write data to specified kafka topic.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Kafka topic name | |
| partition | Text | The topic partition ID to write to. If not provided, uses the configured built-in partitioner | |
| partition_key | Text | The key to be applied to the messages, all messages with the same key will be sent to the same partition. Only used when partiition parameter is not provided. If partition_key and partition_key_col are provided, key present in partition_key_col will be used |
|
| partition_key_col | Text | Column name that contains the key to be applied to the messages, all messages with the same key will be sent to the same partition. Only used when partiition parameter is not provided. If partition_key and partition_key_col are provided, key present in partition_key_col will be used |
|
| retention_ms | Text | maximum time in milliseconds to retain a log before discarding old log segments to free up space. Default 7 days |
|
| num_partitions | Text | Number of partitions for the topic. Only used when creating new topic. These are advanced options and should be changed only if the configs set at broker level are not working as expected |
|
| replication_factor | Text | Replication factor for the topics. Only used when creating new topic. These are advanced options and should be changed only if the configs set at broker level are not working as expected |
This bot also accepts wildcard parameters. Any additional name = 'value' parameters are passed to the bot.