Bots From Extension: exec
Bot and Pipeline Execution
This extension provides 6 bots.
Bot @exec:bulk-collect
Bot Position In Pipeline: Sink
Run bulk data collection queries provided through input dataframe and return aggregate output
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| status_dataset | Text | Output dataset name for the status of each collection | |
| data_label_col | Text | data_label | Name of the column in input for the data label |
| bot_col* | Text | Name of the column in input for identifying the bot which will be used for data collection | |
| query_col* | Text | Name of the column in input for query which will be used for data collection from the specified bot |
Bot @exec:get-input
Bot Position In Pipeline: Source Sink
Return the input dataframe provided for this pipeline, if any
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| alternate_dataset | Text | If the input is not available, load the specifiedd dataset and use as input |
Bot @exec:run-pipeline
Bot Position In Pipeline: Source Sink
Run a specified pipeline. Sends current input of this bot as input to the pipeline and returns the output of the pipeline
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Name of the pipeline to execute | |
| ignore_failures | Text | no | If set to 'yes', execution failures of inner pipeline will not cause failure of this pipeline. |
Example Pipelines Using this Bot
Bot @exec:run-pipeline-by-row
Bot Position In Pipeline: Source Sink
Run one pipeline per each input row. Expects pipeline name in a column.Each row is treated as an input by itself and sent to the specified pipeline. Status of each row is captured in output columns.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| pipeline_col* | Text | Name of column in input dataframe that has the pipeline name | |
| pipeline_type* | Text | Type of Pipeline - 'Draft' or 'Published' | |
| save_output | Text | no | Save output as a JSON string in the output column 'pipeline_exec_output'. Value must be one of 'yes' or 'no' |
| input_dataset | Text | Input dataset to pass to the pipeline. |
Bot @exec:run-pipeline-by-row-multi-proc
Bot Position In Pipeline: Source Sink
Concurrently one or more pipelines, one pipeline per each input row, using specified pool of processes. Expects pipeline name in a column.Each row is treated as an input by itself and sent to the specified pipeline. Status of each row is captured in output columns.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| pipeline_col* | Text | Name of column in input dataframe that has the pipeline name | |
| save_output | Text | no | Save output as a JSON string in the output column 'pipeline_exec_output'. Value must be one of 'yes' or 'no' |
| num_procs | Text | 0 | Number of concurrent processes. Must be between 0 and 50. If set to 0, it will use number of CPUs on the system as the value. |
| input_dataset | Text | Input dataset to pass to the pipeline. |
Bot @exec:run-pipeline-multi-proc
Bot Position In Pipeline: Source Sink
Run a specified pipeline concurrently with specified number of processes. Sends current input of this bot as input to the pipeline and returns the output of the pipeline
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Name of the pipeline to execute | |
| num_rows | Text | 100 | Number of rows to be passed as input per each process |
| num_procs | Text | 0 | Number of concurrent processes. Must be between 0 and 50. If set to 0, it will use number of CPUs on the system as the value. |
| collect_output | Text | yes | Collect output from each child process and merge. Specify 'yes' or 'no' |