Bots From Extension: exec
Bot and Pipeline Execution
This extension provides 21 bots.
Bot @exec:break-loop
Bot Position In Pipeline: Sink
Break out from the current loop iteration
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
Bot @exec:bulk-collect
Bot Position In Pipeline: Sink
Run bulk data collection queries provided through input dataframe and return aggregate output
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| status_dataset | Text | Output dataset name for the status of each collection | |
| data_label_col | Text | data_label | Name of the column in input for the data label |
| bot_col* | Text | Name of the column in input for identifying the bot which will be used for data collection | |
| query_col* | Text | Name of the column in input for query which will be used for data collection from the specified bot |
Bot @exec:call-function
Bot Position In Pipeline: Source Sink
Call (invoke) a function that was defined in the pipeline
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Name of the function to call |
Bot @exec:count-loop
Bot Position In Pipeline: Sink
Execute the block a specified number of iterations, with a specified delay between each iteration
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| count | Text | 0 | How many times to run the loop. if count is 0, this will run as infinite loop |
| delay_seconds | Text | 0 | Number of seconds to wait after each iteration of loop |
Bot @exec:define-function
Bot Position In Pipeline: Sink
Define a new function within the pipeline which can be called later in the pipeline
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Name of the function to define |
Bot @exec:end-function
Bot Position In Pipeline: Sink
End the function definition
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
Bot @exec:end-if
Bot Position In Pipeline: Sink
Ends the if-condition block and returns the dataset that was used before the block
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
Bot @exec:end-loop
Bot Position In Pipeline: Sink
Ends the for-loop iteration
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
Bot @exec:for-loop
Bot Position In Pipeline: Sink
Iterate through input dataframe num_rows at a time
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| num_rows | Text | 1 | Number of rows to from input on each iteration |
Bot @exec:get-input
Bot Position In Pipeline: Source Sink
Return the input dataframe provided for this pipeline, if any
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| alternate_dataset | Text | If the input is not available, load the specifiedd dataset and use as input |
Bot @exec:get-variable
Bot Position In Pipeline: Sink
Get the specified variable and return as a dataframe. This bot can be used only inside a function definition.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Variable name | |
| return_empty | Text | no | If variable is not found return an empty dataframe instead. Value must be one of 'yes' or 'no' |
| make_copy | Text | no | Make a copy of dataframe and return the copy. Value must be one of 'yes' or 'no' |
Bot *exec:if-condition
Bot Position In Pipeline: Sink
Run the subsequent bots only if CFXQL query matches. end-if bot must be called to exit this block
This bot expects a Full CFXQL.
Bot applies the Query on the data that is already loaded from previous bot or from a source.
Bot *exec:if-shape
Bot Position In Pipeline: Sink
Run the subsequent bots only if CFXQL query matches the num_rows, num_cols, columns based parameter query. end-if bot must be called to exit this block
This bot expects a Full CFXQL.
Bot applies the Query on the data that is already loaded from previous bot or from a source.
Bot @exec:retry-loop
Bot Position In Pipeline: Sink
Execute the block one time, if it fails retry for specified number attempts
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| retries | Text | 0 | Number of additional attempts to try if the block execution fails |
| on_failure_callback | Text | 0 | Function name. If the execution fails after specified retries also, call this function |
| retry_delay_millis | Text | 0 | Number of milli seconds to wait before retrying |
Bot @exec:return-function
Bot Position In Pipeline: Sink
Return from the function by returning current input to this bot as function output
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
Bot @exec:run-pipeline
Bot Position In Pipeline: Source Sink
Run a specified pipeline. Sends current input of this bot as input to the pipeline and returns the output of the pipeline
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Name of the pipeline to execute | |
| ignore_failures | Text | no | If set to 'yes', execution failures of inner pipeline will not cause failure of this pipeline. |
| import_functions | Text | no | If set to 'yes', this will import functions from another pipeline. Valid values are 'yes' or 'no' |
Example Pipelines Using this Bot
Bot @exec:run-pipeline-by-row
Bot Position In Pipeline: Source Sink
Run one pipeline per each input row. Expects pipeline name in a column.Each row is treated as an input by itself and sent to the specified pipeline. Status of each row is captured in output columns.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| pipeline_col* | Text | Name of column in input dataframe that has the pipeline name | |
| pipeline_type* | Text | Type of Pipeline - 'Draft' or 'Published' | |
| save_output | Text | no | Save output as a JSON string in the output column 'pipeline_exec_output'. Value must be one of 'yes' or 'no' |
| input_dataset | Text | Input dataset to pass to the pipeline. |
Bot @exec:run-pipeline-by-row-multi-proc
Bot Position In Pipeline: Source Sink
Concurrently one or more pipelines, one pipeline per each input row, using specified pool of processes. Expects pipeline name in a column.Each row is treated as an input by itself and sent to the specified pipeline. Status of each row is captured in output columns.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| pipeline_col* | Text | Name of column in input dataframe that has the pipeline name | |
| save_output | Text | no | Save output as a JSON string in the output column 'pipeline_exec_output'. Value must be one of 'yes' or 'no' |
| num_procs | Text | 0 | Number of concurrent processes. Must be between 0 and 50. If set to 0, it will use number of CPUs on the system as the value. |
| input_dataset | Text | Input dataset to pass to the pipeline. |
Bot @exec:run-pipeline-from-dict
Bot Position In Pipeline: Source Sink
Run one pipeline per each input row. Expects pipeline name in a column.Each row is treated as an input by itself and sent to the specified pipeline. Status of each row is captured in output columns.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| pipeline_dataset* | Text | Name of the dataset that contains 'pipeline_name' and 'tag_name' | |
| tag_name* | Text | Tag name to run the corresponding pipeline from pipeline_dataset. | |
| skip_pipeline_on_error | Text | yes | Skip child pipeline execution on error (yes/no). |
| pipeline_type | Text | Published | Type of Pipeline - 'Draft' or 'Published' |
| input_dataset | Text | Input dataset to pass to the pipeline. |
Bot @exec:run-pipeline-multi-proc
Bot Position In Pipeline: Source Sink
Run a specified pipeline concurrently with specified number of processes. Sends current input of this bot as input to the pipeline and returns the output of the pipeline
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Name of the pipeline to execute | |
| num_rows | Text | 100 | Number of rows to be passed as input per each process |
| num_procs | Text | 0 | Number of concurrent processes. Must be between 0 and 50. If set to 0, it will use number of CPUs on the system as the value. |
| collect_output | Text | yes | Collect output from each child process and merge. Specify 'yes' or 'no' |
Bot @exec:set-variable
Bot Position In Pipeline: Sink
Save current dataframe as a variable. This bot can be used only inside a function definition. Variable can be accessed only within the function.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| name* | Text | Variable name. Must start with a letter and may only contain letters, digits, and underscore | |
| append | Text | no | Append to existing data if available. Value must be one of 'yes' or 'no' |
| return_merged | Text | no | Applicable only if append is set to 'yes'. If return_merged is set to 'yes', this bot will return merged output. Value must be one of 'yes' or 'no' |