Bots From Extension: file
Read & Write From Files
This extension provides 17 bots.
Bot @files:appendfile
Bot Position In Pipeline: Sink
Save dataframe to file using filename, if file exists, append the data. Supports only CSV format.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| filename* | Text | Filename or URL for the output data. Only CSV files are supported for append. |
Bot @files:archivefile
Bot Position In Pipeline: Sink
Archive dataframe to file using filename and folder format specified. File is saved as compressed (gzipped) json.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| archive_folder* | Text | Root folder for creating archive sub-directories and files. For e.g. - /Archive | |
| file_format | Text | /%Y/%m/%d/data.json | Defines the folder structure and filename to use. Supports Jinja style variables in file format. E.g. - /syslog/%Y/%m/%d/%H/%M/{{CATEQUIP}}.json |
| bucket_minutes | Text | 5 | Frequency of rolling over to a new bucket/folder |
| timestamp_column | Text | Timestamop column name in the source data. This field is used to extract the timestamp value and create folder sctructure based on that value. If not specified, current time is used |
|
| compress | Text | no | Should the archieved file be compressed? If you want files to be compressed, set it to 'yes'. |
| append | Text | no | If set to 'yes', data is appended to existing file |
| fix_path_elements | Text | yes | Fix the path |
| overwrite | Text | no | Set it to 'yes' to overwrite the archieve file if it exists |
Bot @files:datasets-to-xlsx
Bot Position In Pipeline: Source
Export specified datasets to an Excel file, each dataset as a different sheet. Returns summary of export.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| filename* | Text | Excel filename for the exported data | |
| datasets* | Text | Regular expression listing of datasets to be included in export |
Bot @files:ec-entity-analysis
Bot Position In Pipeline: Source
Processes EC zip file and Builds relationships between the entities
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| input_zipfile_path | Text | Path to EdgeCollector ZIP file | |
| input_asset_path | Text | Path to EdgeCollector Json Folder | |
| output_path* | Text | Path to store output files | |
| device_ops_view | Text | False | Enable to create device-ops-view json |
| device_view | Text | False | Enable to create device-view json |
| poe_summary | Text | False | Enable to create poe_summary json |
Bot @files:ec-get-collected-data
Bot Position In Pipeline: Source
Parse collected JSON data from a CloudFabrix EdgeCollector ZIP Filepath
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| filename* | Text | Path to EdgeCollector ZIP filename | |
| ip_pattern | Text | .* | IP Address pattern for which data should be loaded |
| ip_list | Text | Comma searated IP Addresses for which data should be loaded | |
| json_path | Text | JSON path to a specific object in each input JSON file (seperated by .), example: 'base.data' |
Bot @files:ec-get-discovery-results
Bot Position In Pipeline: Sink
Parse collected JSON data from a CloudFabrix EdgeCollector Agent results
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| input_column | Text | DISCOVERY_ZIP_FILE | Column Name in input dataframe which is usually output of EdgeCollector Agent discovery results |
| ip_pattern | Text | .* | IP Address pattern for which data should be loaded |
| ip_list | Text | Comma searated IP Addresses for which data should be loaded | |
| json_path | Text | JSON path to a specific object in each input JSON file (seperated by .), example: 'base.data' | |
| minio_file_location | Text | Location of stored Edgecollector Agent discovery file in minio | |
| read_failed_devices | Text | no | Specify 'yes' to include failed devices or 'no' to load only successful devices. By default loads successfull devices only |
Bot @files:ec-get-raw-data
Bot Position In Pipeline: Source
Get collected raw text from a CloudFabrix EdgeCollector ZIP File
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| filename* | Text | Path to EdgeCollector ZIP filename | |
| ip_pattern | Text | .* | IP Address pattern for which data should be loaded |
| raw_filename_pattern | Text | .* | Regular expression pattern to filter for specific filenames |
Bot @files:export-content-to-zip-file
Bot Position In Pipeline: Sink
Export each row as a file into a ZIP file
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| output_zipfile* | Text | Output ZIP filename | |
| filename_column | Text | filename | Column with filename to be added to ZIP, one per each row |
| content_column | Text | content | File content column |
| encoding_column | Text | encoding | Column name with encoding type. Encoding type can be 'text' or 'binary'. If column not found, uses 'text' encoding |
Bot @files:get-zip-fileinfo
Bot Position In Pipeline: Sink
List information about each file within the provided ZIP File
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| filename* | Text | Path to ZIP file | |
| is_minio_file | Text | yes | 'yes' if the zip file is located in minio. Valid values are 'yes' or 'no' |
Bot @files:load-content-from-file
Bot Position In Pipeline: Sink
Load content from file specified by 'filename' column into 'content' column
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| filename_column | Text | filename | Column with filename to be loaded per each row |
| content_column | Text | content | File content column where loaded data will be placed |
| encoding_column | Text | encoding | Column name with encoding type. Encoding type can be 'text' or 'binary'. If column found, uses 'text' encoding |
Bot @files:loadfile
Bot Position In Pipeline: Source
Load dataframe from file using filename, filetype, sheet and line_read parameters
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| filename* | Text | Filename or URL for the input data | |
| filetype | Text | File type is either csv/json/xlsx. If not specified, it will be derived from extension. | |
| sheet | Text | Specify sheet name if file type is xlsx. If not specified, first sheet will be returned. | |
| line_read | Text | no | Only applicable for JSON. By default, the whole file is loaded as JSON. If you want to load as a json object per line, set it to 'yes'. |
| sep | Text | , | Only applicable for csv. Field delimiter to use. |
| skipinitialspace | Text | no | Only applicable for csv. Skip spaces after delimiter. |
| skiprows | Text | Only applicable for csv. Line numbers to skip (0-indexed) or number of lines to skip (int) at the start of the file. |
|
| skipfooter | Text | 0 | Only applicable for csv. Number of lines at bottom of file to skip. |
Bot @files:rda-get-discovery-results
Bot Position In Pipeline: Sink
Parse collected data from a CloudFabrix RDA ZIP Filepath
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| filename* | Text | Path to ZIP file | |
| type | Text | RDA collection type included within the meta.json | |
| descriptors | Text | Comma separated RDA collection descriptors included within the meta.json. Eg: cucm-servers,cucm-servers | |
| is_minio_file | Text | yes | 'yes' if the zip file is located in minio. Valid values are 'yes' or 'no' |
Bot @files:replay-archivefile
Bot Position In Pipeline: Source
Replay archive files using archive mount point, folder format, and time period specified.
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| archive_folder* | Text | Root folder from where to read archive sub-directories and files. For e.g. - /Archive | |
| dir_pattern | Text | %Y/%m/%d/ | Defines the folder structure to use. Supports Jinja style variables in directory pattern. E.g. - /syslog/%Y/%m/%d/%H/%M/{{CATEQUIP}}.json |
| from_timestamp* | Text | Read data from timestamp - for e.g. '2024/01/01. | |
| to_timestamp* | Text | Read data upto timestamp - for e.g. '2024/03/01. | |
| max_rows | Text | 0 | Maximum rows to replay. If not specified, will replay all data in the specified intervals. If specified and > 0, it will stop once specified rows have been read. |
| speed | Text | 1.0 | Speed at which to replay the events. 1 means close to original speed. < 1 means slower than original. > 1 means faster than original. This is an approximate and cannot be guaranteed. 0 means no introduced latency and try to replay as fast as possible. |
| data_batch_size | Text | 100 | Number of rows to return for each iteration. |
| file_batch_size | Text | 100 | Number of archived file to process for each iteration. |
| log_replay_activity | Text | 0 | A value of 1 instructs the bot to wrire job details to 'rda_file_replay_jobs_stream' stream and replay details to 'rda_file_archive_replays' stream. Defaut is 1 |
Bot @files:replay-archivefile-direct
Bot Position In Pipeline: Sink
Replay archive files using archive mount point, folder format, and time period specified. Archived data is streamed directly to the provided <stream_name>
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| archive_folder* | Text | Root folder from where to read archive sub-directories and files. For e.g. - /Archive | |
| stream_name* | Text | 5 | Name of the RDA stream to write data to |
| dir_pattern | Text | %Y/%m/%d/ | Defines the folder structure to use. Supports Jinja style variables in directory pattern. E.g. - /syslog/%Y/%m/%d/%H/%M/{{CATEQUIP}}.json |
| from_timestamp* | Text | Read data from timestamp - for e.g. '2024/01/01. | |
| max_rows | Text | 0 | Maximum rows to replay. If not specified, will replay all data in the specified intervals. If specified and > 0, it will stop once specified rows have been read. |
| speed | Text | 1.0 | Speed at which to replay the events. 1 means close to original speed. < 1 means slower than original. > 1 means faster than original. This is an approximate and cannot be guaranteed. 0 means no introduced latency and try to replay as fast as possible. |
| data_batch_size | Text | 500 | Number of rows to return for each iteration. |
| file_batch_size | Text | 500 | Number of file to process for each iteration. |
| to_timestamp* | Text | Read data upto timestamp - for e.g. '2024/03/01. | |
| log_replay_activity | Text | 1 | A value of 1 instructs the bot to wrire job details to 'rda_file_replay_jobs_stream' stream and replay details to 'rda_file_archive_replays' stream. Defaut is 1 |
Bot @files:rerun-replay-archivefile-direct
Bot Position In Pipeline: Sink
Rerun a previous archivefile replay by providing the job name. Archived data is streamed directly to the previous stream or to the provided <stream_name>
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| job_name* | Text | 5 | Name of the Job to rerun file archive replay |
| stream_name | Text | Name of the RDA stream to write data to | |
| force | Text | 0 | If force is 1, bot reloads all the files again. If 0, reloads only the files that failed loading during last run |
Bot @files:save-content-to-file
Bot Position In Pipeline: Sink
Save content from 'content' column into file specified by 'filename' column
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| filename_column | Text | filename | Column with filename to be saved per each row |
| content_column | Text | content | File content column |
| encoding_column | Text | encoding | Column name with encoding type. Encoding type can be 'text' or 'binary'. If column not found, uses 'text' encoding |
| minio_location | Text | Location to save the file in minio |
Bot @files:savefile
Bot Position In Pipeline: Sink
Save dataframe to file using filename, filetype and sheet parameters
This bot expects a Restricted CFXQL.
Each parameter may be specified using '=' operator and AND logical operation
Following are the parameters expected for this Bot
| Parameter Name | Type | Default Value | Description |
|---|---|---|---|
| filename* | Text | Filename for the output data | |
| filetype | Text | auto | File type is either csv/json/xlsx. If not specified, it will be derived from extension. |
| sheet | Text | Sheet 1 | Specify sheet name if file type is xlsx. If not specified, sheet name will be 'Sheet 1' |