Skip to content

Pipelines as Jinja Templates

1. Jinja Template Support in RDA 8.0

Overview

RDA 8.0 introduces a powerful new feature that enables users to create Jinja templates for dynamically generating pipelines. This enhancement offers greater flexibility and control over pipeline execution.

1.1 Key Capabilities

  • Pipeline Generation with Jinja – Users can define pipelines using the Jinja template engine. The final pipeline is rendered just before execution, ensuring adaptability to runtime conditions.

  • Runtime Context – Supports a JSON-based runtime context, which can be injected into the Jinja template to dynamically generate the final pipeline.

  • Model Schema – Users can define a model schema (as a JSON Schema) to validate and enforce the structure of input data expected at runtime, ensuring consistency and correctness. For more details on JSON Schema please click here.

  • Dynamic Forms – Users can also define the schema as a dynamic form, to render a form that user can use to pass the context to pipeline from the UI. For more information on Dynamic Forms please click here.

1.2 Benefits

This functionality allows users to dynamically adjust pipeline behavior based on runtime parameters, improving adaptability and reducing the need for manual modifications.

  • Ability to create pipelines using jinja template engine. The actual pipeline will be created by rendering jinja template just before actual execution of the pipeline.

  • Ability to take runtime context (A json object), that can be used in jinja template to render the final pipeline to be executed.

  • Ability to specify a model schema to enforce the data model of the inputs expected at runtime by the jinja template.

  • Ability to change the actual functionality of the pipeline based on certain run time inputs.

2. Example Use Cases

  • Dynamic Credential and Bot Selection – In an MSP deployment, pipelines may need to use different credentials or bots depending on the customer_id. By passing customer_id as part of the runtime context, a Jinja template can dynamically generate pipeline content tailored to each customer’s credentials.

  • Looping Through Bots Based on Query Results – Instead of using dataloops, users can leverage Jinja templating to simplify the process of iterating through different bots based on the results of a pstream query.

  • Dynamic Webhook ID Resolution – Instead of hardcoding webhook URLs, Jinja templates can be used to query and dynamically retrieve webhook IDs based on endpoint names. Since webhook IDs are typically UUID-based and vary across systems, this approach enhances portability by eliminating the need for hardcoded values.

3. Sections of Pipeline Template

  • Add a directive to indicate this pipeline is template based.
%%template_type="jinja"

Supported Template Types:

Supported Input Models:

  • Pipeline template require inputs at runtime. A “model” should be defined to define expected run time inputs and their constraints.

  • This model can be defined as either json schema and/or RDAF dynamic form format.

  • More information about json schema: https://json-schema.org/

    - Examples: https://json-schema.org/learn/miscellaneous-examples

  • To define input as json schema following construct is used:

%%start_inline_object type="context_schema"
{
….
….
}
%%end_inline_object
  • To define input as dynamic form following construct is used:
%%start_inline_object type="context_dynamic_form" 
{
…
…
}
%%end_inline_object

4. Example of Json Schema

  • This pipeline containing following construct requires 2 parameters customer_id and pstream to be passed at runtime.
%%start_inline_object type="context_schema"
{
    "type": "object",
    "properties": {
        "customer_id": {
            "type": "string"
        },
        "pstream": {
            "type": "string"
        }
    },
    "required": [
        "pstream",
        "customer_id"
    ]
}
%%end_inline_object

5. Example of Dynamic Form

  • Dynamic form can be defined to indicate the UI to create a form when user tries to execute the pipeline manually.

  • The pipeline containing following construct will display customer_id & pstream as 2 required inputs, when the pipeline is run manually.

%%start_inline_object name="context_dynamic_form" and type="data" and format="json"
{
    "formFieldList": [
        {
            "fieldId": "customer_id",
            "label": "ID of the customer",
            "required": true,
            "dataType": "string"
        },
        {
            "fieldId": "pstream",
            "label": "Pstream to write results",
            "required": true,
            "dataType": "string"
        }
    ]
}
%%end_inline_object

6. Contents of the Pipeline

  • Following is a jinja template pipeline that uses the values customer_id & pstream that are passed to it at run time.

  • The Jinja template in the pipeline queries pstream rda_secrets_meta using customer_id.

  • Loops through the results and calls various bots in a loop.

  • write-stream writes to a pstream that is passed at run time.

%% import_src_type = "linux-inventory"
@dm:empty
-->@dm:save name="temp-netstat"
-->@dm:addrow ip_address="10.95.125.190"
-->@dm:addrow ip_address="10.95.125.192"
-->@dm:save name="temp-input-dataset"
{% for cred in engine.query_stream_data("rda_secrets_meta", cfxql_query="name contains '" ~ customer_id ~ "' and type is 'linux-inventory'") %}
  -->@dm:recall name="temp-input-dataset"
  -->@{{cred.name}}:system-info column_name="ip_address"
  -->@dm:save name="temp-netstat" and append="yes"
{% endfor %}
-->@dm:recall name="temp-netstat"
-->@rn:write-stream name="{{pstream}}"

7. Executing the pipeline - Manual Execution From UI

Json Schema

Json Schema

Json Schema

8. Executing the Pipeline - Blueprint

Json Schema

9. Executing the Pipeline - Studio

  • Provide an inline object named run_context for studio to process the pipeline substitution and execute.
%%start_inline_object name="run_context" and type="data" and format="json"
{
   "customer_id": "OmniCorp",
   "pstream": "linux_inventory_stream"
}
%%end_inline_object

10. Executing the Pipeline - rdac

  • The runtime context required to run the pipeline can be passed using --runtime-context as json string.

Json Schema