Skip to content

Guide to Application Dependency Mapping using RDA Bots

This guide has following sections:

  1. Application Dependency Mapping (ADM) concept in RDA
  2. Application Dependency Mapping (ADM) artifacts in RDA
  3. Creating Nodes dataset for an ADM topology stack in RDA
  4. Creating Edges dataset for an ADM topology stack in RDA
  5. Generating an ADM topology stack in RDA

1. Application Dependency Mapping concept in RDA

Application Dependency Mappings (ADM) are essential for implementation of any Observability and AIOps projects. RDA provides necessary bots to automatically create & manage ADMs from any Cloud Native, Traditional Datacenter and Hybrid Cloud infrastructure.

RDA uses Stacks artifact as a way to refer to Application Dependency Mappings. Using Stacks, following can be accomplished:

  • Automatically identify impacted entities when an alert is detected (Impact Analysis)
  • Root cause identification
  • Identify cohorts (Similar entities in a business application): Critical for doing metric & event correlation.

Application dependency mapping in RDA is established using LRID (left to right impact declaration) model. Within the LRID model, each object is identified as either impacting node (left) or impacted node (right).

The direction of the impact is from left node to right node, i.e. node(s) listed under Left node impacts the node(s) listed under Right node. Below table provides a sample example of LRID model. Relation Type defines how Left node is related to Right node.

Left node Right node Relation Type
web01 bizapp_ecommerce member-of
app01 web01 depended-by
dbhost01 app01 depended-by
esxihost01 vm01 runs
datastore01 vm01 contains
lun01 datastore01 depended-by
storage_controller01 lun01 contains

In the above example of application dependency mapping for a Business application called bizapp_ecommerce, it's underlying resource objects are defined as either Left node or Right node based on how they impact their neighbouring nodes and subsequently the upstream Business application.

What is a node ? - A node is an application or infrastructure component such as Web server, App server, Database, Virtual Machine, Switch, Server, Storage, Volume etc. which are connected to each other within the ADM stack.

ADM

2. Application Dependency Mapping artifacts in RDA

In order to generate an application dependency mapping (ADM) topology as depicted in the above picture, below are two types of datasets that need to be generated.

  • Nodes dataset: Nodes dataset consists of the properties of each node (such as application component, vm, server, switch, storage volume etc.) within the ADM topology. Additional to selecting each node's important properties such as name, ip address, uuid, mac address, version etc., below are the mandatory properties that need to be defined for each node:

    • layer: Supported values are application, application component, virtualization, compute, network, storage
    • node_id: Unique identifier of a node within the ADM topology
    • node_type: Supported values are application, Webserver, Appserver, Database, Infraservice, Container, Process, Service, VM, OS, Host, Server, Switch, Router, Firewall, Load Balancer, LUN, Volume, Storage Pool, Storage Controller
    • node_label: User friendly label name for each node within the ADM topology

  • Edges dataset: In edges dataset, each node is classified either as left node or right node using it's node_id that was defined in the above nodes dataset. Below are the properties that need to be defined within the edges dataset.

    • left_id: node_id of a node which is an impacting node
    • right_id: node_id of a node which is an impacted node
    • relation_type: It defines how Left node is related to Right node. Below are some of the supported relation types, but not limited to
      • runs
      • runs-on
      • connects
      • connected-to
      • member-of
      • contains
      • depended-by

3. Creating Nodes dataset for an ADM topology stack in RDA

Below one is a sample pipeline that creates a Nodes dataset using 2 tier application stack's sample data.

Info

When any of the node_id field or any other field which is going to be used as left_id or right_id and if it has has numeric value, please make sure to set it's data type as String using the below bot. 

1
--> @dm:to-type columns = 'node_id' & type = 'str'


  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
## A Sample RDA pipeline to create ADM nodes dataset using the demo data
## The Demo data was generated from an APM tool which has the inventory data of one or more Business applications,
## Web/App services, Database services etc.

## Step-1:
## Load the business application dataset and define each business application node's properties.

## Start a new block in the pipeline
--> @c:new-block
    ## Load the demo business application dataset
    --> @files:loadfile
            filename = 'https://bot-docs.cloudfabrix.io/data/datasets/demo_adm_biz_application_dataset.csv'

    ## Filter stale applications if there are any
    --> *dm:filter app_name does not contain 'Petclinic'

    ## Add a new field called 'layer' and define it as 'Application'
    --> @dm:eval
            layer = "'Application'"

    ## Add a new field called 'node_type' and define it as 'Application'
    --> @dm:eval
            node_type = "'Application'"

    ## Clone the field 'app_name' to 'node_label'
    --> @dm:map 
            from = 'app_name' & to = 'node_label'

    ## Clone the field 'app_id' to 'node_id'
    --> @dm:map 
            from = 'app_id' & to = 'node_id'

    ## Set the app_id & node_id columns / field's data type as String
    --> @dm:to-type columns = 'app_id,node_id' & type = 'str'

    ## Save the business application nodes data as a temporary in-memory dataset for further use within this pipeline
    --> @dm:save
            name = 'temp-biz_application_nodes'

## Step-2:
## Load the database services dataset and define each database service's node properties.

## Start a new block in the pipeline
--> @c:new-block

    ## Load the demo database service connections dataset
    --> @files:loadfile
            filename = 'https://bot-docs.cloudfabrix.io/data/datasets/demo_adm_apps_to_db_conn_dataset.csv'

    ## Filter the records from 'exitPointType' field if it contains DB and JDBC values, remove any stale application connections and get the required fields
    --> *dm:filter 
            exitPointType contains 'DB|JDBC' &
            app_name does not contain 'Petclinic'
            get HOST,app_name

    ## Remove the duplicate records using HOST field
    --> @dm:dedup 
            columns = 'HOST'

    ## Save the database service connections data as a temporary in-memory dataset for further use within this pipeline
    --> @dm:save 
            name = 'temp-db_backend_connections'

## Start a new block in the pipeline
--> @c:new-block

    ## Load the demo database servers dataset
    --> @files:loadfile
            filename = 'https://bot-docs.cloudfabrix.io/data/datasets/demo_adm_db_server_nodes_dataset.csv'

    ## Filter the records that contain any stale DB services and get the required fields and rename them
    --> *dm:filter name does not contain 'linux-petclinic-db01'
            get host as 'db_dns_name',ipAddress as 'db_ip_address',name as 'db_hostname',id as 'db_id'

    ## Set the db_id column / field's data type as String
    --> @dm:to-type columns = 'db_id' & type = 'str'

    ## Remove the duplicate records using db_ip_address field
    --> @dm:dedup 
            columns = 'db_ip_address'

    ## Save the database servers data as a temporary in-memory dataset for further use within this pipeline
    --> @dm:save 
            name = 'temp-db_servers'

## Start a new block in the pipeline
--> @c:new-block

    ## Reload the saved in-memory dataset 'temp-db_servers'
    --> @dm:recall 
            name = 'temp-db_servers'

    ## Enrich the DB server dataset with DB service connections dataset and add the app_name field
    --> @dm:enrich 
            dict = 'temp-db_backend_connections' & 
            src_key_cols = 'db_ip_address' & 
            dict_key_cols = 'HOST' & 
            enrich_cols = 'app_name'

    ## Remove/Exclude a field called HOST
    --> @dm:selectcolumns 
            exclude = '^HOST$'

    ## Add a new field called 'layer' and define it as 'Application Component'
    --> @dm:eval
            layer = "'Application Component'"

    ## Add a new field called 'node_type' and define it as 'Database'
    --> @dm:eval
            node_type = "'Database'"

    ## Rename the field 'db_id' as 'node_id'
    --> @dm:rename-columns
            node_id = 'db_id'

    ## Derive database node's 'node_label' field from other fields 'db_hostname' & 'node_type'
    --> @dm:map
            from = 'db_hostname,node_type' & 
            to = 'node_label' & 
            func = 'join' & 
            sep = '_'

    ## Enrich the database node's dataset with business application's node_id
    --> @dm:enrich 
            dict = 'temp-biz_application_nodes' & 
            src_key_cols = 'app_name' & 
            dict_key_cols = 'app_name' & 
            enrich_cols = 'app_id' & 
            enrich_cols_as = 'biz_app_id'

    ## Set the biz_app_id column / field's data type as String
    --> @dm:to-type columns = 'biz_app_id' & type = 'str'

    ## Save the database nodes data as a temporary in-memory dataset for further use within this pipeline
    --> @dm:save
            name = 'temp-application_database_nodes'


## Step-3:
## Load the applications (web/app server) dataset and define each application service's node properties.

## Start a new block in the pipeline
--> @c:new-block

    ## Load the demo applications (web/app server) dataset
    --> @files:loadfile
            filename = 'https://bot-docs.cloudfabrix.io/data/datasets/demo_adm_application_nodes_dataset.csv'

    ## Filter the records from 'ipAddress' field if it has a special character and a private IP address,
    ## remove any stale application services and get few required fields and rename them
    --> *dm:filter
            ipAddress does not contain ':' & 
            ipAddress does not contain '172.17.0.1' &
            ipAddress does not contain '192.168.115' &
            app_name does not contain 'Petclinic'
            get name as 'app_hostname',
            tierName as 'app_tier',
            ipAddress as 'app_ip_address',
            app_name as 'app_web_name',
            id as 'app_web_id'

    ## Set the app_web_id column / field's data type as String
    --> @dm:to-type columns = 'app_web_id' & type = 'str'

    ## Filter the application nodes that are Web server applications
    --> *dm:filter
            app_tier = 'Web'

    ## Add a new field called 'layer' and define it as 'Application Component'
    --> @dm:eval
            layer = "'Application Component'"

    ## Add a new field called 'node_type' and define it as 'Webserver'
    --> @dm:eval
            node_type = "'Webserver'"

    ## Rename the field 'app_id' as 'node_id'
    --> @dm:rename-columns
            node_id = 'app_web_id'

    ## Derive application service's 'node_label' field from other fields 'app_name' & 'node_type'
    --> @dm:map
            from = 'app_web_name,node_type' & 
            to = 'node_label' & 
            func = 'join' & 
            sep = '_'

    ## Enrich the application service node's dataset with business application's node_id
    --> @dm:enrich
            dict = 'temp-biz_application_nodes' & 
            src_key_cols = 'app_web_name' & 
            dict_key_cols = 'app_name' & 
            enrich_cols = 'app_id' & 
            enrich_cols_as = 'biz_app_id'

    ## Set the biz_app_id column / field's data type as String
    --> @dm:to-type columns = 'biz_app_id' & type = 'str'

    ## Remove/Exclude a field called app_name & app_web_name
    --> @dm:selectcolumns 
            exclude = '^app_name$|^app_web_name$'

    ## Enrich the application service node's dataset with database service's node_id
    --> @dm:enrich
            dict = 'temp-application_database_nodes' & 
            src_key_cols = 'biz_app_id' & 
            dict_key_cols = 'biz_app_id' & 
            enrich_cols = 'node_id' & 
            enrich_cols_as = 'db_node_id'

    ## Set the db_node_id column / field's data type as String
    --> @dm:to-type columns = 'db_node_id' & type = 'str'

    ## Save the application web/app service nodes data as a temporary in-memory dataset for further use within this pipeline
    --> @dm:save
            name = 'temp-application_web_service_nodes'

## Step-4:
## Concatenate all of the nodes of Business application, Web/App services and Database services

## Start a new block in the pipeline
--> @c:new-block

    ## Use the below bot to combine all of node datasets that were created above
    --> @dm:concat names = '^temp-biz_application_nodes$|^temp-application_database_nodes$|^temp-application_web_service_nodes$'

    ## Set the node_id column / field's data type as String
    --> @dm:to-type columns = 'node_id' & type = 'str'

    ## Save the combined nodes as ADM nodes dataset
    --> @dm:save name = 'rda_adm_biz_app_db_nodes_databaset'

4. Creating Edges dataset for an ADM topology stack in RDA

Below one is a sample pipeline that creates a Edges dataset using 2 tier application stack's sample data.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
## A Sample RDA pipeline to create ADM 'Edges dataset' using the demo data
## The Demo data was generated from an APM tool which has the inventory data of one or more Business applications,
## Web/App services, Database services etc.
## 'Nodes dataset' that was generated will be used as input for populating the 'Edges Dataset'

## Step-1:
## Load the 'Nodes dataset' and establish relationship between Web/App, DB and Biz application nodes

## Start a new block in the pipeline
--> @c:new-block

    ## Reload the 'Nodes dataset'
    --> @dm:recall 
            name = 'rda_adm_biz_app_db_nodes_databaset'

    ## Define relationship between Web/App and Biz application nodes
    ## Here, Web/App node is going to be classified as impacting node (Left node) and 
    ## Biz application node is going to be classified as impacted node (Right node)

    ## Filter the Web/App nodes
    --> *dm:filter node_type = 'Webserver'
            get node_id as 'left_id',
            biz_app_id as 'right_id'

    ## Add relationship type between Web/App and Biz application nodes
    --> @dm:eval 
            relation_type = "'member-of'"

    ## Clone the left_id field as left
    --> @dm:map 
            from = 'left_id' &
            to = 'left'

    ## Clone the right_id field as right
    --> @dm:map 
            from = 'right_id' &
            to = 'right'

    ## Save it as in-memory dataset for further use with in the pipeline
    --> @dm:save
            name = 'temp-web_biz_app_relationship'

    ## Define relationship between Database and Biz application nodes
    ## Here, Database node is going to be classified as impacting node (Left node) and 
    ## Biz application node is going to be classified as impacted node (Right node)

    ## Reload the 'Nodes dataset'
    --> @dm:recall 
            name = 'rda_adm_biz_app_db_nodes_databaset'

    ## Filter the Database nodes
    --> *dm:filter node_type = 'Database'
            get node_id as 'left_id',
            biz_app_id as 'right_id'

    ## Add relationship type between Database and Biz application nodes
    --> @dm:eval 
            relation_type = "'member-of'"

    ## Clone the left_id field as left
    --> @dm:map 
            from = 'left_id' &
            to = 'left'

    ## Clone the right_id field as right
    --> @dm:map 
            from = 'right_id' &
            to = 'right'

    ## Save it as in-memory dataset for further use with in the pipeline
    --> @dm:save
            name = 'temp-db_biz_app_relationship'


    ## Define relationship between Web/App and Database nodes
    ## Here, Database node is going to be classified as impacting node (Left node) and 
    ## Web/App node is going to be classified as impacted node (Right node)

    ## Reload the 'Nodes dataset'
    --> @dm:recall 
            name = 'rda_adm_biz_app_db_nodes_databaset'

    ## Filter the Database nodes
    --> *dm:filter node_type = 'Webserver'
            get node_id as 'right_id',
            db_node_id as 'left_id'

    ## Add relationship type between Database and Biz application nodes
    --> @dm:eval 
            relation_type = "'connected-by'"

    ## Clone the left_id field as left
    --> @dm:map 
            from = 'left_id' &
            to = 'left'

    ## Clone the right_id field as right
    --> @dm:map 
            from = 'right_id' &
            to = 'right'

    ## Save it as in-memory dataset for further use with in the pipeline
    --> @dm:save
            name = 'temp-webapp_db_relationship'

## Step-2:
## Concatenate all of the Edge (relationships) datasets of Business application, Web/App services and Database services

## Start a new block in the pipeline
--> @c:new-block

    ## Use the below bot to combine all of edge (relationships) datasets that were created above
    --> @dm:concat names = '^temp-web_biz_app_relationship$|^temp-db_biz_app_relationship$|^temp-webapp_db_relationship$'

    ## Set the left,lef_id,right,right_id columns / field's data type as String
    --> @dm:to-type columns = 'left,left_id,right,right_id' & type = 'str'

    ## Save the combined nodes as ADM nodes dataset
    --> @dm:save name = 'rda_adm_biz_app_db_edges_databaset'

5. Generating an ADM topology stack in RDA

Below sample pipeline generates a final ADM stack processing both Nodes dataset and Edges dataset

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
## A Sample RDA pipeline to create an ADM stack using both 'Nodes dataset' and 'Edges dataset'.

## Start a new block in the pipeline
--> @c:new-block

    ## Create an ADM application stack by processing both 'Nodes dataset' and 'Edges dataset' created above
    --> @dm:stack-create topology_nodes='rda_adm_biz_app_db_nodes_databaset' and topology_edges='rda_adm_biz_app_db_edges_databaset' and name = 'adm_2_tier_stacks'

    ## Save the ADM topology stack
    --> @dm:stack-save name='adm_2_tier_stacks'

## Start a new block in the pipeline
--> @c:new-block

    ## Generate an ADM application topology stack for visualization
    --> @dm:stack-topology input_stack='adm_2_tier_stacks' and stack_label='2 Tier ADM Topology Stacks' and infra = 'true'

To see the final ADM topology stack in cfxCloud portal, go to Data menu, under Dependency Mappings, click on View Details to see the newly created adm_2_tier_stacks

ADM