Skip to content

li-replay-logs-to-dev-env

Pipeline: li-replay-logs-to-dev-env

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
## read data from stream
@rn:read-stream
            name = "replay-server-syslogs"   &
            group = "group-replay-server-syslogs" &
            batch_count = '1000' &
            batch_wait_time = '10'
    --> @dm:add-missing-columns
            columns="rda_gw_client_ip,syslog_severity" &
            value="None"
    --> @dm:enrich-using-rule-dict
            dict = "petclinic-env-dict" &
            rule_column = "rule" &
            enrich_columns = "env"
    --> @dm:eval
            syslog_severity = "syslog_severity.upper()"
    --> @rn:write-stats-to-stream
            name = "log-intel-replay-stats" &
            groupby = "env,syslog_severity" &
            type = "syslog" &
            mode = "replay"
    ## Fix any null or nan values in data
    --> @dm:fixnull-regex

    ## Append to Elasticsearch index
    --> @es-dev-v2:append-index
            index="petclinic_logs_from_archive"


Extensions used in this Pipeline

Source Name Extension Type
rn rn
es-dev-v2 elasticsearch_v2


Artifacts used in this Pipeline

Artifact Type Artifact Name Access
rda-network-stream replay-server-syslogs read
dataset petclinic-env-dict read
rda-network-stream log-intel-replay-stats write


Bots used in this Pipeline

@rn:read-stream @dm:add-missing-columns @dm:enrich-using-rule-dict @dm:eval @rn:write-stats-to-stream @dm:fixnull-regex @es-dev-v2:append-index