Skip to content

Add logstash parsers for cmsweb services #76

@vkuznet

Description

@vkuznet

This the placeholder ticket to add cmsweb service specific parsers to logstash. For simplicity let's collect all messages collected by filebeats:

  • workqueue
  • reqmon
"message":"[14/Oct/2019:16:39:19] reqmon-df69c6598-tk58z 127.0.0.1 \"GET /wmstatsserver/data/info HTTP/1.1\" 200 OK [data: 346 in 39 out 1982 us ] [auth: OK \"\" \"\" ] [ref: \"\" \"ServerMonitor-reqmon\" ]"
  • reqmgr2
[15/Oct/2019:02:00:39] reqmgr2-89f7df4fd-95g9q 137.138.33.200 "GET /reqmgr2/data/request?name=amaltaro_TaskChain_InclParents_Oct2019_Val_191010_125845_9547 HTTP/1.1" 200 OK [data: 2369 in 1862 out 530492 us ] [auth: OK "/DC=ch/DC=cern/OU=computers/CN=wmagent/vocms0192.cern.ch" "" ] [ref: "" "WMCore.Services.Requests/v002" ]
  • reqmgr2ms
"message":"[14/Oct/2019:16:40:37] reqmgr2ms-6796895484-7vzj6 127.0.0.1 \"GET /ms-transferor/data/status HTTP/1.1\" 200 OK [data: 351 in 158 out 683 us ] [auth: OK \"\" \"\" ] [ref: \"\" \"ServerMonitor-reqmgr2ms\" ]"
  • couchdb
"message":"[Mon, 14 Oct 2019 14:39:59 GMT] [info] [<0.26674.38>] 127.0.0.1 - - GET /workqueue/_design/WorkQueue/index.html 200"
  • phedex
"message":"::1 - - - [14/Oct/2019:16:44:48 +0200] \"GET /phedex/datasvc/doc HTTP/1.1\" 200 15229 \"-\" \"ServerMonitor-phedex\""
  • dbs
"message":"INFO:cherrypy.access:[14/Oct/2019:16:43:52] dbs-phys03-w-5fd9b6ffc-9kqxb 127.0.0.1 \"GET /dbs/ HTTP/1.1\" 200 OK [data: 298 in 468 out 1320 us ] [auth: OK \"\" \"\" ] [ref: \"\" \"ServerMonitor/2.0\" ]"
"message":"127.0.0.1 - - [14/Oct/2019:16:45:47] \"GET / HTTP/1.1\" 200 22 \"\" \"ServerMonitor/2.0\""
  • das
{"DASQuery":{"query":"dataset=/TTToSemiLeptonic_TuneCP5_13TeV-powheg-pythia8/RunIIFall17MiniAODv2-PU2017_12Apr2018_94X_mc2017_realistic_v14-v2/MINIAODSIM","hash":"705c64bd8a82ee31aa9607dada4a6208","spec":{"dataset":"/TTToSemiLeptonic_TuneCP5_13TeV-powheg-pythia8/RunIIFall17MiniAODv2-PU2017_12Apr2018_94X_mc2017_realistic_v14-v2/MINIAODSIM"},"fields":["dataset"],"pipe":"","instance":"prod/global","detail":false,"system":"","filters":{},"aggregators":[],"error":"","tstamp":1570577955},"PID":"705c64bd8a82ee31aa9607dada4a6208","ProcessTime":10.645284837,"Unix":1570577956,"level":"info","msg":"ready","time":"2019-10-09T01:39:16+02:00"}
  • crabserver
"message":"[14/Oct/2019:16:45:22]  RESTSQL:ovUDzLxBPpVb RELEASED cmsweb_analysis_preprod@devdb11 timeout=300 inuse=0 idle=1"
  • crabcache
"message":"[14/Oct/2019:16:46:21] crabcache-6c7f6559d6-5ksbz 127.0.0.1 \"GET /crabcache/info HTTP/1.1\" 200 OK [data: 340 in 69 out 608 us ] [auth: OK \"\" \"\" ] [ref: \"\" \"ServerMonitor-crabcache\" ]"
  • dqmgui
  • t0wmadatasvc
"message":"[14/Oct/2019:16:44:21] t0wmadatasvc-74d87c769b-p8zcx 127.0.0.1 \"GET /t0wmadatasvc/prod/hello HTTP/1.1\" 200 OK [data: 352 in 25 out 64701 us ] [auth: OK \"\" \"\" ] [ref: \"\" \"ServerMonitor-t0wmadatasvc\" ]"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions