telegraf사용시 kafka로 데이터 json형태로 보내는 방법

telegraf는 metric수집에 최적화되어있습니다. 다만, 그대로 사용하게 되면 influxdb에서 사용하는 데이터형태(csv와 유사)로 데이터가 전송되는데요. kafka에서 데이터를 잘 활용하기 위해 json을 사용한다면 consumer에서도 스키마에 대한 걱정없이(?) 잘 사용할 수 있습니다.

 

telegraf에서 data형태를 json형태로 바꾸는 예제를 아래에 설명드리도록 하겠습니다. telegraf에 대한 설명은 아래 링크를 확인해주세요.

https://blog.voidmainvoid.net/260

Fluentd vs Telegraf 차이점 알아보기

Telegraf에서 cpu, memory 수집해서 kafka로 보내기

telegraf는 input, process, output으로 이루어져 있습니다. input을 cpu와 ram메트릭으로하고 output을 kafka로 설정하였습니다.

[agent]
  interval = "10s"

[[outputs.kafka]]
  brokers = ["localhost:9092"]
  ## Kafka topic for producer messages
  topic = "test"
  data_format = "json"

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  fielddrop = ["time_*"]

[[inputs.mem]]

위 설정에 대한 설명은 아래와 같습니다.
- 메트릭을 수집하는 간격은 10초
- 브로커는 localhost의 카프카
- 토픽이름은 test
- 데이터 형태는 json
- 수집하는 메트릭은 cpu(percpu, totalcpu 포함), ram

수집한 데이터의 결과를 확인하면 아래와 같습니다. 수집된 데이터를 kafka-console-producer로 확인하였습니다.

./kafka-console-consumer --bootstrap-server localhost:9092 --topic test     
{"fields":{"active":13227597824,"available":18011025408,"available_percent":52.4189829826355,"buffered":0,"cached":0,"commit_limit":0,"committed_as":0,"dirty":0,"free":7449378816,"high_free":0,"high_total":0,"huge_page_size":0,"huge_pages_free":0,"huge_pages_total":0,"inactive":10561646592,"low_free":0,"low_total":0,"mapped":0,"page_tables":0,"shared":0,"slab":0,"sreclaimable":0,"sunreclaim":0,"swap_cached":0,"swap_free":0,"swap_total":0,"total":34359738368,"used":16348712960,"used_percent":47.5810170173645,"vmalloc_chunk":0,"vmalloc_total":0,"vmalloc_used":0,"wired":3116810240,"write_back":0,"write_back_tmp":0},"name":"mem","tags":{"host":"SKP1003855MA0001.local"},"timestamp":1594174970}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":98.86257108930693,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.44997187675770267,"usage_user":0.687457033935379},"name":"cpu","tags":{"cpu":"cpu-total","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":100,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu15","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.4005994005994,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.1998001998001998,"usage_user":0.3996003996003996},"name":"cpu","tags":{"cpu":"cpu14","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.9,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.1,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu13","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":98.7,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.3,"usage_user":1},"name":"cpu","tags":{"cpu":"cpu12","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.9,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.1,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu11","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.1991991991992,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.3003003003003003,"usage_user":0.5005005005005005},"name":"cpu","tags":{"cpu":"cpu10","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.8001998001998,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.0999000999000999,"usage_user":0.0999000999000999},"name":"cpu","tags":{"cpu":"cpu9","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.5,"usage_user":0.5},"name":"cpu","tags":{"cpu":"cpu8","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.9,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.1,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu7","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

json형태이고 데이터를 prettify해보면 json계층이 잘 이루어진것을 확인할 수 있습니다.

{
  "fields": {
    "usage_guest": 0,
    "usage_guest_nice": 0,
    "usage_idle": 99.9,
    "usage_iowait": 0,
    "usage_irq": 0,
    "usage_nice": 0,
    "usage_softirq": 0,
    "usage_steal": 0,
    "usage_system": 0.1,
    "usage_user": 0
  },
  "name": "cpu",
  "tags": {
    "cpu": "cpu7",
    "host": "SKP1003855MA0001.local"
  },
  "timestamp": 1594174980
}

cpu이름과 host이름이 태깅되어있고 name키로 어떤 데이터인지 확인합니다. field에서는 해당 데이터의 요소들을 확인할 수 있습니다.