본문 바로가기

빅데이터/Kafka

telegraf사용시 kafka로 데이터 json형태로 보내는 방법

telegraf는 metric수집에 최적화되어있습니다. 다만, 그대로 사용하게 되면 influxdb에서 사용하는 데이터형태(csv와 유사)로 데이터가 전송되는데요. kafka에서 데이터를 잘 활용하기 위해 json을 사용한다면 consumer에서도 스키마에 대한 걱정없이(?) 잘 사용할 수 있습니다.

 

telegraf에서 data형태를 json형태로 바꾸는 예제를 아래에 설명드리도록 하겠습니다. telegraf에 대한 설명은 아래 링크를 확인해주세요.

https://blog.voidmainvoid.net/260

 

Fluentd vs Telegraf 차이점 알아보기

Telegraf와 fluentd는 아주 유사해보인다. 둘다 configuration파일 기반으로 작동하며 plugin을 통해 개발자가 custom하게 만든 input, filter, output 플러그인들을 사용하여 데이터를 처리, 전송 가능하다. 이..

blog.voidmainvoid.net

Telegraf에서 cpu, memory 수집해서 kafka로 보내기

telegraf는 input, process, output으로 이루어져 있습니다. input을 cpu와 ram메트릭으로하고 output을 kafka로 설정하였습니다.

[agent]
  interval = "10s"

[[outputs.kafka]]
  brokers = ["localhost:9092"]
  ## Kafka topic for producer messages
  topic = "test"
  data_format = "json"

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  fielddrop = ["time_*"]

[[inputs.mem]]

위 설정에 대한 설명은 아래와 같습니다.
- 메트릭을 수집하는 간격은 10초
- 브로커는 localhost의 카프카
- 토픽이름은 test
- 데이터 형태는 json
- 수집하는 메트릭은 cpu(percpu, totalcpu 포함), ram

수집한 데이터의 결과를 확인하면 아래와 같습니다. 수집된 데이터를 kafka-console-producer로 확인하였습니다.

./kafka-console-consumer --bootstrap-server localhost:9092 --topic test     
{"fields":{"active":13227597824,"available":18011025408,"available_percent":52.4189829826355,"buffered":0,"cached":0,"commit_limit":0,"committed_as":0,"dirty":0,"free":7449378816,"high_free":0,"high_total":0,"huge_page_size":0,"huge_pages_free":0,"huge_pages_total":0,"inactive":10561646592,"low_free":0,"low_total":0,"mapped":0,"page_tables":0,"shared":0,"slab":0,"sreclaimable":0,"sunreclaim":0,"swap_cached":0,"swap_free":0,"swap_total":0,"total":34359738368,"used":16348712960,"used_percent":47.5810170173645,"vmalloc_chunk":0,"vmalloc_total":0,"vmalloc_used":0,"wired":3116810240,"write_back":0,"write_back_tmp":0},"name":"mem","tags":{"host":"SKP1003855MA0001.local"},"timestamp":1594174970}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":98.86257108930693,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.44997187675770267,"usage_user":0.687457033935379},"name":"cpu","tags":{"cpu":"cpu-total","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":100,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu15","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.4005994005994,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.1998001998001998,"usage_user":0.3996003996003996},"name":"cpu","tags":{"cpu":"cpu14","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.9,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.1,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu13","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":98.7,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.3,"usage_user":1},"name":"cpu","tags":{"cpu":"cpu12","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.9,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.1,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu11","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.1991991991992,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.3003003003003003,"usage_user":0.5005005005005005},"name":"cpu","tags":{"cpu":"cpu10","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.8001998001998,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.0999000999000999,"usage_user":0.0999000999000999},"name":"cpu","tags":{"cpu":"cpu9","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.5,"usage_user":0.5},"name":"cpu","tags":{"cpu":"cpu8","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

{"fields":{"usage_guest":0,"usage_guest_nice":0,"usage_idle":99.9,"usage_iowait":0,"usage_irq":0,"usage_nice":0,"usage_softirq":0,"usage_steal":0,"usage_system":0.1,"usage_user":0},"name":"cpu","tags":{"cpu":"cpu7","host":"SKP1003855MA0001.local"},"timestamp":1594174980}

json형태이고 데이터를 prettify해보면 json계층이 잘 이루어진것을 확인할 수 있습니다.

{
  "fields": {
    "usage_guest": 0,
    "usage_guest_nice": 0,
    "usage_idle": 99.9,
    "usage_iowait": 0,
    "usage_irq": 0,
    "usage_nice": 0,
    "usage_softirq": 0,
    "usage_steal": 0,
    "usage_system": 0.1,
    "usage_user": 0
  },
  "name": "cpu",
  "tags": {
    "cpu": "cpu7",
    "host": "SKP1003855MA0001.local"
  },
  "timestamp": 1594174980
}

cpu이름과 host이름이 태깅되어있고 name키로 어떤 데이터인지 확인합니다. field에서는 해당 데이터의 요소들을 확인할 수 있습니다.