^_^
文章 标签 分类 关于
^_^

目录

fluentd 镜像构建与常用配置

 收录于 Log
   约 4688 字   预计阅读 10 分钟 

镜像

基础镜像

quay.io/fluentd_elasticsearch/fluentd:v2.9.0. 基础镜像 Dockerfile 项目地址

构建自己的镜像

1
2
3
4
5
6

FROM quay.io/fluentd_elasticsearch/fluentd:v2.9.0

RUN gem uninstall fluent-plugin-kubernetes_metadata_filter --version=2.4.1 && \
    gem uninstall kubeclient --version=4.5.0 && \
    gem uninstall fluentd --version=1.8.0 && \
    gem install fluentd fluent-plugin-kubernetes_metadata_filter kubeclient fluent-plugin-kafka

配置

基本命令

配置文件主要由以下命令组成：

  • source: 确定输入源
  • match: 确定输出目标
  • filter: 确定事件处理管道
  • system: 设置系统范围的配置
  • label: 对输出和内部过滤器进行分组路由
  • include: 包含其他配置文件

详见官方文档

输入插件

in_tail

in_tail 输入插件允许 Fluentd 从文本文件的尾部读取事件。它的行为类似于tail -F命令。

安装

内置插件，无需安装。

示例与参数

示例:

1
2
3
4
5
6
7
8
9

<source>
@type tail
path /var/log/httpd-access.log
pos_file /var/log/td-agent/httpd-access.log.pos
tag apache.access
<parse>
@type apache2
</parse>
</source>

关键参数表:

名称 类型 默认 版本 说明
@type string tail 值必须是 tail
tag string required parameter 0.14.0 事件的标签
path string required parameter 0.14.0 读取路径，多个路径以',‘分隔
exclude_path array [] 0.14.0 监听列表中排出文件的路径
refresh_interval time 60(seconds) 0.14.0 刷新监视文件列表的间隔，当路径包含 * 时使用
read_from_head bool false 0.14.0 从文件头而不是底部开始读取日志
read_lines_limit integer 1000 0.14.0 每次 IO 读取的行数
pos_file string nil 0.14.0 强烈建议使用此参数，fluentd 会将其最后一次读取的位置记录到该文件中。一个源的多个位置可以记录在一个 pos_file 中。不要在多个 in_tail 输入源配置相同的 pos_file，这样会导致不可预知的错误。
指令 required 日志的格式，in_tail 使用解析器插件来解析当前日志。内置解析器见下表。

内置解析器:

  • regexp
  • apache2
  • apache_error
  • nginx
  • syslog
  • csv
  • tsv
  • ltsv
  • json
  • msgpack
  • multiline
  • none

解析器详细文档

fluent-plugin-systemd

systemd 输入插件从系统 journal 采集日志

安装

1

gem install fluent-plugin-systemd

示例与参数

输入示例:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

<source>
  @type systemd
  tag kubelet
  path /var/log/journal
  matches [{ "_SYSTEMD_UNIT": "kubelet.service" }]
  read_from_head true

  <storage>
    @type local
    path /var/log/fluentd-journald-kubelet-cursor.json
  </storage>

  <entry>
    fields_strip_underscores true
    fields_lowercase true
  </entry>
</source>

<match kubelet>
  @type stdout
</match>

<system>
  root_dir /var/log/fluentd
</system>

关键参数表:

名称 类型 默认 版本 说明
@type string systemd 值必须是 systemd
path string /var/log/journal systemd journal 日志路径
matches array 设置该配置项过滤出需要采集的日志，如 kernel 日志或特定 service 的日志
storage plugin 配置存储插件来存储 journald 游标
read_from_head bool false true 表示从头开始读，否则从末尾读，如果 cursor 存在则忽略该配置
entry filter 可以用来转换字段大小写，移除开头的下划线等
tag string required 该输入源的事件添加标签

systemd filter 插件对 journal 日志进行处理

过滤示例:

1
2
3
4
5
6
7

<filter kube-proxy>
  @type systemd_entry
  field_map {"MESSAGE": "log", "_PID": ["process", "pid"], "_CMDLINE": "process", "_COMM": "cmd"}
  field_map_strict false
  fields_lowercase true
  fields_strip_underscores true
</filter>

过滤插件

fluent-plugin-kubernetes_metadata_filter

Kubernetes元数据插件过滤器使用pod和名称空间元数据丰富容器日志记录。

安装

1

gem install fluent-plugin-kubernetes_metadata_filter

示例与参数

配置示例:

1
2
3
4
5
6
7

<filter kubernetes.var.log.containers.**.log>
  @type kubernetes_metadata
</filter>

<match **>
  @type stdout
</match>

输入示例:

1
2
3
4
5

{
  "log": "2015/05/05 19:54:41 \n",
  "stream": "stderr",
  "time": "2015-05-05T19:54:41.240447294Z"
}

输出示例:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

{
  "log": "2015/05/05 19:54:41 \n",
  "stream": "stderr",
  "docker": {
    "id": "df14e0d5ae4c07284fa636d739c8fc2e6b52bc344658de7d3f08c36a2e804115",
  }
  "kubernetes": {
    "host": "jimmi-redhat.localnet",
    "pod_name":"fabric8-console-controller-98rqc",
    "pod_id": "c76927af-f563-11e4-b32d-54ee7527188d",
    "container_name": "fabric8-console-container",
    "namespace_name": "default",
    "namespace_id": "23437884-8e08-4d95-850b-e94378c9b2fd",
    "namespace_annotations": {
      "fabric8.io/git-commit": "5e1116f63df0bac2a80bdae2ebdc563577bbdf3c"
    },
    "namespace_labels": {
      "product_version": "v1.0.0"
    },
    "labels": {
      "component": "fabric8Console"
    }
  }
}

关键参数表:

名称 类型 默认 说明
@type string kubernetes_metadata 必须为 kubernetes_metadata
kubernetes_url string apiserver url，如果在 pod 里可以默认读取对应的环境变量
apiVersion string v1 使用的 api 版本
ca_file string 用于Kubernetes服务器证书验证的CA文件的路径
verify_ssl bool true 验证SSL证书
client_cert string path to a client cert file to authenticate to the API server
client_key string path to a client key file to authenticate to the API server
tag_to_kubernetes_name_regexp string 用于从当前fluentd标签提取kubernetes元数据（pod名称，容器名称，名称空间）的正则表达式。
cache_size int 1000 k8s 元数据缓存大小，用来减少对API服务器的请求
cache_ttl int 3600 每个缓存元素的 TTL(单位为秒)，负值表示禁用 TTL
watch bool true 监听 k8s 上的 pods 元数据更新
watch_retry_interval int 10 观察连接失败时，重试退避的时间间隔（以秒为单位）。
annotation_match array [] 与 annotations 字段名称匹配的正则表达式数组。匹配的 annotation 将添加到日志记录。默认不采集 annotation 数据。
lookup_from_k8s_field bool true 如果存在kubernetes字段，从给定的子字段中查找元数据，例如kubernetes.namespace_name，kubernetes.pod_name等。
skip_labels bool false 不采集元数据里的所有标签字段
skip_container_metadata bool false 跳过部分容器元数据的采集，不会跳过 container_image,container_image_id 字段。
skip_master_url bool false 不采集元数据里的 master_url 字段
skip_namespace_metadata bool false 从元数据中跳过namespace_id字段。

输出插件

fluent-plugin-elasticsearch

fluent-plugin-kafka

安装

1

gem install fluent-plugin-kafka --no-document

input 示例与参数

示例:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

<source>
  @type kafka_group
  brokers kafka:9092
  consumer_group log-logstash-kafka-to-es
  topics log-logstash
  # use_record_time true
  # ruby-kafka consumer options
  max_bytes 10485760
  max_wait_time 5
  offset_commit_interval 5
  offset_commit_threshold 100
  start_from_beginning true
  @label @kubernetes
</source>

关键参数表:

名称 类型 默认 说明

output 示例与参数

示例:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

    <match kubernetes.**>
      @type kafka2
      # list of seed brokers
      brokers kafka:9092
      use_event_time true
      # sasl_over_ssl false
      # username brokeruser
      # password brokerpassword
      # buffer settings
      <buffer log-logstash>
        @type file
        path /var/log/fluentd-buffers/log-logstash.buffer
        flush_interval 30s
        chunk_limit_size 128M
        total_limit_size 1024M
        flush_mode interval
        overflow_action drop_oldest_chunk
      </buffer>
      # data type settings
      <format>
        @type json
      </format>
      # topic settings
      # topic_key test-topic
      default_topic log-logstash
      # producer settings
      required_acks 1
      compression_codec gzip
    </match>

关键参数表:

名称 类型 默认 说明
topic_key string 日志记录的那个字段可以用来表示 topic
default_topic string 这个一定要设置

使用 fluent-plugin-prometheus插件监控

使用 prometheus 插件暴露 fluentd 的指标。

安装

1

gem install fluent-plugin-prometheus

使用

prometheus input plugin

必须配置该插件来暴露其他插件采集的指标

示例:

1
2
3

<source>
  @type prometheus
</source>

关键参数表:

名称 类型 默认 说明
@type string 必须为 prometheus
bind string 0.0.0.0 binding interface
port int 24231 监听的端口
metrics_path string /metrics metrics HTTP endpoint
aggregated_metrics_path string /aggregated_metrics 如果使用多个 woker 模式，每个 worker 监听 port+fluent_worker_id 端口，为了一次采集所有 worker 的数据，可以直接使用该地址采集

prometheus_monitor input plugin

该插件收集 fluentd 的内部指标。

示例:

1
2
3

<source>
  @type prometheus_monitor
</source>

关键参数表:

名称 类型 默认 说明
labels 额外的为该指标添加标签
interval int 5 更新monitor_agent信息的时间间隔（以秒为单位）

采集的指标:

名称 类型 解释
fluentd_status_buffer_queue_length
fluentd_status_buffer_total_bytes
fluentd_status_buffer_newest_timekey
fluentd_status_buffer_oldest_timekey
fluentd_status_retry_count

prometheus_output_monitor input plugin

该插件收集Fluentd中输出插件的内部指标。跟 prometheus_monitor 相似，但专门用于输出插件。有很多 prometheus_monitor 不具备的指标，例如 num_errors, retry_wait 等。

示例:

1
2
3

<source>
  @type prometheus_output_monitor
</source>

关键参数表:

名称 类型 默认 说明
labels 额外的为该指标添加标签
interval int 5 更新monitor_agent信息的时间间隔（以秒为单位）

采集的指标:

名称 类型 input/output/buffer 解释
fluentd_output_status_retry_count output
fluentd_output_status_num_error output
fluentd_output_status_emit_count output
fluentd_output_status_retry_wait output
fluentd_output_status_emit_records output
fluentd_output_status_write_count output
fluentd_output_status_rollback_count output
fluentd_output_status_flush_time_count output
fluentd_output_status_slow_flush_count output
fluentd_output_status_buffer_total_bytes buffer
fluentd_output_status_buffer_stage_length buffer
fluentd_output_status_buffer_stage_byte_size buffer
fluentd_output_status_buffer_queue_length buffer
fluentd_output_status_buffer_queue_byte_size buffer
fluentd_output_status_buffer_newest_timekey buffer
fluentd_output_status_buffer_oldest_timekey buffer
fluentd_output_status_buffer_available_space_ratio buffer

prometheus_tail_monitor input plugin

该插件收集 fluentd 中 in_tail 插件的内部指标。

示例:

1
2
3

<source>
  @type prometheus_tail_monitor
</source>

关键参数表:

名称 类型 默认 说明
额外的为该指标添加标签
interval int 5 更新monitor_agent信息的时间间隔（以秒为单位）

采集的指标:

名称 类型 解释
fluentd_tail_file_position 插件从文件读取的当前字节
fluentd_tail_file_inode 文件的 inode 值

指标默认包含的标签:

名称 解释
plugin_id 给插件配置的 @id 值
type 插件名称。当前只支持 in_tail
path 文件路径

prometheus filter/output plugin

过滤/输出插件都从记录中检测指标，只需读取记录，不影响记录的值。

支持的指标类型

counter type

示例:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

<metric>
  name message_foo_counter
  type counter
  desc The total number of foo in message.
  key foo
  <labels>
    tag ${tag}
    host ${hostname}
    foo bar
  </labels>
</metric>

关键参数表:

名称 required/optional 说明
name required 指标名称
type required 指标类型
desc required 指标描述
key optional 使用该键名对应的值计算 counter，若未设置或为空，counter 每次加 1
optional 额外添加的标签
gauge type

示例:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

<metric>
  name message_foo_gauge
  type gauge
  desc The total number of foo in message.
  key foo
  <labels>
    tag ${tag}
    host ${hostname}
    foo bar
  </labels>
</metric>

关键参数表:

名称 required/optional 说明
name required 指标名称
type required 指标类型
desc required 指标描述
key required 使用该键名对应的值作为 gauge 的值
optional 额外添加的标签
summary type

示例:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

<metric>
  name message_foo
  type summary
  desc The summary of foo in message.
  key foo
  <labels>
    tag ${tag}
    host ${hostname}
    foo bar
  </labels>
</metric>

关键参数表:

名称 required/optional 说明
name required 指标名称
type required 指标类型
desc required 指标描述
key required 使用该键名对应的值作为 summary 的值
optional 额外添加的标签
histogram type

示例:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

<metric>
  name message_foo
  type histogram
  desc The histogram of foo in message.
  key foo
  buckets 0.1, 1, 5, 10
  <labels>
    tag ${tag}
    host ${hostname}
    foo bar
  </labels>
</metric>

关键参数表:

名称 required/optional 说明
name required 指标名称
type required 指标类型
desc required 指标描述
key required 使用该键名对应的值作为指标的值
buckets optional buckets of record for instrumentation
optional 额外添加的标签

filter 配置示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

<filter message>
  @type prometheus
  <metric>
    name message_foo_counter
    type counter
    desc The total number of foo in message.
    key foo
  </metric>
</filter>

<match message>
  @type stdout
</match>

output 配置示例

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

<match message>
  @type copy
  <store>
    @type prometheus
    <metric>
      name message_foo_counter
      type counter
      desc The total number of foo in message.
      key foo
    </metric>
  </store>
  <store>
    @type stdout
  </store>
</match>

配置面板

grafana 面板配置如下，支持多个数据源

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 34,
  "iteration": 1583483879006,
  "links": [],
  "panels": [
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$datasource",
      "description": "使用 tail file 的方式采集 kubernetes 容器日志，通过 file position 计算每秒的采集速率。",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 2,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "hideZero": false,
        "max": true,
        "min": false,
        "rightSide": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "sum(irate(fluentd_tail_file_position{service=\"$service\", pod=\"$pod\"}[5m]))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "tail file position per second",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Fluentd container log input bytes/s",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "Bps",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$datasource",
      "description": "容器日志输出到 kafka 每秒的条数。",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 12,
        "x": 12,
        "y": 0
      },
      "id": 3,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "hideZero": false,
        "max": true,
        "min": false,
        "rightSide": false,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "irate(fluentd_output_status_emit_records{service=\"$service\", pod=\"$pod\", plugin_id=\"kafka-logstash\"}[1m])",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{plugin_id}} ",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Fluentd container log output rate",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "short",
          "label": "output record rate",
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$datasource",
      "description": "Current used output buffer size.",
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 6
      },
      "id": 6,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "hideZero": false,
        "max": true,
        "min": false,
        "rightSide": true,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "fluentd_output_status_buffer_total_bytes{service=\"$service\", pod=\"$pod\", type!~\"detect_exceptions|null\"}",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{plugin_id}} ",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Fluentd output buffer size",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "bytes",
          "label": "current buffer size",
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$datasource",
      "description": "输出到 kafka 重试和错误的速率。",
      "fill": 1,
      "gridPos": {
        "h": 6,
        "w": 24,
        "x": 0,
        "y": 13
      },
      "id": 5,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "hideZero": false,
        "max": true,
        "min": false,
        "rightSide": true,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "irate(fluentd_output_status_retry_count{service=\"$service\", pod=\"$pod\", type!~\"detect_exceptions|null\"}[1m])",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{plugin_id}} -retry",
          "refId": "A"
        },
        {
          "expr": "irate(fluentd_output_status_num_errors{service=\"$service\", pod=\"$pod\", type!~\"detect_exceptions|null\"}[1m])",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{plugin_id}} -error",
          "refId": "B"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Fluentd output retry/error rate",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "short",
          "label": "retry/error rate",
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "$datasource",
      "description": "输出到 kafka 每秒的条数。",
      "fill": 1,
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 19
      },
      "id": 4,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "hideZero": false,
        "max": true,
        "min": false,
        "rightSide": true,
        "show": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "options": {},
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "irate(fluentd_output_status_emit_records{service=\"$service\", pod=\"$pod\", type!~\"detect_exceptions|null\"}[1m])",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "{{plugin_id}} ",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Fluentd output rate",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "decimals": null,
          "format": "short",
          "label": "output record rate",
          "logBase": 1,
          "max": null,
          "min": "0",
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": false
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "refresh": "5s",
  "schemaVersion": 18,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "tags": [],
          "text": "prometheus",
          "value": "prometheus"
        },
        "hide": 0,
        "includeAll": false,
        "label": null,
        "multi": false,
        "name": "datasource",
        "options": [],
        "query": "prometheus",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      },
      {
        "allValue": null,
        "current": {
          "tags": [],
          "text": "log-fluentd-elasticsearch",
          "value": "log-fluentd-elasticsearch"
        },
        "datasource": "$datasource",
        "definition": "label_values(fluentd_output_status_write_count, service)",
        "hide": 0,
        "includeAll": false,
        "label": "服务:",
        "multi": false,
        "name": "service",
        "options": [],
        "query": "label_values(fluentd_output_status_write_count, service)",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      },
      {
        "allValue": null,
        "current": {
          "text": "log-fluentd-elasticsearch-2jt96",
          "value": "log-fluentd-elasticsearch-2jt96"
        },
        "datasource": "$datasource",
        "definition": "label_values(fluentd_output_status_write_count{service=\"$service\"}, pod)",
        "hide": 0,
        "includeAll": false,
        "label": "实例:",
        "multi": false,
        "name": "pod",
        "options": [],
        "query": "label_values(fluentd_output_status_write_count{service=\"$service\"}, pod)",
        "refresh": 2,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-30m",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "browser",
  "title": "Fluentd 监控",
  "uid": "fluentd",
  "version": 9
}
更新于 2020-03-22
 fluentd
返回 | 主页