go-stash 是一个高效的从 Kafka 获取，根据配置的规则进行处理，然后发送到 ElasticSearch 集群的工具。
go-stash 有大概 logstash 5 倍的吞吐性能，并且部署简单，一个可执行文件即可。
安装
cd stash && go build stash.go
Quick Start
- 可执行文件方式
./stash -f etc/config.yaml
- docker 方式，确保配置文件路径正确
docker run -d -v `pwd`/etc:/app/etc kevinwan/go-stash
config.yaml示例如下:
Clusters: - Input: Kafka: Name: go-stash Log: Mode: file Brokers: - "172.16.48.41:9092" - "172.16.48.42:9092" - "172.16.48.43:9092" Topic: ngapplog Group: stash Conns: 3 Consumers: 10 Processors: 60 MinBytes: 1048576 MaxBytes: 10485760 Offset: first Filters: - Action: drop Conditions: - Key: status Value: 503 Type: contains - Key: type Value: "app" Type: match Op: and - Action: remove_field Fields: - message - source - beat - fields - input_type - offset - "@version" - _score - _type - clientip - http_host - request_time Output: ElasticSearch: Hosts: - "http://172.16.188.73:9200" - "http://172.16.188.74:9200" - "http://172.16.188.75:9200" Index: "go-stash-{{yyyy.MM.dd}}" MaxChunkBytes: 5242880 GracePeriod: 10s Compress: false TimeZone: UTC
ES性能写入测试
测试环境
- stash服务器：3台 4核 8G
- es服务器： 15台 16核 64G
关键配置
- Input: Conns: 3 Consumers: 10 Processors: 60 MinBytes: 1048576 MaxBytes: 10485760 Filters: - Action: remove_field Fields: - message - source - beat - fields - input_type - offset - request_time Output: Index: "nginx_pro-{{yyyy.MM.d}}" Compress: false MaxChunkBytes: 5242880 TimeZone: UTC
写入速度平均在15W/S以上
评论