日志采集工具 Vladimir

Apache
跨平台
2019-08-05
zmatsh

Vladimir 采集系统

日志采集工具,支持正则 热加载日志文件,发送状态等信息都会实时上报 ETCD

TODO

  • 将日志文件一行一行读取出来发送到自定义的 Sink | 100%
  • 支持自动重试(fib)
  • 心跳检测发送到自定义的 Topic[health] | 100%
  • 日志启动点(创建 CheckPoint) | 100%
  • Offset 同步到 Etcd | 100%
  • Web 监控心跳以及创建新的采集任务 | 100%
  • 热启动插件 | 10%

Logs - 采集工具


./Logs_Pulse  install   -- 安装服务
./Logs_Pulse  start     -- 启动服务
./Logs_Pulse  stop      -- 停止服务
./Logs_Pulse  restart   -- 重启服务

Owner

  • 邮箱echo emhhbmdqaWFueGlubmV0QGdtYWlsLmNvbQo= | base64 --decode

命令行客户端使用教程


(base) ➜  cli ./cli         
NAME:
   pulse client - a tool for driving pulse cli

USAGE:
   cli [global options] command [command options] [arguments...]

VERSION:
   v1 darwin/amd64 go1.11.4

AUTHOR:
   Zhangjianxin <zhangjianxinnet@gmail.com>

COMMANDS:
     list      pulse on etcd config
     services  pulse on services
     task      addtask on services
     clear     delete root path / data
     host      host list
     help, h   Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version



(base) ➜  cli ./cli host
|---------------|
| Host          |
|---------------|
| 192.168.2.150 |
|---------------|
| 192.168.2.150 |
|---------------|
| 192.168.2.150 |
|---------------|
| 192.168.2.150 |
|---------------|


(base) ➜  cli ./cli list
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| Topic | Log_path                                                                          | Service             | Send_rate | IP            | Type  |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| test  | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | sentry-test-service | 1000      | 192.168.2.150 | kafka |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| test  | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | sentry-test-service | 1000      | 192.168.2.150 | http  |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|
| test  | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est2.log | sentry-test-service | 1000      | 192.168.2.150 | http  |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|


(base) ➜  cli ./cli services
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| Offset | FilePatch                                                                         | Time                | Topic | TPS | QPS | IP            |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| 39     | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | 2019-04-27 03:38:32 | test  | 0   | 0   | 192.168.2.150 |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| 63     | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est.log  | 2019-04-27 03:45:17 | test  | 0   | 0   | 192.168.2.150 |
|------------------------------------------------------------------------------------------------------------------------------------------------------|
| 87     | /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/est2.log | 2019-04-27 03:49:32 | test  | 0   | 0   | 192.168.2.150 |
|------------------------------------------------------------------------------------------------------------------------------------------------------|




(base) ➜  cli 
    ./cli task  -host 192.168.43.223  -t buried-point-mall -f /var/log/system.log -s buried-point-mall-service-test -p http
    ./cli task   -re "" -host 192.168.2.150  -t buried-point-mall -f /Users/zhangjianxin/home/GO_LIB/src/github.com/uk0/Octopoda/release/logs/ -s buried-point-mall-service-test -p http
    
    ./cli task   -re "" -host 172.17.193.23  -t sentry-test -f /var/log/messages -s service-test -p http
    
    ./cli task  -re "" -host  172.19.184.193 -t sentry-test -f /home/soft/Octopoda/release/logs/pulse.log -s system-pulse -p http
    ./cli task  -re "" -host  172.19.184.193 -t buried-point-mall -f /data/sa/ -s sa-pulse -p http
    
    
    ./cli task  -re "" -host 172.17.193.23  -t sentry-test -f /home/Octopoda/release/logs/ -s service-test2 -p http
    ./cli task  -re "" -host 192.168.131.249  -t buried-point-sma-r2p10 -f  /home/admin/app/logs/ -s service-test4 -p http
    ./cli task  -re "" -host 192.168.131.249  -t buried-point-sma-r2p10 -f  /home/admin/app/logs/behaviorSave.log -s service-test4 -p http
    
    
    
    ./cli task  -re "" -host 192.168.131.249  -t sentry-test -f  /home/Octopoda/release/logs/pulse.log -s system-pulse -p http

提交作业


./cli task 
         -host 192.168.2.150  #采集某个节点
         -t test  #发送到topic
         -f /var/log/system.log # 采集日志
         -s sentry-test-service # 服务名字定义
         -p http #发送方式


# create topic
kafka-topics --create --zookeeper cdh-m1.sxkj.online:2181 --replication-factor 1 --partitions 1 --topic kdc_pdp

kerberos KDC Proxy

https://medium.com/@Santii/how-to-setup-kerberos-behind-a-reverse-proxy-nginx-d692f609191c
的码云指数为
超过 的项目
加载中

评论(0)

暂无评论

暂无资讯

暂无问答

V-Ray渲染时代

在众多新技术的冲击下,作为使用范围最广泛的渲染器的V-Ray也不甘落后,在Siggraph 2013上Chaos Group似乎给我们带来了V-Ray 3.0的重大更新。因此有业内人士猜想,V-Ray渲染时代即将随着这次...

2013/08/22 10:17
26
0
Russia Proposes First Multinational Cryptocurrency

Expert Blog is Cointelegraph’s new series of articles by crypto industry leaders. It covers everything from Blockchain technology and cryptocurrencies to ICO regulation and inv...

2018/01/15 13:44
9
0
bucket4j使用实例

序 本文主要演示一下bucket4j的几个使用实例 maven <dependency> <groupId>com.github.vladimir-bukhtoyarov</groupId> <artifactId>bucket4j-core</artifactId> <version>4.0.1</version> ...

2018/09/01 23:26
88
0
WebGL样品与演示

来源:http://blog.vlad1.com/2009/09/21/webgl-samples/ 下载9月18日以后Firefox的Nightly版,在地址栏输入about:config,搜索“webgl”,双击“webgl.enabled_for_all_sites”,将其值从“...

2016/07/04 20:21
22
0
正确地猜测用户的意图

正确地猜测用户的意图

2015/06/03 19:13
382
0
Lucene的FuzzyQuery中用到的Levenshtein Distance(LD)算法

主题:Levenshtein Distance(LD); 相关介绍:Levenshtein distance是由俄国科学家Vladimir Levenshtein在1965年设计并以他的名字命名的。如果不能拼写或发Levenshtein音,通常可以称它edit di...

2014/10/31 14:13
133
0
文学 2014年2月23日 20:43:43

文学 2014年2月23日 20:43:43

2014/03/23 21:22
8
0
聊聊token bucket算法的实现

序 本文主要研究一下token bucket算法的实现 限流算法概述 主要有如下几种: 基于信号量Semaphore 只有数量维度,没有时间维度 基于fixed window 带上了时间维度,不过在两个窗口的临界点容易...

2018/08/28 16:23
204
0

没有更多内容

加载失败,请刷新页面

返回顶部
顶部