bds 正在参加 2021 年度 OSC 中国开源项目评选,请投票支持!
bds 在 2021 年度 OSC 中国开源项目评选 中已获得 {{ projectVoteCount }} 票,请投票支持!
2021 年度 OSC 中国开源项目评选 正在火热进行中,快来投票支持你喜欢的开源项目!
2021 年度 OSC 中国开源项目评选 >>> 中场回顾
bds 获得 2021 年度 OSC 中国开源项目评选「最佳人气项目」 !
授权协议 Apache-2.0 License
开发语言 Java
操作系统 跨平台
软件类型 开源软件
所属分类 云计算云原生
开源组织
地区 不详
投 递 者 首席测试
适用人群 未知
收录时间 2021-12-09

软件简介

BDS

logo

Introduction

JD Cloud Blockchain Data Service (BDS) is a realtime data aggregating, analyzing and visualization service for chain-like unstructured data from all kinds of 3rd party Blockchains

Splitter is the key module of Blockchain Data Service (BDS) and provides data analysis capability.

Splitter is responsible for consuming blockchain data from message queue (kafka) and inserting data into persistent data storage services (relational database, data warehouse, etc.) for further processing

Architecture

Architecture

Environment Deployment

Install BDS

Environment initialization

Before compiling and running BDS, you must install go's compilation environment locally: go install

Install Splitter steps

  1. Set the path of project : $GOPATH/src/github.com/jdcloud-bds/bds/
  2. Inputgo build -v github.com/jdcloud-bds/bds/cmd/bds-splitter,compile to get executable file bds-splitter
  3. Build new configuration file splitter.conf, see /config/splitter_example.conf configuration file template
  4. Run program ./bds-splitter -c splitter.conf

Install confluent and kafka

Install kafka

See kafka

Modify config/server.properties
  • message.max.bytes=1048576000

Install confluent

see confluent

Unzip the confluent package and run Confluent REST Proxy

Modify /etc/kafka-rest/kafka-rest.properties
  • max.request.size = 1048576000
  • buffer.memory = 1048576000
  • send.buffer.bytes = 1048576000

Database

Database we now support SQL Server, PostgreSQL, you can choose one as a data storage method.

SQL Server

Buy JCS For SQL Server

PostgreSQL

Buy JCS For PostgreSQL

After you run the database, you need to manually create new database and use the database name initialization splitter.conf.

Install Grafana

See Grafana Official

Source code

Splitter Modules

Development Steps

  1. Define the data structure of Kafka messages.
  2. Define table structure.
  3. Analyze Kafka message and store data in database.

Contributing

Contributing guide

License

Apache License 2.0

Project Demonstration

Blockchain Data Service

展开阅读全文

代码

评论

点击引领话题📣
暂无内容
发表了博客
{{o.pubDate | formatDate}}

{{formatAllHtml(o.title)}}

{{parseInt(o.replyCount) | bigNumberTransform}}
{{parseInt(o.viewCount) | bigNumberTransform}}
没有更多内容
暂无内容
发表了问答
{{o.pubDate | formatDate}}

{{formatAllHtml(o.title)}}

{{parseInt(o.replyCount) | bigNumberTransform}}
{{parseInt(o.viewCount) | bigNumberTransform}}
没有更多内容
暂无内容
0 评论
0 收藏
分享
OSCHINA
登录后可查看更多优质内容
返回顶部
顶部
返回顶部
顶部