Learning
This project is a companion repository to the Apache Kafka Connect course on Udemy.
https://links.datacumulus.com/kafka-connect-coupon
Kafka Connect Source GitHub
This connector allows you to get a stream of issues and pull requests from your GitHub repository, using the GitHub Api: https://developer.github.com/v3/issues/#list-issues-for-a-repository
Issues are pulled based on updated_at field, meaning any update to an issue or pull request will appear in the stream.
The connector writes to topic that is great candidate to demonstrate log compaction. It's also a fun way to automate your GitHub workflow.
It's finally aimed to be an educative example to demonstrate how to write a Source Connector a little less trivial than the FileStreamSourceConnector provided in Kafka.
Contributing
This connector is not perfect and can be improved, please feel free to submit any PR you deem useful.
Configuration
name=GitHubSourceConnectorDemo
tasks.max=1
connector.class=com.simplesteph.kafka.GitHubSourceConnector
topic=github-issues
github.owner=kubernetes
github.repo=kubernetes
since.timestamp=2017-01-01T00:00:00Z
# I heavily recommend you set those two fields:
auth.username=your_username
auth.password=your_password
Running in development
Note: Java 8 is required for this connector.
Make sure config/worker.properties is configured to wherever your kafka cluster is
./build.sh
./run.sh
The simplest way to run run.sh is to have docker installed. It will pull a Dockerfile and run the connector in standalone mode above it.
Deploying
Note: Java 8 is required for this connector.
TODO
评论