Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update gradle and remove support for Kafka v0.8. #33

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@
/out
build
/.gradle
/.idea
2 changes: 1 addition & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ env:
- TERM=dumb

jdk:
- oraclejdk8
- openjdk8

script:
- ./gradlew check --info
16 changes: 12 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,23 @@
# 0.5.2
# 0.6.0
- Remove support for Kafka v0.8.

# 0.5.2
- Convert INFINITY values to 0.

# 0.4.0
# 0.5.1
- Fix metrics change log level

# 0.5.0
- `0.5.0` add support to report new producer/consumer metrics in kafka-0.9
- Compatible with Kafka 0.8
- A complete list of all the metrics supported in the metrics reporter can be found [here](http://docs.confluent.io/2.0.1/kafka/monitoring.html)

# 0.4.0
- `0.4.0` adds support for tags on metrics. See [dogstatsd extensions](http://docs.datadoghq.com/guides/dogstatsd/#tags). If your statsd server does not support tags, you can disable them in the Kafka configuration. See property `external.kafka.statsd.tag.enabled` below.

- The statsd client is [`com.indeed:java-dogstatsd-client:2.0.11`](https://github.com/indeedeng/java-dogstatsd-client/tree/java-dogstatsd-client-2.0.11).
- support new `MetricNames` introduced by kafka `0.8.2.x`
- remove JVM metrics. Only the metrics from Kafka `MetricRegistry` are sent.

# 0.3.0
- send JVM metrics

- send JVM metrics
49 changes: 3 additions & 46 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,39 +24,15 @@ Metrics can be filtered based on the metric name and the metric dimensions (min,


## Releases

### 0.5.2
- Convert INFINITY values to 0.

### 0.5.1
- Fix metrics change log level

### 0.5.0

- `0.5.0` add support to report new producer/consumer metrics in kafka-0.9
- Compatible with Kafka 0.8
- A complete list of all the metrics supported in the metrics reporter can be found [here](http://docs.confluent.io/2.0.1/kafka/monitoring.html)


### 0.4.0

- `0.4.0` adds support for tags on metrics. See [dogstatsd extensions](http://docs.datadoghq.com/guides/dogstatsd/#tags). If your statsd server does not support tags, you can disable them in the Kafka configuration. See property `external.kafka.statsd.tag.enabled` below.

- The statsd client is [`com.indeed:java-dogstatsd-client:2.0.11`](https://github.com/indeedeng/java-dogstatsd-client/tree/java-dogstatsd-client-2.0.11).
- support new `MetricNames` introduced by kafka 0.8.2.x

### 0.3.0

- initial release
See the [changelog](CHANGELOG.md).

## How to install?

- [Download](https://bintray.com/airbnb/jars/kafka-statsd-metrics2/view) or build the shadow jar for `kafka-statsd-metrics`.
- Install the jar in Kafka classpath, typically `./kafka_2.11-0.9.0.1/libs/`
- In the Kafka config file, `server.properties`, add the following properties. Default values are in parenthesis.

## How to use metrics in Kafka 0.9 / 0.8?
### New metrics in kafka 0.9
## How to use metrics in Kafka 0.9?

1. Add `metric.reporters` in producer.properties or consumer.properties
```bash
Expand All @@ -71,30 +47,11 @@ Producer:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --new-consumer --zookeeper localhost:2181 --topic test -from-beginning --consumer.config config/consumer.properties
```

### Old metrics in kafka 0.8

1. Add `kafka.metrics.reporters` in producer.properties or consumer.properties
```bash
# declare the reporter if old producer/consumer is used
kafka.metrics.reporters=com.airbnb.kafka.kafka08.StatsdMetricsReporter
```
2. Run old-producer or old-consumer

```bash
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test --producer.config config/producer.properties --old-producer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --zookeeper localhost:2181 --topic test -from-beginning --consumer.config config/consumer.properties
```

2. Run old-consumer


### Configurations
```bash
# declare the reporter if new producer/consumer is used
metric.reporters=com.airbnb.kafka.StatsdMetricsReporter

# declare the reporter if old producer/consumer is used
kafka.metrics.reporters=com.airbnb.kafka.kafka08.StatsdMetricsReporter
metric.reporters=com.airbnb.kafka.kafka09.StatsdMetricsReporter

# enable the reporter, (false)
external.kafka.statsd.reporter.enabled=true
Expand Down
31 changes: 10 additions & 21 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ buildscript {
jcenter()
}
dependencies {
classpath 'com.github.jengelman.gradle.plugins:shadow:0.9.0-M3'
classpath 'com.github.jengelman.gradle.plugins:shadow:5.1.0'
}
}

Expand All @@ -14,7 +14,7 @@ apply plugin: 'idea'
apply plugin: 'com.github.johnrengelman.shadow'

group = 'com.airbnb'
description = 'StatsD Support for Kafka Metrics (kafka version: 0.8 and 0.9)'
description = 'StatsD Support for Kafka Metrics (kafka version: 0.9)'

repositories {
mavenCentral()
Expand All @@ -23,7 +23,8 @@ repositories {

dependencies {
compile 'com.indeed:java-dogstatsd-client:2.0.11'
compile 'org.apache.kafka:kafka_2.11:0.9.0.1'
compile 'org.apache.kafka:kafka-clients:0.9.0.1'
compile 'com.yammer.metrics:metrics-core:2.2.0'
compile 'org.slf4j:slf4j-log4j12:+'

testCompile('junit:junit:4.11',
Expand All @@ -36,23 +37,11 @@ compileJava {
options.encoding = 'UTF-8'
}

configurations {
//manually excludes some unnecessary dependencies
compile.exclude module: 'zookeeper'
compile.exclude module: 'zkclient'
compile.exclude module: 'javax'
compile.exclude module: 'jline'
compile.exclude module: 'jms'
compile.exclude module: 'jmxri'
compile.exclude module: 'jmxtools'
compile.exclude module: 'mail'
}

shadowJar {
exclude 'META-INF/*.DSA'
exclude 'META-INF/*.RSA'
exclude 'META-INF/maven/*'
exclude dependency('org.apache.kafka:kafka_2.10:0.9.0.1')
exclude dependency('org.slf4j:slf4j-log4j12')
dependencies {
exclude 'META-INF/maven/*'
exclude dependency('org.slf4j:slf4j-log4j12')
exclude dependency('org.slf4j:slf4j-api')
exclude dependency('log4j:log4j')
}
}

2 changes: 1 addition & 1 deletion gradle.properties
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
version=0.5.2
version=0.6.0

Binary file modified gradle/wrapper/gradle-wrapper.jar
Binary file not shown.
3 changes: 1 addition & 2 deletions gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
#Wed Jul 30 14:46:29 PDT 2014
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-5.6.4-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-1.12-bin.zip
98 changes: 61 additions & 37 deletions gradlew
Original file line number Diff line number Diff line change
@@ -1,25 +1,59 @@
#!/usr/bin/env bash
#!/usr/bin/env sh

#
# Copyright 2015 the original author or authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

##############################################################################
##
## Gradle start up script for UN*X
##
##############################################################################

# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS=""
# Attempt to set APP_HOME
# Resolve links: $0 may be a link
PRG="$0"
# Need this for relative symlinks.
while [ -h "$PRG" ] ; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG=`dirname "$PRG"`"/$link"
fi
done
SAVED="`pwd`"
cd "`dirname \"$PRG\"`/" >/dev/null
APP_HOME="`pwd -P`"
cd "$SAVED" >/dev/null

APP_NAME="Gradle"
APP_BASE_NAME=`basename "$0"`

# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'

# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD="maximum"

warn ( ) {
warn () {
echo "$*"
}

die ( ) {
die () {
echo
echo "$*"
echo
Expand All @@ -30,6 +64,7 @@ die ( ) {
cygwin=false
msys=false
darwin=false
nonstop=false
case "`uname`" in
CYGWIN* )
cygwin=true
Expand All @@ -40,31 +75,11 @@ case "`uname`" in
MINGW* )
msys=true
;;
NONSTOP* )
nonstop=true
;;
esac

# For Cygwin, ensure paths are in UNIX format before anything is touched.
if $cygwin ; then
[ -n "$JAVA_HOME" ] && JAVA_HOME=`cygpath --unix "$JAVA_HOME"`
fi

# Attempt to set APP_HOME
# Resolve links: $0 may be a link
PRG="$0"
# Need this for relative symlinks.
while [ -h "$PRG" ] ; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG=`dirname "$PRG"`"/$link"
fi
done
SAVED="`pwd`"
cd "`dirname \"$PRG\"`/" >&-
APP_HOME="`pwd -P`"
cd "$SAVED" >&-

CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar

# Determine the Java command to use to start the JVM.
Expand All @@ -90,7 +105,7 @@ location of your Java installation."
fi

# Increase the maximum file descriptors if we can.
if [ "$cygwin" = "false" -a "$darwin" = "false" ] ; then
if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then
MAX_FD_LIMIT=`ulimit -H -n`
if [ $? -eq 0 ] ; then
if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then
Expand All @@ -110,10 +125,11 @@ if $darwin; then
GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\""
fi

# For Cygwin, switch paths to Windows format before running java
if $cygwin ; then
# For Cygwin or MSYS, switch paths to Windows format before running java
if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then
APP_HOME=`cygpath --path --mixed "$APP_HOME"`
CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
JAVACMD=`cygpath --unix "$JAVACMD"`

# We build the pattern for arguments to be converted via cygpath
ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`
Expand Down Expand Up @@ -154,11 +170,19 @@ if $cygwin ; then
esac
fi

# Split up the JVM_OPTS And GRADLE_OPTS values into an array, following the shell quoting and substitution rules
function splitJvmOpts() {
JVM_OPTS=("$@")
# Escape application args
save () {
for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
echo " "
}
eval splitJvmOpts $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS
JVM_OPTS[${#JVM_OPTS[*]}]="-Dorg.gradle.appname=$APP_BASE_NAME"
APP_ARGS=$(save "$@")

# Collect all arguments for the java command, following the shell quoting and substitution rules
eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"

# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then
cd "$(dirname "$0")"
fi

exec "$JAVACMD" "${JVM_OPTS[@]}" -classpath "$CLASSPATH" org.gradle.wrapper.GradleWrapperMain "$@"
exec "$JAVACMD" "$@"
Loading