Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to customize statsd port on Spark executors #28

Open
porridge opened this issue Jun 12, 2018 · 0 comments
Open

Add ability to customize statsd port on Spark executors #28

porridge opened this issue Jun 12, 2018 · 0 comments

Comments

@porridge
Copy link

When StatsdMetricsReporter is used in a distributed Apache Spark application, then the kafka configuration properties map is instantiated on one node (called driver), and then propagated to - and used on - multiple other nodes (called executors). See example app.

Moreover, when running on DC/OS, the statsd connection settings (hostname and port) vary between nodes, and are provided to each instance of the application in the process environment.

This means that if the host+port is taken from the environment on the driver, and then propagated and used on executors, metrics will not be exported (at least not correctly). A symptom of this is kafka metrics being produced by the driver but not executors.

One way to solve this problem is to change StatsdMetricsReporter to take $STATSD_UDP_HOST and $STATSD_UDP_PORT from the environment if the external.kafka.statsd.* properties are not present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant