Add exponential histogram support to CloudWatch PMD Exporter #1677

dricross · 2025-05-08T20:55:23Z

Description of the issue

The Cloudwatch/PMD exporter currently drops all exponential histogram metrics.

Description of changes

Adds support for exponential histogram to the CloudWatch/PMD exporter

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Setup something to what our OTLP -> AMP test does (see test repo).

Agent config:

{
  "agent": {
    "metrics_collection_interval": 15,
    "run_as_user": "root",
    "debug": true
  },
  "metrics": {
    "metrics_collected": {
      "otlp": {
        "http_endpoint": "127.0.0.1:1234"
      }
    }
  }
}

Use OTLP metrics and OTLP pusher from test repo to generate exponential histogram metrics.

In CloudWatch, we can see:

min: 0
max: 5
sample count: 72
Sum: 240

Requirements

Before commit the code, please do the following steps.

Run make fmt and make fmt-sh
Run make lint

jefchien · 2025-05-15T21:33:00Z

metric/distribution/exph/exph.go

+
+// ValuesAndCounts outputs two arrays representing the midpoints of each exponential histogram bucket and the
+// counter of datapoints within the corresponding exponential histogram buckets
+func (d *ExpHistogramDistribution) ValuesAndCounts() ([]float64, []float64) {


Does the order of the values matter? Does it have to be positive to negative?

I don't believe so. See PutMetricData API documentation here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_PutMetricData.html. The order of values just needs to match the order of counts.

I selected this order based on the behavior of the awsemfexporter for exponential histograms, e.g. https://github.com/amazon-contributing/opentelemetry-collector-contrib/blame/aws-cwa-dev/exporter/awsemfexporter/datapoint_test.go#L1168

metric/distribution/exph/exph.go

jefchien · 2025-05-15T21:39:10Z

metric/distribution/exph/exph.go

+	posOffsetIndicies := make([]int, 0, len(d.positiveBuckets))
+	for offsetIndex := range d.positiveBuckets {
+		posOffsetIndicies = append(posOffsetIndicies, offsetIndex)
+	}
+	slices.Sort(posOffsetIndicies)
+	slices.Reverse(posOffsetIndicies)


Instead of sorting and then reversing, we can use maps.Keys which returns an iterator and a sort function to reverse it while doing the sort.

https://pkg.go.dev/maps#Keys
https://pkg.go.dev/slices#SortedFunc

Suggested change

posOffsetIndicies := make([]int, 0, len(d.positiveBuckets))

for offsetIndex := range d.positiveBuckets {

posOffsetIndicies = append(posOffsetIndicies, offsetIndex)

}

slices.Sort(posOffsetIndicies)

slices.Reverse(posOffsetIndicies)

posOffsetIndicies := slices.SortedFunc(maps.Keys(d.positiveBuckets), func(a, b int) int {

return cmp.Compare(b, a)

})

very nice, I'll update

jefchien · 2025-05-15T21:45:18Z

metric/distribution/exph/exph.go

+// LowerBoundaryNegativeScale computes the lower boundary for index
+// with scales <= 0.
+func LowerBoundaryNegativeScale(index int, scale int) float64 {
+	return math.Ldexp(1, index<<-scale)


Why are we using math.Ldexp instead of just math.Exp2 if frac is always 1?

https://pkg.go.dev/math#Ldexp

func Ldexp(frac float64, exp int) float64

Ldexp is the inverse of Frexp. It returns frac × 2**exp.

As you mentioned later, it was pulled directly from the OTLP documentation: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#negative-scale-extract-and-shift-the-exponent

jefchien · 2025-05-15T21:49:25Z

metric/distribution/exph/exph.go

+}
+
+func (d *ExpHistogramDistribution) Resize(_ int) []*ExpHistogramDistribution {
+	// for now, do not split data points into separate PMD requests


When we say "for now" does that mean this should be a TODO?

It's something that needs to be done later, yes. Does marking it as TODO specifically do something special?

jefchien · 2025-05-15T21:54:51Z

metric/distribution/exph/exph.go

+}
+
+// MapToIndexScale0 computes a bucket index at scale 0.
+func MapToIndexScale0(value float64) int {


It looks like these were all taken from https://opentelemetry.io/docs/specs/otel/metrics/data-model/#negative-scale-extract-and-shift-the-exponent. Consider putting in a separate file in the package (like mapping.go or math.go) and indicating the source at the top.

Sure, I can move those

not quite complete. need more unit tests

* Move OTLP implementation to separate file * Simplify map key sorting

jefchien · 2025-05-15T22:01:30Z

metric/distribution/exph/exph.go

+	return d.AddEntryWithUnit(value, weight, "")
+}
+
+func (d *ExpHistogramDistribution) AddDistribution(other *ExpHistogramDistribution) {


Are we trying to match the Distribution interface? It doesn't look like this would satisfy the interface as it is now. We would have to make the Distribution interface generic.

jefchien · 2025-05-20T17:54:42Z

plugins/outputs/cloudwatch/convert_otel.go

@@ -82,7 +83,7 @@ func ConvertOtelNumberDataPoints(
 	unit string,
 	scale float64,
 	entity cloudwatch.Entity,
-) []*aggregationDatum {
+) []*aggregationDatum { //nolint:revive


nit: Instead of ignoring the lint, can we just not export the function? It doesn't look like it's used outside of this package.

jefchien · 2025-05-20T18:00:05Z

plugins/outputs/cloudwatch/convert_otel.go

+		}
+		// Assume function pointer is valid.
+		ad.expHistDistribution = exph.NewExpHistogramDistribution()
+		ad.expHistDistribution.ConvertFromOtel(dp, unit)


nit: If we store the unit in the MetricDatum already, why do we need to store it in the distribution?

jefchien · 2025-05-20T18:01:08Z

plugins/outputs/cloudwatch/cloudwatch.go

+	return datums
+}
+
+func (c *CloudWatch) buildMetricDataumExph(metric *aggregationDatum, dimensionsList [][]*cloudwatch.Dimension) []*cloudwatch.MetricDatum {


nit: Typo

Suggested change

func (c *CloudWatch) buildMetricDataumExph(metric *aggregationDatum, dimensionsList [][]*cloudwatch.Dimension) []*cloudwatch.MetricDatum {

func (c *CloudWatch) buildMetricDatumExph(metric *aggregationDatum, dimensionsList [][]*cloudwatch.Dimension) []*cloudwatch.MetricDatum {

github-actions · 2025-05-28T00:13:31Z

This PR was marked stale due to lack of activity.

dricross requested a review from a team as a code owner May 8, 2025 20:55

jefchien reviewed May 15, 2025

View reviewed changes

dricross force-pushed the dricross/exponentialhistogram branch from 49857cf to 97c4271 Compare May 16, 2025 21:08

dricross added 7 commits May 16, 2025 17:08

A dumb commit

2144fa8

adding support for exphistograms

9c5566b

not quite complete. need more unit tests

Do not split exph metrics into separate PMD requests

499444e

fixup tests

bda6c16

linting fixup. remove dead code

e247b86

Fix typo in test

8621351

PR feedback

995902e

* Move OTLP implementation to separate file * Simplify map key sorting

dricross force-pushed the dricross/exponentialhistogram branch from 97c4271 to 995902e Compare May 16, 2025 21:08

jefchien reviewed May 20, 2025

View reviewed changes

github-actions bot added the Stale label May 28, 2025

	func (c CloudWatch) buildMetricDataumExph(metric aggregationDatum, dimensionsList [][]cloudwatch.Dimension) []cloudwatch.MetricDatum {
	func (c CloudWatch) buildMetricDatumExph(metric aggregationDatum, dimensionsList [][]cloudwatch.Dimension) []cloudwatch.MetricDatum {

Add exponential histogram support to CloudWatch PMD Exporter #1677

Are you sure you want to change the base?

Add exponential histogram support to CloudWatch PMD Exporter #1677

Uh oh!

Conversation

dricross commented May 8, 2025

Description of the issue

Description of changes

License

Tests

Requirements

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 28, 2025

Uh oh!

Uh oh!