Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using CreateOrUpdateStream omits domain prefix in SourceConfig when updating #1744

Open
L154x opened this issue Dec 2, 2024 · 1 comment
Labels
defect Suspected defect such as a bug or regression

Comments

@L154x
Copy link

L154x commented Dec 2, 2024

Observed behavior

Here's what I'm executing

	s, err := js.CreateOrUpdateStream(c.ctx, jetstream.StreamConfig{
		Name:        c.streamName,
		Description: "JetStream for all telemetry subjects",
		Subjects:    []string{c.subject},
		Retention:   jetstream.LimitsPolicy,
		Sources:     generateSources(5),
		MaxAge:      24 * time.Hour, // Discard data after 24 hours from the stream
		Discard:     jetstream.DiscardOld,
		Storage:     jetstream.FileStorage,
		Replicas:    1,
	})

func generateSources(count int) []*jetstream.StreamSource {
	sources := make([]*jetstream.StreamSource, count)
	for i := 0; i < count; i++ {
		sourceName := fmt.Sprintf("leaf%d", i+1)
		sources[i] = &jetstream.StreamSource{
			Name:   strings.ToUpper(sourceName),
			Domain: sourceName, // domain name must be individual per leaf node
			SubjectTransforms: []jetstream.SubjectTransformConfig{
				{
					// Aggregate all local subjects into the telemetry namespace
					Source:      fmt.Sprintf("%s.telemetry.>", sourceName),
					Destination: "telemetry.>",
				},
			},
		}
	}
	return sources
}

This code works fine when creating the stream. But when updating the stream (even without changes), the domain prefix is removed.

We traced the communication with Jetstream. We see it's first sending an update without domain prefix and then a CREATE with domain prefix.

If the update succeeds, the prefix will be gone.

[#1] Received on "$JS.API.STREAM.UPDATE.TELEMETRY" with reply "_INBOX.telemetry.qoRiMyK8YZFajxgyQ19bvS.DtvQoHpE"
{"name":"TELEMETRY","description":"JetStream for all telemetry subjects","subjects":["telemetry.\u003e"],"retention":"limits","max_consumers":0,"max_msgs":0,"max_bytes":0,"discard":"old","max_age":86400000000000,"max_msgs_per_subject":0,"storage":"file","num_replicas":1,"sources":[{"name":"LEAF1","subject_transforms":[{"src":"leaf1.telemetry.\u003e","dest":"telemetry.\u003e"}]},{"name":"LEAF2","subject_transforms":[{"src":"leaf2.telemetry.\u003e","dest":"telemetry.\u003e"}]},{"name":"LEAF3","subject_transforms":[{"src":"leaf3.telemetry.\u003e","dest":"telemetry.\u003e"}]},{"name":"LEAF4","subject_transforms":[{"src":"leaf4.telemetry.\u003e","dest":"telemetry.\u003e"}]},{"name":"LEAF5","subject_transforms":[{"src":"leaf5.telemetry.\u003e","dest":"telemetry.\u003e"}]}],"compression":"none","allow_direct":false,"mirror_direct":false,"consumer_limits":{}}

[#2] Received on "$JS.API.STREAM.CREATE.TELEMETRY" with reply "_INBOX.telemetry.qoRiMyK8YZFajxgyQ19bvS.WEt8hInf"
{"name":"TELEMETRY","description":"JetStream for all telemetry subjects","subjects":["telemetry.\u003e"],"retention":"limits","max_consumers":0,"max_msgs":0,"max_bytes":0,"discard":"old","max_age":86400000000000,"max_msgs_per_subject":0,"storage":"file","num_replicas":1,"sources":[{"name":"LEAF1","subject_transforms":[{"src":"leaf1.telemetry.\u003e","dest":"telemetry.\u003e"}],"external":{"api":"$JS.leaf1.API","deliver":""}},{"name":"LEAF2","subject_transforms":[{"src":"leaf2.telemetry.\u003e","dest":"telemetry.\u003e"}],"external":{"api":"$JS.leaf2.API","deliver":""}},{"name":"LEAF3","subject_transforms":[{"src":"leaf3.telemetry.\u003e","dest":"telemetry.\u003e"}],"external":{"api":"$JS.leaf3.API","deliver":""}},{"name":"LEAF4","subject_transforms":[{"src":"leaf4.telemetry.\u003e","dest":"telemetry.\u003e"}],"external":{"api":"$JS.leaf4.API","deliver":""}},{"name":"LEAF5","subject_transforms":[{"src":"leaf5.telemetry.\u003e","dest":"telemetry.\u003e"}],"external":{"api":"$JS.leaf5.API","deliver":""}}],"compression":"none","allow_direct":false,"mirror_direct":false,"consumer_limits":{}}

Expected behavior

The Stream should be the same after it has been created.
There should not be any differences in the create or update command, they should be identical.

Server and client version

github.com/nats-io/nats.go v1.37.0

Host environment

No response

Steps to reproduce

No response

@L154x L154x added the defect Suspected defect such as a bug or regression label Dec 2, 2024
@roeschter
Copy link

We noticed this behavior during a joint demo workshop.
I confirmed by tracing through sub $JS.API that the Golang API- CreateOrUpdateStream is rendering the UPDATE incorrectly

  1. First send an update WITHOUT the external section. - which receives an error if the stream does not exist
  2. THEN send the create WITH the external section.

The second time running the code the update with the missing external section will take effect and break the sourcing agreement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
defect Suspected defect such as a bug or regression
Projects
None yet
Development

No branches or pull requests

2 participants