feat: Hive listener integration #605
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
I might have run into stackabletech/hdfs-operator#686 during development. I recognized that I can have an empty string in my discovery configMap from time to time. Behaviour appears to be flaky, but yet more often then not the emtpy string appears: ExpectedFlaky faulty one |
…tackabletech/hive-operator into feat/hive-listener-integration
|
🟢 |
Maleware
moved this from Development: In Progress
to Development: Waiting for Review
in Stackable Engineering
maltesander
moved this from Development: Waiting for Review
to Development: In Review
in Stackable Engineering
maltesander
requested changes
Jun 10, 2025
maltesander
reviewed
Jun 10, 2025
| chroot: Option<&str>, | ||
| listener_refs: BTreeMap<&String, Listener>, |
There was a problem hiding this comment.
We should explicitly mention this in the release notes / breaking changes that the discovery key changed. Or we keep the HIVE key for backwards compatibility?
Trino wont work anymore etc.
There was a problem hiding this comment.
There are pro and cons about it:
Each rolegroup has it's service and thus it's own entry in the configmap.
like
apiVersion: v1
data:
cluster-internal: thrift://hive-postgres-s3-metastore-cluster-internal.default.svc.cluster.local:9083
default: thrift://172.18.0.2:32658
external-unstable: thrift://172.18.0.2:30607
kind: ConfigMap
metadata:
creationTimestamp: "2025-06-04T13:39:09Z"
labels:
app.kubernetes.io/component: metastore
I like it better as you can decide which hive rolegroup you want to talk to.
However I understand the argument being backwards compatible. I could offer to make it upper case. Keeping compatibility would then mean to change the rg from default to hive to have the same picture as before?
I just don't like
apiVersion: v1
data:
HIVE: thrift://hive-postgres-s3-metastore-cluster-internal.default.svc.cluster.local:9083
thrift://172.18.0.2:32658
thrift://172.18.0.2:30607
kind: ConfigMap
metadata:
creationTimestamp: "2025-06-04T13:39:09Z"
labels:
app.kubernetes.io/component: metastore
as it can't be distinguished anymore if every rolegroup uses NodePort
There was a problem hiding this comment.
The problem is that "HIVE" is hardcoded/expected by the trino operator, using different rolegroups will cause problems (or at least not completely utilize the cluster) as its implemented currently. Will hit that up on Slack.
Co-authored-by: Malte Sander <[email protected]>
…tackabletech/hive-operator into feat/hive-listener-integration
Co-authored-by: Malte Sander <[email protected]>
…tackabletech/hive-operator into feat/hive-listener-integration
Comment on lines
+135
to
+140
| listener_ref: Listener, | ||
| rolegroup: &String, | ||
| chroot: Option<&str>, | ||
| ) -> Result<String, Error> { | ||
| // We only need the first address corresponding to the rolegroup | ||
| let listener_address = listener_ref |
There was a problem hiding this comment.
This should probably be called listener, as it isn't a ref.
Suggested change
| listener_ref: Listener, | |
| rolegroup: &String, | |
| chroot: Option<&str>, | |
| ) -> Result<String, Error> { | |
| // We only need the first address corresponding to the rolegroup | |
| let listener_address = listener_ref | |
| listener: Listener, | |
| rolegroup: &String, | |
| chroot: Option<&str>, | |
| ) -> Result<String, Error> { | |
| // We only need the first address corresponding to the rolegroup | |
| let listener_address = listener |
| @@ -448,6 +417,10 @@ pub struct MetaStoreConfig { | |||
| /// Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details. | |||
| #[fragment_attrs(serde(default))] | |||
| pub graceful_shutdown_timeout: Option<Duration>, | |||
|
|
|||
| /// This field controls which [ListenerClass](DOCS_BASE_URL_PLACEHOLDER/listener-operator/listenerclass.html) is used to expose the webserver. | |||
| #[serde(default)] | |||
There was a problem hiding this comment.
All other fields use fragment_attr, is that needed here too?
Suggested change
| #[serde(default)] | |
| #[fragment_attr(serde(default))] |
There was a problem hiding this comment.
Also, do we want the String::default() to be called? Doesn't it have to be a valid class, or it falls back to cluster-internal?
| hive, | ||
| &resolved_product_image.app_version_label, | ||
| &HiveRole::MetaStore.to_string(), | ||
| "discovery", |
There was a problem hiding this comment.
Would this cause a collision if there are multiple role groups defined?
Maleware
moved this from Development: In Review
to Development: In Progress
in Stackable Engineering
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds listener Support
Definition of Done Checklist
Author
Reviewer
Acceptance