Skip to content

Commit f49b5a1

Browse files
donbeavedonbeave
authored
[bitnami/spring-cloud-data-flow] Fixes and improvements (#2945)
* Ignore .DS_Store * Spring Cloud Data Flow chart fixes and improvements: 1) added ability to configure Hibernate dialect 2) fixed issue with locating application.yaml in kubernetes cluster 3) added JDWP support 4) removed MariaDB from database comments, because external database can be PostgreSQL as well 5) fixed some wrong properties (externalDatabase.server -> externalDatabase.dataflow) * Added changes to values-production.yaml. * Bumped spring-cloud-dataflow to 0.1.3 version. * Added hibernateDialect and jdwp variables to README.md. * Fixed "missing starting space in comment" * Fixed "java.sql.SQLInvalidAuthorizationSpecException: Access denied for user 'dataflow'@'10.244.0.5' (using password: YES)". * Bump version to 0.2.0. * Added support for external RabbitMQ. * Added externalRabbitmq config to README. Fixed typo: rabbitmq.rabbitmq -> rabbitmq.auth. * Fixed config location, use /opt/bitnami/**/conf.
1 parent 73fbb1c commit f49b5a1

13 files changed

+229
-32
lines changed

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
11
*.tgz
22
/.idea/*
33
.vscode
4+
.DS_Store

‎bitnami/spring-cloud-dataflow/Chart.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
apiVersion: v1
22
name: spring-cloud-dataflow
3-
version: 0.1.4
3+
version: 0.2.0
44
appVersion: 2.5.2
55
description: Spring Cloud Data Flow is a microservices-based toolkit for building streaming and batch data processing pipelines in Cloud Foundry and Kubernetes.
66
keywords:

‎bitnami/spring-cloud-dataflow/README.md

+11-1
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,8 @@ The following tables lists the configurable parameters of the Spring Cloud Data
128128
| `server.autoscaling.maxReplicas` | Maximum number of Dataflow server replicas | `nil` |
129129
| `server.autoscaling.targetCPU` | Target CPU utilization percentage | `nil` |
130130
| `server.autoscaling.targetMemory` | Target Memory utilization percentage | `nil` |
131+
| `server.jdwp.enabled` | Enable Java Debug Wire Protocol (JDWP) | `false` |
132+
| `server.jdwp.port` | JDWP TCP port | `5005` |
131133

132134
### Dataflow Skipper parameters
133135

@@ -178,6 +180,8 @@ The following tables lists the configurable parameters of the Spring Cloud Data
178180
| `skipper.autoscaling.maxReplicas` | Maximum number of Skipper server replicas | `nil` |
179181
| `skipper.autoscaling.targetCPU` | Target CPU utilization percentage | `nil` |
180182
| `skipper.autoscaling.targetMemory` | Target Memory utilization percentage | `nil` |
183+
| `skipper.jdwp.enabled` | Enable Java Debug Wire Protocol (JDWP) | `false` |
184+
| `skipper.jdwp.port` | JDWP TCP port | `5005` |
181185
| `externalSkipper.host` | Host of a external Skipper Server | `localhost` |
182186
| `externalSkipper.port` | External Skipper Server port number | `7577` |
183187

@@ -237,14 +241,20 @@ The following tables lists the configurable parameters of the Spring Cloud Data
237241
| `externalDatabase.dataflow.database` | Name of the existing database to be used by Dataflow server | `dataflow` |
238242
| `externalDatabase.skipper.user` | Existing username in the external db to be used by Skipper server | `skipper` |
239243
| `externalDatabase.skipper.database` | Name of the existing database to be used by Skipper server | `skipper` |
244+
| `externalDatabase.hibernateDialect` | Hibernate Dialect used by Dataflow/Skipper servers | `""` |
240245

241246
### RabbitMQ chart parameters
242247

243248
| Parameter | Description | Default |
244249
|-------------------------------------------------|-------------------------------------------------------------------------|---------------------------------------------------------|
245250
| `rabbitmq.enabled` | Enable/disable RabbitMQ chart installation | `true` |
246-
| `rabbitmq.auth.user` | RabbitMQ username | `user` |
251+
| `rabbitmq.auth.username` | RabbitMQ username | `user` |
247252
| `rabbitmq.auth.password` | RabbitMQ password | _random 40 character alphanumeric string_ |
253+
| `externalRabbitmq.enabled` | Enable/disable external RabbitMQ | `false` |
254+
| `externalRabbitmq.host` | Host of the external RabbitMQ | `localhost` |
255+
| `externalRabbitmq.port` | External RabbitMQ port number | `5672` |
256+
| `externalRabbitmq.username` | External RabbitMQ username | `guest` |
257+
| `externalRabbitmq.password` | External RabbitMQ password | `guest` |
248258

249259
### Kafka chart parameters
250260

‎bitnami/spring-cloud-dataflow/templates/_helpers.tpl

+75-9
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,7 @@ Return true if a configmap object should be created for Spring Cloud Skipper
150150
{{- end -}}
151151

152152
{{/*
153-
Return the MariaDB database Hostname
153+
Return the database Hostname
154154
*/}}
155155
{{- define "scdf.database.host" -}}
156156
{{- if .Values.mariadb.enabled }}
@@ -161,7 +161,7 @@ Return the MariaDB database Hostname
161161
{{- end -}}
162162

163163
{{/*
164-
Return the MariaDB database Port
164+
Return the database Port
165165
*/}}
166166
{{- define "scdf.database.port" -}}
167167
{{- if .Values.mariadb.enabled }}
@@ -172,24 +172,35 @@ Return the MariaDB database Port
172172
{{- end -}}
173173

174174
{{/*
175-
Return the MariaDB database driver
175+
Return the database driver
176176
*/}}
177177
{{- define "scdf.database.driver" -}}
178178
{{- if .Values.mariadb.enabled -}}
179179
{{- printf "org.mariadb.jdbc.Driver" -}}
180180
{{- else -}}
181-
{{- .Values.database.driver -}}
181+
{{- .Values.externalDatabase.driver -}}
182182
{{- end -}}
183183
{{- end -}}
184184

185185
{{/*
186-
Return the MariaDB database scheme
186+
Return the database scheme
187187
*/}}
188188
{{- define "scdf.database.scheme" -}}
189189
{{- if .Values.mariadb.enabled -}}
190190
{{- printf "mariadb" -}}
191191
{{- else -}}
192-
{{- .Values.database.scheme -}}
192+
{{- .Values.externalDatabase.scheme -}}
193+
{{- end -}}
194+
{{- end -}}
195+
196+
{{/*
197+
Return the JDBC URL parameters
198+
*/}}
199+
{{- define "scdf.database.jdbc.parameters" -}}
200+
{{- if .Values.mariadb.enabled -}}
201+
{{- printf "?useMysqlMetadata=true" -}}
202+
{{- else -}}
203+
{{- printf "" -}}
193204
{{- end -}}
194205
{{- end -}}
195206

@@ -200,7 +211,7 @@ Return the Data Flow Database Name
200211
{{- if .Values.mariadb.enabled }}
201212
{{- printf "dataflow" -}}
202213
{{- else -}}
203-
{{- printf "%s" .Values.externalDatabase.server.database -}}
214+
{{- printf "%s" .Values.externalDatabase.dataflow.database -}}
204215
{{- end -}}
205216
{{- end -}}
206217

@@ -211,7 +222,7 @@ Return the Data Flow Database User
211222
{{- if .Values.mariadb.enabled }}
212223
{{- printf "dataflow" -}}
213224
{{- else -}}
214-
{{- printf "%s" .Values.externalDatabase.server.user -}}
225+
{{- printf "%s" .Values.externalDatabase.dataflow.user -}}
215226
{{- end -}}
216227
{{- end -}}
217228

@@ -238,7 +249,7 @@ Return the Skipper Database User
238249
{{- end -}}
239250

240251
{{/*
241-
Return the MariaDB secret name
252+
Return the Database secret name
242253
*/}}
243254
{{- define "scdf.database.secretName" -}}
244255
{{- if .Values.mariadb.enabled }}
@@ -248,6 +259,61 @@ Return the MariaDB secret name
248259
{{- end -}}
249260
{{- end -}}
250261

262+
{{/*
263+
Return the RabbitMQ host
264+
*/}}
265+
{{- define "scdf.rabbitmq.host" -}}
266+
{{- if .Values.rabbitmq.enabled }}
267+
{{- printf "%s" (include "scdf.rabbitmq.fullname" .) -}}
268+
{{- else -}}
269+
{{- printf "%s" .Values.externalRabbitmq.host -}}
270+
{{- end -}}
271+
{{- end -}}
272+
273+
{{/*
274+
Return the RabbitMQ Port
275+
*/}}
276+
{{- define "scdf.rabbitmq.port" -}}
277+
{{- if .Values.rabbitmq.enabled }}
278+
{{- printf "%d" (.Values.rabbitmq.service.port | int ) -}}
279+
{{- else -}}
280+
{{- printf "%d" (.Values.externalRabbitmq.port | int ) -}}
281+
{{- end -}}
282+
{{- end -}}
283+
284+
{{/*
285+
Return the RabbitMQ username
286+
*/}}
287+
{{- define "scdf.rabbitmq.user" -}}
288+
{{- if .Values.rabbitmq.enabled }}
289+
{{- printf "%s" .Values.rabbitmq.auth.username -}}
290+
{{- else -}}
291+
{{- printf "%s" .Values.externalRabbitmq.username -}}
292+
{{- end -}}
293+
{{- end -}}
294+
295+
{{/*
296+
Return the RabbitMQ secret name
297+
*/}}
298+
{{- define "scdf.rabbitmq.secretName" -}}
299+
{{- if .Values.rabbitmq.enabled }}
300+
{{- printf "%s" (include "scdf.rabbitmq.fullname" .) -}}
301+
{{- else -}}
302+
{{- printf "%s-%s" (include "scdf.fullname" .) "externalrabbitmq" -}}
303+
{{- end -}}
304+
{{- end -}}
305+
306+
{{/*
307+
Return the Hibernate dialect
308+
*/}}
309+
{{- define "scdf.database.hibernate.dialect" -}}
310+
{{- if .Values.mariadb.enabled -}}
311+
{{- printf "org.hibernate.dialect.MariaDB102Dialect" -}}
312+
{{- else -}}
313+
{{- .Values.externalDatabase.hibernateDialect -}}
314+
{{- end -}}
315+
{{- end -}}
316+
251317
{{/*
252318
Compile all warnings into a single message, and call fail.
253319
*/}}

‎bitnami/spring-cloud-dataflow/templates/externaldb-secrets.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
apiVersion: v1
33
kind: Secret
44
metadata:
5-
name: {{ printf "%s-%s" (include "scdf.fullname" .) "externaldb" -}}
5+
name: {{ printf "%s-%s" (include "scdf.fullname" .) "externaldb" }}
66
labels: {{- include "common.labels.standard" . | nindent 4 }}
77
type: Opaque
88
data:

‎bitnami/spring-cloud-dataflow/templates/externalrabbitmq-secrets.yaml

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
{{- if and (.Values.externalRabbitmq.enabled) (not .Values.rabbitmq.enabled) }}
2+
apiVersion: v1
3+
kind: Secret
4+
metadata:
5+
name: {{ printf "%s-%s" (include "scdf.fullname" .) "externalrabbitmq" }}
6+
labels: {{- include "common.labels.standard" . | nindent 4 }}
7+
type: Opaque
8+
data:
9+
rabbitmq-password: {{ .Values.externalRabbitmq.password | b64enc | quote }}
10+
{{- end }}

‎bitnami/spring-cloud-dataflow/templates/scripts-configmap.yaml

+2
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,9 @@ data:
2323
echo "Rollout exit code: '${return_code}'"
2424
return $return_code
2525
}
26+
{{- if .Values.mariadb.enabled }}
2627
k8s_wait_for_statefulset {{ $releaseNamespace }} {{ $mariadbFullname }}
28+
{{- end }}
2729
{{- if or .Values.skipper.enabled .Values.server.configuration.streamingEnabled }}
2830
{{- if .Values.rabbitmq.enabled }}
2931
k8s_wait_for_statefulset {{ $releaseNamespace }} {{ $rabbitmqFullname }}

‎bitnami/spring-cloud-dataflow/templates/server/configmap.yaml

+6-2
Original file line numberDiff line numberDiff line change
@@ -56,16 +56,20 @@ data:
5656
port: {{ $rsocketPort }}
5757
{{- end }}
5858
{{- end }}
59+
{{- $hibernateDialect := include "scdf.database.hibernate.dialect" . }}
60+
{{- if $hibernateDialect }}
5961
jpa:
6062
properties:
6163
hibernate:
62-
dialect: org.hibernate.dialect.MariaDB102Dialect
64+
dialect: {{ $hibernateDialect }}
65+
{{- end }}
6366
datasource:
6467
{{- $databaseScheme := include "scdf.database.scheme" . }}
6568
{{- $databaseHost := include "scdf.database.host" . }}
6669
{{- $databasePort := include "scdf.database.port" . }}
6770
{{- $databaseName := include "scdf.database.server.name" . }}
68-
url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}?useMysqlMetadata=true'
71+
{{- $jdbcParameters := include "scdf.database.jdbc.parameters" . }}
72+
url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}{{ $jdbcParameters }}'
6973
driverClassName: {{ include "scdf.database.driver" . }}
7074
username: {{ include "scdf.database.server.user" . }}
7175
password: ${mariadb-password}

‎bitnami/spring-cloud-dataflow/templates/server/deployment.yaml

+21-4
Original file line numberDiff line numberDiff line change
@@ -84,12 +84,12 @@ spec:
8484
value: "8080"
8585
- name: SPRING_CLOUD_CONFIG_ENABLED
8686
value: "false"
87+
- name: SPRING_CLOUD_KUBERNETES_CONFIG_ENABLE_API
88+
value: "false"
8789
- name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLE_API
88-
value: "true"
90+
value: "false"
8991
- name: SPRING_CLOUD_KUBERNETES_SECRETS_PATHS
9092
value: "/etc/secrets"
91-
- name: SPRING_CLOUD_KUBERNETES_CONFIG_NAME
92-
value: {{ (include "scdf.server.configmapName" .) | quote }}
9393
- name: SPRING_CLOUD_DATAFLOW_SERVER_URI
9494
{{- if ne $serverServicePort 80 }}
9595
value: {{ printf "http://%s-server.%s.svc.%s:%d" $fullname $releaseNamespace $clusterDomain $serverServicePort | quote }}
@@ -122,6 +122,10 @@ spec:
122122
fieldPath: metadata.namespace
123123
- name: KUBERNETES_TRUST_CERTIFICATES
124124
value: {{ ternary "true" "false" .Values.server.configuration.trustK8sCerts | quote }}
125+
{{- if .Values.server.jdwp.enabled }}
126+
- name: JAVA_TOOL_OPTIONS
127+
value: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address={{ .Values.server.jdwp.port }}"
128+
{{- end }}
125129
{{- range $key, $value := .Values.server.extraEnvVars }}
126130
- name: {{ $key }}
127131
value: "{{ $value }}"
@@ -141,6 +145,11 @@ spec:
141145
- name: http
142146
containerPort: 8080
143147
protocol: TCP
148+
{{- if .Values.server.jdwp.enabled }}
149+
- name: jdwp
150+
containerPort: {{ .Values.server.jdwp.port }}
151+
protocol: TCP
152+
{{- end }}
144153
{{- if .Values.server.livenessProbe.enabled }}
145154
livenessProbe:
146155
httpGet:
@@ -174,17 +183,25 @@ spec:
174183
- name: database
175184
mountPath: /etc/secrets/database
176185
readOnly: true
186+
- name: config
187+
mountPath: /opt/bitnami/spring-cloud-dataflow/conf
188+
readOnly: true
177189
{{- if .Values.server.sidecars }}
178190
{{- include "common.tplvalues.render" ( dict "value" .Values.server.sidecars "context" $) | nindent 8 }}
179191
{{- end }}
180192
volumes:
181193
- name: database
182194
secret:
183195
secretName: {{ include "scdf.database.secretName" . }}
196+
- name: config
197+
configMap:
198+
name: {{ include "scdf.server.configmapName" . }}
199+
items:
200+
- key: application.yaml
201+
path: application.yml
184202
{{- if .Values.waitForBackends.enabled }}
185203
- name: scripts
186204
configMap:
187205
name: {{ include "scdf.fullname" . }}-scripts
188206
defaultMode: 0755
189207
{{- end }}
190-

‎bitnami/spring-cloud-dataflow/templates/skipper/configmap.yaml

+10-6
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,10 @@ data:
1515
kubernetes:
1616
accounts:
1717
{{ .Values.skipper.configuration.accountName }}:
18-
{{- if .Values.rabbitmq.enabled }}
19-
{{- $rabbitmqHost := include "scdf.rabbitmq.fullname" . }}
20-
{{- $rabbitmqPort := int .Values.rabbitmq.service.port }}
21-
{{- $rabbitmqUser := .Values.rabbitmq.auth.username }}
18+
{{- if or (.Values.rabbitmq.enabled) (.Values.externalRabbitmq.enabled) }}
19+
{{- $rabbitmqHost := include "scdf.rabbitmq.host" . }}
20+
{{- $rabbitmqPort := include "scdf.rabbitmq.port" . }}
21+
{{- $rabbitmqUser := include "scdf.rabbitmq.user" . }}
2222
environmentVariables: 'SPRING_RABBITMQ_HOST={{ $rabbitmqHost }},SPRING_RABBITMQ_PORT={{ $rabbitmqPort }},SPRING_RABBITMQ_USERNAME={{ $rabbitmqUser }},SPRING_RABBITMQ_PASSWORD=${rabbitmq-password}'
2323
{{- else if .Values.kafka.enabled }}
2424
environmentVariables: 'SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS=${{ printf "{" }}{{ template "scdf.envrelease" . }}_KAFKA_SERVICE_HOST}:${{ printf "{" }}{{ template "scdf.envrelease" . }}_KAFKA_SERVICE_PORT},SPRING_CLOUD_STREAM_KAFKA_BINDER_ZK_NODES=${{ printf "{" }}{{ template "scdf.envrelease" . }}_ZOOKEEPER_SERVICE_HOST}:${{ printf "{" }}{{ template "scdf.envrelease" . }}_ZOOKEEPER_SERVICE_PORT}'
@@ -31,16 +31,20 @@ data:
3131
{{- end }}
3232
readinessProbeDelay: {{ .Values.deployer.readinessProbe.initialDelaySeconds }}
3333
livenessProbeDelay: {{ .Values.deployer.livenessProbe.initialDelaySeconds }}
34+
{{- $hibernateDialect := include "scdf.database.hibernate.dialect" . }}
35+
{{- if $hibernateDialect }}
3436
jpa:
3537
properties:
3638
hibernate:
37-
dialect: org.hibernate.dialect.MariaDB102Dialect
39+
dialect: {{ $hibernateDialect }}
40+
{{- end }}
3841
datasource:
3942
{{- $databaseScheme := include "scdf.database.scheme" . }}
4043
{{- $databaseHost := include "scdf.database.host" . }}
4144
{{- $databasePort := include "scdf.database.port" . }}
4245
{{- $databaseName := include "scdf.database.skipper.name" . }}
43-
url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}?useMysqlMetadata=true'
46+
{{- $jdbcParameters := include "scdf.database.jdbc.parameters" . }}
47+
url: 'jdbc:{{ $databaseScheme }}://{{ $databaseHost }}:{{ $databasePort }}/{{ $databaseName }}{{ $jdbcParameters }}'
4448
driverClassName: {{ include "scdf.database.driver" . }}
4549
username: {{ include "scdf.database.skipper.user" . }}
4650
password: ${mariadb-password}

0 commit comments

Comments
 (0)