[GSoC] Add unit tests for tune API

[helenxie-bit]

What this PR does / why we need it:
This PR adds unit tests and fix some minor bugs of tune API.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

• Docs included if any changes are user facing

[helenxie-bit]

/area gsoc

[helenxie-bit]

Ref: #2339

[andreyvelich]

Thank you for doing this @helenxie-bit!

[helenxie-bit]

@andreyvelich The unit test for tune API is ready. I have also fixed some errors in this PR. Please have a look when you have time. Thanks!

[andreyvelich]

Thank you for doing this @helenxie-bit!
/assign @tenzen-y @johnugeorge @deepanker13 @droctothorpe @Electronic-Waste

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from johnugeorge. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[helenxie-bit]

@andreyvelich Thank you so much for reviewing this! I’ve replied to all your feedback, and please take a look when you have time.

[Electronic-Waste]

Thanks for your great contribution @helenxie-bit ! I left some suggestions about the name of UTs for your reference

I guess it will be more friendly for other developers to understand these UTs if we could specify the scope of each UTs:)

[helenxie-bit]

Thanks for your great contribution @helenxie-bit ! I left some suggestions about the name of UTs for your reference

I guess it will be more friendly for other developers to understand these UTs if we could specify the scope of each UTs:)

@Electronic-Waste Thank you so much for your suggestions! I have incorporated your advice into the UTs.

[helenxie-bit]

@andreyvelich I’ve added the kubeflow-training dependency to the unit tests, allowing values and parameters to be fetched from there instead of redefining them within the Katib unit tests. You can find the change here:

katib/test/unit/v1beta1/requirements.txt

Line 3 in b24b44f

kubeflow-training[huggingface]==1.8.1

Additionally, I have added verification for the experiment object in two valid flow cases in the following section:

katib/sdk/python/v1beta1/kubeflow/katib/api/katib_client_test.py

Lines 587 to 639 in b24b44f

	call_args = mock_create_experiment.call_args
	experiment = call_args[0][0]
	if test_name == "valid flow with custom objective tuning":
	# Verify input_params
	args_content = "".join(
	experiment.spec.trial_template.trial_spec.spec.template.spec.containers[
	0
	].args
	)
	assert "'a': '${trialParameters.a}'" in args_content
	# Verify trial_params
	assert experiment.spec.trial_template.trial_parameters == [
	V1beta1TrialParameterSpec(name="a", reference="a"),
	]
	# Verify experiment_params
	assert experiment.spec.parameters == [
	V1beta1ParameterSpec(
	name="a",
	parameter_type="int",
	feasible_space=V1beta1FeasibleSpace(min="10", max="100"),
	),
	]
	elif test_name == "valid flow with external model tuning":
	# Verify input_params
	args_content = "".join(
	experiment.spec.trial_template.trial_spec.spec.pytorch_replica_specs[
	"Master"
	]
	.template.spec.containers[0]
	.args
	)
	assert (
	'"learning_rate": "${trialParameters.learning_rate}"'
	in args_content
	)
	# Verify trial_params
	assert experiment.spec.trial_template.trial_parameters == [
	V1beta1TrialParameterSpec(
	name="learning_rate", reference="learning_rate"
	),
	]
	# Verify experiment_params
	assert experiment.spec.parameters == [
	V1beta1ParameterSpec(
	name="learning_rate",
	parameter_type="double",
	feasible_space=V1beta1FeasibleSpace(
	min="1e-05", max="5e-05"
	),
	),
	]

Please review when you have time. Thanks!

[helenxie-bit]

@andreyvelich Please review this PR when you have time 👀. I'll adjust the e2e tests for the tune API after this PR is merged, as it fixes some small bugs in the tune API.

[helenxie-bit]

Since @mahdikhashan is currently working on the notebook with instructions for this API, and this PR also includes some bug fixes for the API, @andreyvelich, would it be better to separate the bug fixes into a standalone PR to make the review process more convenient? Please let me know your thoughts.

[andreyvelich]

Since @mahdikhashan is currently working on the notebook with instructions for this API, and this PR also includes some bug fixes for the API, @andreyvelich, would it be better to separate the bug fixes into a standalone PR to make the review process more convenient? Please let me know your thoughts.

Sure, please create separate PR for the bug fixes.

[Electronic-Waste]

Basically LGTM! I left a few comments concerned with objective_metric_name for you.

Thanks for your great contribution! @helenxie-bit

[Electronic-Waste]

It would be better if we also verify the objective_metric_name parameter.