dataset weighting

[vince62s]

Hello,

For finetuning, it seems relevant to overweight a smaller dataset WRT the main data used to train the model.
My suggestion would be to add a weight field in the dataset config list and to take it into account in the load_tts_samples() function.
it's simple and should make the job.
@erogol Any insights?

[erogol]

Yeah makes sense. However, I am not sure if it would make a huge difference in performance. Any experience with that?

[vince62s]

yes it does. for now as a workaround just adding N times the same small finetuning dataset and the more I add the quicker and the higher I am reaching a high SECS.

[erogol]

Speaker sampler has already implemented in the VITS model. Would that help in this case ? @Edresson

[vince62s]

would require some changes because rigth now, language sampler and speaker sampler are mutually exclusive and here

TTS/TTS/tts/models/base_tts.py

Lines 357 to 363 in c63bb48

	if sampler is None:
	if getattr(config, "use_language_weighted_sampler", False):
	print(" > Using Language weighted sampler")
	sampler = get_language_weighted_sampler(dataset.items)
	elif getattr(config, "use_speaker_weighted_sampler", False):
	print(" > Using Language weighted sampler")
	sampler = get_speaker_weighted_sampler(dataset.items)

language sampler prevails (for a good reason).

[yiwei0730]

@Edresson In the paper, we used only the language sampler and created a new language for the speaker that we adapted (ensuring 1/4 of the batch was composed by the speaker adapted)
I read the YourTTS paper and I see this in the adaptation training a one minutes new data, can i ask for how can i use the code in this TTS project to adapt my new dataset with the adapt training, since I didn't see how the code to chose the data 1/4 in the batch.

[vince62s]

since they add initially 3 languages and created a fourth one with the "adapted speaker" data it balanced to 1/4 automatically, but this is not a good way to do things in the long run.

[yiwei0730]

Thanks for your reply @vince62s.
Yes, I think this is a good issue for how to use the less resource data to adapt the data. I have seen the AdaSpeech a long time, they use a conditional layer norm to create the condition for the adaption with 20 unseen speaker data. So i am so interested for how to adapting in the low resources with unseen data.
But in the TTS code, I haven't seen the adapting training code in there, so want to ask for the author @Edresson,
By the base_tts.py code, if i want to use different speaker manager in here by multispeaker manager, so I need to use more dataset item list in the training config?

[yiwei0730]

@vince62s If it is not a good way to do thing in the long run, which methods you prefer to use, like the adaspeech using, using a novel layer or references the other papers?

[Edresson]

Hi @vince62s In YourTTS paper, we used only the language sampler and created a new language for the speaker that we adapted (ensuring 1/4 of the batch was composed by the speaker adapted). To do so, just edit the nn.embedding layer of language, inserting in the last position a copy of the embedding of the language you want to adapt. As the language name in config.json you need to use some name that will be in last considering the alphabetical order of the training languages (considering the training languages "en", "pt" and "fr" a good name is "spk_adpt_en" for example).This approach is far to be optima, for this reason we implemented a way to balancer the language and speaker in a controllable way in this PR #1216

I didn't understand this part "By the base_tts.py code, if i want to use different speaker manager in here by multispeaker manager, so I need to use more dataset item list in the training config?" Could you please elaborate?