Sources for the genre "Conversation"

[ximenina]

Our starting point for face-to-face conversations is CHILDES. However, other talkbanks are available on the website , where CHILDES is hosted.

• Which talkbanks are the most appropriate to be included in TeDDi?

• Do you know some other sources where we can extract text for this genre that covers many languages, included low-resourced languages?

@tsamardzic @christianbentz @bambooforest

[ximenina]

From talkbank.org I think the following corpora is suitable:

Conversation Banks

CABank

SamtaleBank

ClassBank

Child Language Banks

CHILDES

PhonBank

HomeBank

• We just have to check the annotation specifications to be able to extract the text and filter out unwanted tags and annotations (as much as possible).

[ximenina]

Spoken Corpora from the Clarin project

It comprises several sources with transcriptions of spontaneous and planned speech.

It covers 15 languages: Arabic, Czech, Dutch, Estonian, Finnish, French, German, Hungarian, Italian, Nepali, Norwegian, Polish, Skoti Saami, Slovenian, Spanish, and Swedish.

• We just need to check that the resource is downloable, that the genre is suitable (for example, spontaneous speech, interviews, conversations, etc) and be aware of the particular tags, anotations used in each corpus.

[ximenina]

Other potential resources:

• OpenSLR is a site that hosts speech and language resources. Some of them might be useful especially when they have orthographic transcription available (and the corpus corresponds to the Conversation genre). So far, I've identified these resources:

Multilingual TEDx
MAGICDATA Mandarin Chinese Conversational Speech Corpus

• There are many sources of spoken corpora. Unfortunately, they do not properly represent conversational corpora, i.e., they come from Wikipedia or predefined sentences that the speakers read. In any case, some of these multilingual sources might be useful to obtain text for the under-resource languages (although different genre):

CommonVoice Hundreds of languages, many low-resourced

https://www.openslr.org/79/ Kannada critically endangered
https://www.openslr.org/126/ Kannada critically endangered

[ximenina]

• I updated the potential sources for conversational corpora, in case you want to have a look. @vukbatanovic @tsamardzic @christianbentz

[tsamardzic]

There might be some data for us here too: https://www.amazon.science/blog/amazon-releases-51-language-dataset-for-language-understanding (might not be for conversation)