Text to Speech: Get timing for each word together with the audio response

[namnm]

Hi. Is it possible to get timing for each word together with the audio response?

[abersheeran]

Your mean, want to get caption?

[namnm]

@abersheeran For example: I have a text "Hello, how are you today?" then I get the audio from api with 5s long. The response should also include the timestamp info when each word was spoken aside with that audio buffer:

{
  "text": "hello",
  "start_time": 0.5,
  "end_time": 1
}

With the above example, the word hello was spoken beginning at 0.5s and ending at 1s (0.5 spoken duration for that word).
Then similar with the rest words, the result can come with an array.

MORE REFERENCE:
ElevenLabs: https://elevenlabs.io/docs/api-reference/text-to-speech/convert-with-timestamps
Google Speech to Text: https://stackoverflow.com/a/64151885/9214970

[babangida4321]

#17

[jcmelati]

Any development here? I really need this feature.

[abersheeran]

We are in development, but there is no confirmed launch date for now.

[Mhdiwe]

Try this: https://theaivoicegenerator.com/timestamped-script-to-voice-generator/
its using fish.audio too

@abersheeran For example: I have a text "Hello, how are you today?" then I get the audio from api with 5s long. The response should also include the timestamp info when each word was spoken aside with that audio buffer:

{
"text": "hello",
"start_time": 0.5,
"end_time": 1
}
With the above example, the word hello was spoken beginning at 0.5s and ending at 1s (0.5 spoken duration for that word). Then similar with the rest words, the result can come with an array.

MORE REFERENCE: ElevenLabs: https://elevenlabs.io/docs/api-reference/text-to-speech/convert-with-timestamps Google Speech to Text: https://stackoverflow.com/a/64151885/9214970

[sashenmethoddata]

@abersheeran Want to know if there's a work around for this in the interim.

I've tried the below:

Using TTS to convert the text to speech
Then STT the audio return above (response = self.fish_audio_session.asr(ASRRequest(audio=audio_data, ignore_timestamps=False)))

This dose give start and end times for sentences, but when mapping trying to map the times back to the original text its always a miss.

[Mhdiwe]

Try here

https://theaivoicegenerator.com/ai-timestamp-srt-generator/