@Sascha353 Im probably at a stage where this could quickly be implemented as theoretically all the bits are in place.... it would only be a small bit of code and a checkbox to add a streaming option to TG. As you say, played back at the console vs TG actually doing it. So Im popping this on the list of things to try at some point.

Thanks for the heads-up. Yeah, everything comes together nicely :) I actually followed the progress of your repo and also saw the implementation of streaming for the tts_chunk_player. Really impressive what you have created over the last weeks. Also the SillyTavern support! In Browser streaming is also very handy and a good alternative to server-side, if you access via mobile and still want the advantage of streaming!

May I ask what was the intended use of the tts_chunk_player and do you plan on improving on it either in this repo or in an independent one? I feel like it could basically be a standalone tool for audio book generation. If this was the direction you was heading, speaker identification to assign individual voices would be an incredible useful feature.

The only bummer is the fact that coqui.ai is shutting down and they will probably sit on their license. So mid term there will be most likely other tts engines which will be more capable.

BTW: I was hitting the character limit of 250 once during testing of tts_chunk_player some weeks ago but can't reproduce it now (I do not have the same text anymore and updated alltalk since then). Did you improve the chunking with a "sentence lookahead" or something to make sure the current + the next chunk does not exceed the limit? Else I can create a bug report, if I'm able to reproduce it.

@Sascha353 Oh you tried the SillyTavern bits? I was kind of stealthing that in at the moment while I submit the PR to ST.

As for the chunk player, it was just the idea there was more flexibility and things you can do with a TTS and maybe people wanted long format TTS they could take with them or do something more with, like audio books, or their lecture notes or whatever really. Have a look on the Feature Requests/api suite #74 is that the sort of thing you mean?

As for Coqui Im waiting to see what happens on that one. I've got confirmation that everything that costs no money is remaining up and but Im waiting to see what happens with their github etc as they wont close it down, but someone will need to take it on, and theres a lots of TTS engines out there relying on their code.... So I cant see it falling to the wayside and Ive seen other AI companies showing an interest in taking on the assets. Time will tell on that one.

The 250 character limit you cant fully get around if you have sentences longer than 250 characters. Its a complex one... If you split a long sentence, the TTS engine will break the flow of the sentence making it sound like 2x sentences, but go over the 250 characters and you *may (not guaranteed) start to get TTS degradation. It may be something I could flag later on as a "check these sentences" before committing to one wav, allowing you to regenerate if they sound funny.

Ill probably do bits and bobs with the with it. Though Ive been working so damn hard on it for a month+ now (generally) I need things to stabilise a little. Make sure generally all the code is good and clean up any issues, before I go on doing other things.

Github informs about all the changes, no stealthing possible :D I tested it just briefly. It works good and it's fast when using streaming without narrator. I also found one possible improvement already. If you have multiple characters talking or the same character talks again while audio is still playing, it will play new audio in parallel. Could be solved with an option for synchronous playback where audio is queued and only played if the current playback is done. Alternatively new playback interrupts current playback, maybe as a flag. Also, Narrator support in streaming mode would be nice of course :).

Yes, #74 seems to be close to what I had in mind. However, not all e-books have speaker tags like "Speaker1: ", so without a sophisticated AI for speaker identification it's probably difficult to have a solution for all cases. For those where speakers are always at the beginning of a sentence it could be done with regex, I guess. I also thought about having it in the tts_chunk_player UI but if the API supports it, someone could build the logic of reading text, identifying speakers and calling the TTS API.

I didn't encounter the character limit since then. I'll try to track it down, when I see it again.

it will play new audio in parallel Im about to make the PR changes they requested... and one is to shift handling of streaming playback to "global TTS controls"... so it will end up being whatever ST does, Ill have no control over that.

Narrator support in streaming mode technically yes, but.... Id have to put A lot of code into the ST extension.... and theres no way Im submitting a PR to ST with another 400+lines of code. Its already about 3-4x bigger than all bar 1 other TTS extension. I just wanted to get the main bit in for now. See how that goes, deal with bugs/issues+new user base (possibly) and bump other features in later on.

Hey @erew123 sorry i missed your reply its been a long time.

No need for an NDA at this point but if we can speak in private you can msg me on discord - https://discord.gg/GU9T8mPt
(PatriotSoul)

I'v tried to find your contact you can also share here if you would like and i'd be happy to contact you.

Thank you looking forward hearing from you.

@mercuryyy No probs. Can you give me 24-48 hours and Ill drop you a msg on discord. Im just in the final bits of getting this v2 tested/out and Ill catch up once Ive cleared my desk of all this !

Yeah sure thing! Excited for the release. If you need help testing let me know.

Not forgotten... Just been cramming like crazy to get the BETA out AllTalk v2 BETA Download Details & Discussion

I will drop you a msg on Discord in a day or so!

sent you a friend request on Discord as the link had expired. It will be from Kudo***

Hi @gboross I'm assuming you don't have Text-generation-webui? https://github.com/oobabooga/text-generation-webui

If not, you would do a standalone installation (2nd option down here) and basically a duplicate of this video here (please ensure you have the correct sections of the C++ tools installed as its a Python requirement.

The docker installs within Windows Subsystem for Linux and will install DeepSpeed as standard (CUDA/Nvidia card being a requirement here). There's a CUDA and non CUDA version of the docker, so you will be given the choice as to which to use. If you dont use the CUDA version, you are basically running in CPU mode as AMD and Mac Metal support are still not working (due to AMD and Apple lagging behind with issues currently).

Are you referring to the speaker.pth file? coqui-ai/TTS#1688 and https://docs.coqui.ai/en/latest/models/xtts.html#inference to use the models trained speakers?

Thanks

Thank you for your response. I've made some progress, but for some reason, it doesn't want to download the xttsv2_2.0.2 model when I run start_alltalk. Anyway, I'm trying with Docker; I've got it running, but I can't seem to find the endpoints and documentation. Actually, I intend to use it as a streaming server, similar to how we've been using the Coqui streaming server, but that one doesn't have deepspeed, nor is it further developed. I've also attached an example of the .json file I'm thinking of.
So, the Docker is up and running, what's the next step? :D

default_speaker.json

Ah, I think I got it working on Docker, sorry for all the silly questions.
However, how do I determine if deepspeed is actually working or not? Because I don't feel any difference between this and the Coqui xtts streaming server, where it supposedly isn't present.

Also, here, before every generation, it analyzes the specific speaker's .wav file, right? That adds to the latency too, doesn't it?

Oh, I figured it out in the meantime, sorry for all the questions. It's brutally fast with deepspeed. The request for the speaker.json still stands. :)

Hi @gboross I've had a look for anything to do with speaker.json files to do with XTTSv2 (that AllTalk is based upon). I never saw any reference to them when I developed them and I've had decent hunt for anything to do with them. I cant find any reference or code samples for XTTSv2 using them https://docs.coqui.ai/en/latest/models/xtts.html#inference and this from 2022 suggests that Coqui moved away from JSON files coqui-ai/TTS#1688 other than for VITS models.

Having said that, I looked specifically at their code https://github.com/coqui-ai/xtts-streaming-server/blob/main/server/main.py and also their video, which does show it downloading an XTTSv2 model. I suspect this is related to the speaker.pth file and possibly replacing that to use that with conditioned latents. However, Ive made no provision currently in AllTalk for doing this, because Coqui made no reference to either being able to do this in any code samples or therefore any explanation as to how it works. Best I can tell it is doing exactly what AllTalk is doing on each run, its just AllTalk uses the wav sample file each time and their XTTS streaming server is using a pre-calculated JSON file that has gone through this code:

def predict_speaker(wav_file: UploadFile):
    """Compute conditioning inputs from reference audio file."""
    temp_audio_name = next(tempfile._get_candidate_names())
    with open(temp_audio_name, "wb") as temp, torch.inference_mode():
        temp.write(io.BytesIO(wav_file.file.read()).getbuffer())
        gpt_cond_latent, speaker_embedding = model.get_conditioning_latents(
            temp_audio_name
        )
    return {
        "gpt_cond_latent": gpt_cond_latent.cpu().squeeze().half().tolist(),
        "speaker_embedding": speaker_embedding.cpu().squeeze().half().tolist(),
    }


def postprocess(wav):
    """Post process the output waveform"""
    if isinstance(wav, list):
        wav = torch.cat(wav, dim=0)
    wav = wav.clone().detach().cpu().numpy()
    wav = wav[None, : int(wav.shape[0])]
    wav = np.clip(wav, -1, 1)
    wav = (wav * 32767).astype(np.int16)
    return wav


def encode_audio_common(
    frame_input, encode_base64=True, sample_rate=24000, sample_width=2, channels=1
):
    """Return base64 encoded audio"""
    wav_buf = io.BytesIO()
    with wave.open(wav_buf, "wb") as vfout:
        vfout.setnchannels(channels)
        vfout.setsampwidth(sample_width)
        vfout.setframerate(sample_rate)
        vfout.writeframes(frame_input)

    wav_buf.seek(0)
    if encode_base64:
        b64_encoded = base64.b64encode(wav_buf.getbuffer()).decode("utf-8")
        return b64_encoded
    else:
        return wav_buf.read(

As I say, this is effectively what AllTalk is doing each time you request TTS to be generated. However, AllTalk isn't built to handle JSON files for the speaker file. Can it be done, yes, but its not a simple change at all. In fact it would be a re-write of quite a lot of bits of code, through to the API, model loaders and other such things. I dont know of any way to reverse a JSON to a wav, I actually suspect its probably not possible. Do you have the original sample wav file that you used to generate your JSON file as that would probably be your simplest route?