GitHub热门项目: csm

GitHub项目：csm 仓库地址：https://github.com/SesameAILabs/csm Stars：14683 | 作者：SesameAILabs 项目描述：A Conversational Speech Generation Model ================================================== README 内容： # CSM **2025/05/20** - CSM is availabile natively in [Hugging Face Transformers](https://huggingface.co/docs/transformers/main/en/model_doc/csm) 🤗 as of version `4.52.1`, more info available [in our model repo](https://huggingface.co/sesame/csm-1b) **2025/03/13** - We are releasing the 1B CSM variant. The checkpoint is [hosted on Hugging Face](https://huggingface.co/sesame/csm_1b). --- CSM (Conversational Speech Model) is a speech generation model from [Sesame](https://www.sesame.com) that generates RVQ audio codes from text and audio inputs. The model architecture employs a [Llama](https://www.llama.com/) backbone and a smaller audio decoder that produces [Mimi](https://huggingface.co/kyutai/mimi) audio codes. A fine-tuned variant of CSM powers the [interactive voice demo](https://www.sesame.com/voicedemo) shown in our [blog post](https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice). A hosted [Hugging Face space](https://huggingface.co/spaces/sesame/csm-1b) is also available for testing audio generation. ## Requirements * A CUDA-compatible GPU * The code has been tested on CUDA 12.4 and 12.6, but it may also work on other versions * Similarly, Python 3.10 is recommended, but newer versions may be fine * For some audio operations, `ffmpeg` may be required * Access to the following Hugging Face models: * [Llama-3.2-1B](https://huggingface.co/meta-llama/Llama-3.2-1B) * [CSM-1B](https://huggingface.co/sesame/csm-1b) ### Setup ```bash git clone git@github.com:SesameAILabs/csm.git cd csm python3.10 -m venv .venv source .venv/bin/activate pip install -r requirements.txt # Disable lazy compilation in Mimi export NO_TORCH_COMPILE=1 # You will need access to CSM-1B and Llama-3.2-1B huggingface-cli login ``` ### Windows Setup The `triton` package cannot be installed in Windows. Instead use `pip install triton-windows`. ## Quickstart This script will generate a conversation between 2 characters, using a prompt for each character. ```bash python run_csm.py ``` ## Usage If you want to write your own applications with CSM, the following examples show basic usage. #### Generate a sentence This will use a random speaker identity, as no prompt or context is provided. ```python from generator import load_csm_1b import torchaudio import torch if torch.backends.mps.is_available(): device = "mps" elif torch.cuda.is_available(): device = "cuda" else: device = "cpu" generator = load_csm_1b(device=device) audio = generator.generate( text="Hello from Sesame.", speaker=0, context=[], max_audio_length_ms=10_000, ) torchaudio.save("audio.wav", audio.unsqueeze(0).cpu(), generator.sample_rate) ``` #### Generate with context CSM sounds best when provided with context. You can prompt or provide context to the mo